9851-9900 of 10000 results (139ms)
2024-09-19 §
01:25 <pt1979@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update dns name for frack new switches - pt1979@cumin2002" [production]
01:24 <pt1979@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update dns name for frack new switches - pt1979@cumin2002" [production]
01:21 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
00:46 <sukhe> sudo cumin 'puppetserver1003* or puppetserver2003*' 'systemctl start sync-puppet-volatile.service' [production]
00:45 <sukhe> sukhe@puppetserver1002:~$ sudo systemctl start sync-puppet-volatile.service [production]
00:41 <swfrench-wmf> force-reboot of puppetserver1001 via ipmitool (unresponsive for over 30m) [production]
2024-09-18 §
22:43 <swfrench@deploy1003> helmfile [eqiad] DONE helmfile.d/services/eventstreams: sync [production]
22:43 <swfrench@deploy1003> helmfile [eqiad] START helmfile.d/services/eventstreams: sync [production]
22:19 <jynus> inserting without binlog missing heartbeat reecod on x1 codfw hosts [production]
22:11 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw [production]
21:55 <mutante> seaborgium - apt-get clean (disk space before: 98% used, now: 76% used, was alerting) [production]
20:59 <ladsgroup@cumin1002> START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw [production]
20:45 <toyofuku@deploy1003> Finished scap sync-world: Backport for [[gerrit:1073836|Enable dark mode for all logged in users on all projects (T370099)]], [[gerrit:1073835|Deploy Vector 2022 on several Wikimedia wikis (T374255)]], [[gerrit:1073839|Limit quick surveys to wikis with messages defined (T374654)]] (duration: 12m 52s) [production]
20:40 <toyofuku@deploy1003> toyofuku, jdlrobson: Continuing with sync [production]
20:35 <toyofuku@deploy1003> toyofuku, jdlrobson: Backport for [[gerrit:1073836|Enable dark mode for all logged in users on all projects (T370099)]], [[gerrit:1073835|Deploy Vector 2022 on several Wikimedia wikis (T374255)]], [[gerrit:1073839|Limit quick surveys to wikis with messages defined (T374654)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
20:32 <toyofuku@deploy1003> Started scap sync-world: Backport for [[gerrit:1073836|Enable dark mode for all logged in users on all projects (T370099)]], [[gerrit:1073835|Deploy Vector 2022 on several Wikimedia wikis (T374255)]], [[gerrit:1073839|Limit quick surveys to wikis with messages defined (T374654)]] [production]
18:08 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2220 (re)pooling @ 100%: T375050', diff saved to https://phabricator.wikimedia.org/P69310 and previous config saved to /var/cache/conftool/dbconfig/20240918-180800-arnaudb.json [production]
17:55 <dreamyjazz@deploy1003> Finished scap sync-world: Backport for [[gerrit:1073853|Revert^2 "Create group for assigning checkuser-temporary-account right" (T369187)]] (duration: 08m 18s) [production]
17:52 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2220 (re)pooling @ 75%: T375050', diff saved to https://phabricator.wikimedia.org/P69309 and previous config saved to /var/cache/conftool/dbconfig/20240918-175255-arnaudb.json [production]
17:51 <dreamyjazz@deploy1003> dreamyjazz: Continuing with sync [production]
17:49 <dreamyjazz@deploy1003> dreamyjazz: Backport for [[gerrit:1073853|Revert^2 "Create group for assigning checkuser-temporary-account right" (T369187)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
17:47 <dreamyjazz@deploy1003> Started scap sync-world: Backport for [[gerrit:1073853|Revert^2 "Create group for assigning checkuser-temporary-account right" (T369187)]] [production]
17:37 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2220 (re)pooling @ 50%: T375050', diff saved to https://phabricator.wikimedia.org/P69308 and previous config saved to /var/cache/conftool/dbconfig/20240918-173749-arnaudb.json [production]
17:29 <sukhe> re-enable puppet on A:cp to finish rolling out T347114 [production]
17:29 <sukhe> re-enable puppet on A:cp to finish rolling out T368755 [production]
17:22 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2220 (re)pooling @ 25%: T375050', diff saved to https://phabricator.wikimedia.org/P69306 and previous config saved to /var/cache/conftool/dbconfig/20240918-172243-arnaudb.json [production]
17:09 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2217 (re)pooling @ 100%: T373104', diff saved to https://phabricator.wikimedia.org/P69305 and previous config saved to /var/cache/conftool/dbconfig/20240918-170918-arnaudb.json [production]
17:09 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2216 (re)pooling @ 100%: T373104', diff saved to https://phabricator.wikimedia.org/P69304 and previous config saved to /var/cache/conftool/dbconfig/20240918-170913-arnaudb.json [production]
17:09 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2215 (re)pooling @ 100%: T373104', diff saved to https://phabricator.wikimedia.org/P69303 and previous config saved to /var/cache/conftool/dbconfig/20240918-170909-arnaudb.json [production]
17:09 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2194 (re)pooling @ 100%: T373104', diff saved to https://phabricator.wikimedia.org/P69302 and previous config saved to /var/cache/conftool/dbconfig/20240918-170903-arnaudb.json [production]
17:08 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2193 (re)pooling @ 100%: T373104', diff saved to https://phabricator.wikimedia.org/P69301 and previous config saved to /var/cache/conftool/dbconfig/20240918-170858-arnaudb.json [production]
17:08 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2172 (re)pooling @ 100%: T373104', diff saved to https://phabricator.wikimedia.org/P69300 and previous config saved to /var/cache/conftool/dbconfig/20240918-170849-arnaudb.json [production]
17:08 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2140 (re)pooling @ 100%: T373104', diff saved to https://phabricator.wikimedia.org/P69299 and previous config saved to /var/cache/conftool/dbconfig/20240918-170843-arnaudb.json [production]
17:08 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2130 (re)pooling @ 100%: T373104', diff saved to https://phabricator.wikimedia.org/P69298 and previous config saved to /var/cache/conftool/dbconfig/20240918-170838-arnaudb.json [production]
17:08 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2129 (re)pooling @ 100%: T373104', diff saved to https://phabricator.wikimedia.org/P69297 and previous config saved to /var/cache/conftool/dbconfig/20240918-170833-arnaudb.json [production]
17:07 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2220 (re)pooling @ 15%: T375050', diff saved to https://phabricator.wikimedia.org/P69296 and previous config saved to /var/cache/conftool/dbconfig/20240918-170738-arnaudb.json [production]
17:02 <mutante> copied vopsbot.db from alert1001 to alert1002; restarted vopsbot [production]
16:57 <hnowlan@deploy1003> helmfile [codfw] DONE helmfile.d/services/shellbox-video: apply [production]
16:56 <hnowlan@deploy1003> helmfile [codfw] START helmfile.d/services/shellbox-video: apply [production]
16:56 <hnowlan@deploy1003> helmfile [eqiad] DONE helmfile.d/services/shellbox-video: apply [production]
16:56 <hnowlan@deploy1003> helmfile [eqiad] START helmfile.d/services/shellbox-video: apply [production]
16:54 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2217 (re)pooling @ 75%: T373104', diff saved to https://phabricator.wikimedia.org/P69294 and previous config saved to /var/cache/conftool/dbconfig/20240918-165412-arnaudb.json [production]
16:54 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2216 (re)pooling @ 75%: T373104', diff saved to https://phabricator.wikimedia.org/P69293 and previous config saved to /var/cache/conftool/dbconfig/20240918-165407-arnaudb.json [production]
16:54 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2215 (re)pooling @ 75%: T373104', diff saved to https://phabricator.wikimedia.org/P69292 and previous config saved to /var/cache/conftool/dbconfig/20240918-165403-arnaudb.json [production]
16:53 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2194 (re)pooling @ 75%: T373104', diff saved to https://phabricator.wikimedia.org/P69291 and previous config saved to /var/cache/conftool/dbconfig/20240918-165357-arnaudb.json [production]
16:53 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2193 (re)pooling @ 75%: T373104', diff saved to https://phabricator.wikimedia.org/P69290 and previous config saved to /var/cache/conftool/dbconfig/20240918-165352-arnaudb.json [production]
16:53 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2172 (re)pooling @ 75%: T373104', diff saved to https://phabricator.wikimedia.org/P69289 and previous config saved to /var/cache/conftool/dbconfig/20240918-165344-arnaudb.json [production]
16:53 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2140 (re)pooling @ 75%: T373104', diff saved to https://phabricator.wikimedia.org/P69288 and previous config saved to /var/cache/conftool/dbconfig/20240918-165337-arnaudb.json [production]
16:53 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2130 (re)pooling @ 75%: T373104', diff saved to https://phabricator.wikimedia.org/P69287 and previous config saved to /var/cache/conftool/dbconfig/20240918-165332-arnaudb.json [production]
16:53 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2129 (re)pooling @ 75%: T373104', diff saved to https://phabricator.wikimedia.org/P69286 and previous config saved to /var/cache/conftool/dbconfig/20240918-165327-arnaudb.json [production]