1951-2000 of 10000 results (33ms)
2021-06-25 §
08:55 <klausman@cumin2001> START - Cookbook sre.hosts.reboot-single for host ml-serve-ctrl2002.codfw.wmnet [production]
08:54 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd2001.codfw.wmnet [production]
08:52 <klausman@cumin2001> START - Cookbook sre.hosts.reboot-single for host ml-etcd2001.codfw.wmnet [production]
08:52 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd2002.codfw.wmnet [production]
08:51 <klausman@cumin2001> START - Cookbook sre.hosts.reboot-single for host ml-etcd2002.codfw.wmnet [production]
08:51 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd2003.codfw.wmnet [production]
08:48 <klausman@cumin2001> START - Cookbook sre.hosts.reboot-single for host ml-etcd2003.codfw.wmnet [production]
08:12 <klausman@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve-ctrl1001.eqiad.wmnet [production]
08:07 <klausman@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-serve-ctrl1001.eqiad.wmnet [production]
08:07 <klausman@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve-ctrl1002.eqiad.wmnet [production]
08:04 <klausman@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-serve-ctrl1002.eqiad.wmnet [production]
08:04 <klausman@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd1001.eqiad.wmnet [production]
08:01 <elukey> reboot an-worker1101 to unblock stuck GPU [production]
08:00 <klausman@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-etcd1001.eqiad.wmnet [production]
08:00 <klausman@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd1002.eqiad.wmnet [production]
07:58 <klausman@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-etcd1002.eqiad.wmnet [production]
07:58 <klausman@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd1003.eqiad.wmnet [production]
07:57 <klausman@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-etcd1003.eqiad.wmnet [production]
07:42 <moritzm> imported Jenkins 2.289.1 to thirdparty/ci for buster-wikimedia T285531 [production]
07:30 <dcausse> depool and restart blazegraph on wdqs1005 [production]
07:17 <dcausse> installing openjdk-8-dbg on wdqs1005 to debug blazegraph [production]
2021-06-24 §
23:02 <legoktm> reverted cumin1001 spicerack live hacks [production]
22:57 <legoktm@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) [production]
22:55 <legoktm@cumin1001> START - Cookbook sre.switchdc.mediawiki.08-start-maintenance [production]
22:55 <legoktm@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=0) [production]
22:55 <legoktm@cumin1001> START - Cookbook sre.switchdc.mediawiki.08-restore-ttl [production]
22:36 <volans> set x2 codfw master back to RW [production]
22:30 <legoktm@cumin1001> END (ERROR) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=97) [production]
22:29 <legoktm@cumin1001> START - Cookbook sre.switchdc.mediawiki.03-set-db-readonly [production]
22:29 <legoktm@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0) [production]
22:29 <legoktm@cumin1001> [DRY-RUN] MediaWiki read-only period starts at: 2021-06-24 22:29:25.643909 [production]
22:29 <legoktm@cumin1001> START - Cookbook sre.switchdc.mediawiki.02-set-readonly [production]
22:09 <legoktm@cumin1001> END (FAIL) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=99) [production]
22:09 <legoktm@cumin1001> START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance [production]
22:06 <legoktm@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.00-warmup-caches (exit_code=0) [production]
22:05 <legoktm@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-warmup-caches [production]
22:04 <legoktm@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) [production]
22:04 <legoktm@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl [production]
22:01 <legoktm@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) [production]
22:01 <legoktm@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-disable-puppet [production]
21:59 <legoktm@cumin1001> END (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) [production]
21:58 <legoktm@cumin1001> START - Cookbook sre.switchdc.mediawiki.00-disable-puppet [production]
21:47 <legoktm> live hacked spicerack on cumin1001 to revert https://gerrit.wikimedia.org/r/c/operations/software/spicerack/+/700963/ [production]
20:58 <legoktm> starting dry run and live test of DC switchover [production]
20:53 <legoktm> legoktm@phab1001:~$ sudo /srv/phab/phabricator/bin/remove destroy M320 (spam) [production]
20:44 <volans> uploaded spicerack_0.0.55 to apt.wikimedia.org buster-wikimedia,bullseye-wikimedia [production]
20:28 <legoktm> re-enabled daily digests for wikimedia-l - T285486 [production]
19:10 <dduvall@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.37.0-wmf.11 [production]
19:07 <dduvall@deploy1002> Synchronized php: group1 wikis to 1.37.0-wmf.11 (duration: 01m 04s) [production]
19:06 <dduvall@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.37.0-wmf.11 [production]