2021-06-25
§
|
08:55 |
<klausman@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ml-serve-ctrl2002.codfw.wmnet |
[production] |
08:54 |
<klausman@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd2001.codfw.wmnet |
[production] |
08:52 |
<klausman@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ml-etcd2001.codfw.wmnet |
[production] |
08:52 |
<klausman@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd2002.codfw.wmnet |
[production] |
08:51 |
<klausman@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ml-etcd2002.codfw.wmnet |
[production] |
08:51 |
<klausman@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd2003.codfw.wmnet |
[production] |
08:48 |
<klausman@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host ml-etcd2003.codfw.wmnet |
[production] |
08:12 |
<klausman@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve-ctrl1001.eqiad.wmnet |
[production] |
08:07 |
<klausman@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host ml-serve-ctrl1001.eqiad.wmnet |
[production] |
08:07 |
<klausman@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve-ctrl1002.eqiad.wmnet |
[production] |
08:04 |
<klausman@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host ml-serve-ctrl1002.eqiad.wmnet |
[production] |
08:04 |
<klausman@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd1001.eqiad.wmnet |
[production] |
08:01 |
<elukey> |
reboot an-worker1101 to unblock stuck GPU |
[production] |
08:00 |
<klausman@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host ml-etcd1001.eqiad.wmnet |
[production] |
08:00 |
<klausman@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd1002.eqiad.wmnet |
[production] |
07:58 |
<klausman@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host ml-etcd1002.eqiad.wmnet |
[production] |
07:58 |
<klausman@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd1003.eqiad.wmnet |
[production] |
07:57 |
<klausman@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host ml-etcd1003.eqiad.wmnet |
[production] |
07:42 |
<moritzm> |
imported Jenkins 2.289.1 to thirdparty/ci for buster-wikimedia T285531 |
[production] |
07:30 |
<dcausse> |
depool and restart blazegraph on wdqs1005 |
[production] |
07:17 |
<dcausse> |
installing openjdk-8-dbg on wdqs1005 to debug blazegraph |
[production] |
2021-06-24
§
|
23:02 |
<legoktm> |
reverted cumin1001 spicerack live hacks |
[production] |
22:57 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) |
[production] |
22:55 |
<legoktm@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.08-start-maintenance |
[production] |
22:55 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=0) |
[production] |
22:55 |
<legoktm@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.08-restore-ttl |
[production] |
22:36 |
<volans> |
set x2 codfw master back to RW |
[production] |
22:30 |
<legoktm@cumin1001> |
END (ERROR) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=97) |
[production] |
22:29 |
<legoktm@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.03-set-db-readonly |
[production] |
22:29 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0) |
[production] |
22:29 |
<legoktm@cumin1001> |
[DRY-RUN] MediaWiki read-only period starts at: 2021-06-24 22:29:25.643909 |
[production] |
22:29 |
<legoktm@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.02-set-readonly |
[production] |
22:09 |
<legoktm@cumin1001> |
END (FAIL) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=99) |
[production] |
22:09 |
<legoktm@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance |
[production] |
22:06 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.00-warmup-caches (exit_code=0) |
[production] |
22:05 |
<legoktm@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.00-warmup-caches |
[production] |
22:04 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) |
[production] |
22:04 |
<legoktm@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl |
[production] |
22:01 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) |
[production] |
22:01 |
<legoktm@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.00-disable-puppet |
[production] |
21:59 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) |
[production] |
21:58 |
<legoktm@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.00-disable-puppet |
[production] |
21:47 |
<legoktm> |
live hacked spicerack on cumin1001 to revert https://gerrit.wikimedia.org/r/c/operations/software/spicerack/+/700963/ |
[production] |
20:58 |
<legoktm> |
starting dry run and live test of DC switchover |
[production] |
20:53 |
<legoktm> |
legoktm@phab1001:~$ sudo /srv/phab/phabricator/bin/remove destroy M320 (spam) |
[production] |
20:44 |
<volans> |
uploaded spicerack_0.0.55 to apt.wikimedia.org buster-wikimedia,bullseye-wikimedia |
[production] |
20:28 |
<legoktm> |
re-enabled daily digests for wikimedia-l - T285486 |
[production] |
19:10 |
<dduvall@deploy1002> |
rebuilt and synchronized wikiversions files: all wikis to 1.37.0-wmf.11 |
[production] |
19:07 |
<dduvall@deploy1002> |
Synchronized php: group1 wikis to 1.37.0-wmf.11 (duration: 01m 04s) |
[production] |
19:06 |
<dduvall@deploy1002> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.37.0-wmf.11 |
[production] |