|
2021-06-24
ยง
|
| 23:02 |
<legoktm> |
reverted cumin1001 spicerack live hacks |
[production] |
| 22:57 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) |
[production] |
| 22:55 |
<legoktm@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.08-start-maintenance |
[production] |
| 22:55 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=0) |
[production] |
| 22:55 |
<legoktm@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.08-restore-ttl |
[production] |
| 22:36 |
<volans> |
set x2 codfw master back to RW |
[production] |
| 22:30 |
<legoktm@cumin1001> |
END (ERROR) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=97) |
[production] |
| 22:29 |
<legoktm@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.03-set-db-readonly |
[production] |
| 22:29 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0) |
[production] |
| 22:29 |
<legoktm@cumin1001> |
[DRY-RUN] MediaWiki read-only period starts at: 2021-06-24 22:29:25.643909 |
[production] |
| 22:29 |
<legoktm@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.02-set-readonly |
[production] |
| 22:09 |
<legoktm@cumin1001> |
END (FAIL) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=99) |
[production] |
| 22:09 |
<legoktm@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance |
[production] |
| 22:06 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.00-warmup-caches (exit_code=0) |
[production] |
| 22:05 |
<legoktm@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.00-warmup-caches |
[production] |
| 22:04 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) |
[production] |
| 22:04 |
<legoktm@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl |
[production] |
| 22:01 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) |
[production] |
| 22:01 |
<legoktm@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.00-disable-puppet |
[production] |
| 21:59 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) |
[production] |
| 21:58 |
<legoktm@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.00-disable-puppet |
[production] |
| 21:47 |
<legoktm> |
live hacked spicerack on cumin1001 to revert https://gerrit.wikimedia.org/r/c/operations/software/spicerack/+/700963/ |
[production] |
| 20:58 |
<legoktm> |
starting dry run and live test of DC switchover |
[production] |
| 20:53 |
<legoktm> |
legoktm@phab1001:~$ sudo /srv/phab/phabricator/bin/remove destroy M320 (spam) |
[production] |
| 20:44 |
<volans> |
uploaded spicerack_0.0.55 to apt.wikimedia.org buster-wikimedia,bullseye-wikimedia |
[production] |
| 20:28 |
<legoktm> |
re-enabled daily digests for wikimedia-l - T285486 |
[production] |
| 19:10 |
<dduvall@deploy1002> |
rebuilt and synchronized wikiversions files: all wikis to 1.37.0-wmf.11 |
[production] |
| 19:07 |
<dduvall@deploy1002> |
Synchronized php: group1 wikis to 1.37.0-wmf.11 (duration: 01m 04s) |
[production] |
| 19:06 |
<dduvall@deploy1002> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.37.0-wmf.11 |
[production] |
| 19:04 |
<dduvall> |
preparing to roll group2 to 1.37.0-wmf.11 (T281152) (cc risky patch contacts Amir1 Krinkle DannyS712) |
[production] |
| 19:02 |
<cmjohnson@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 18:56 |
<cmjohnson@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 18:21 |
<cmjohnson@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 18:18 |
<cmjohnson@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 18:17 |
<cmjohnson@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 18:12 |
<cmjohnson@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 17:12 |
<dduvall@deploy1002> |
Synchronized php: group1 wikis to 1.37.0-wmf.11 (duration: 01m 06s) |
[production] |
| 17:11 |
<dduvall@deploy1002> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.37.0-wmf.11 |
[production] |
| 17:08 |
<dduvall> |
re-rolling group1 to 1.37.0-wmf.11 (T281152) following deployment of blocker fixes (cc risky patch contacts Amir1 Krinkle DannyS712) |
[production] |
| 16:12 |
<twentyafterfour> |
restarted php7.3-fpm on phab1001 |
[production] |
| 15:43 |
<hnowlan> |
running `nodetool decommission` on maps2007 |
[production] |
| 15:42 |
<hnowlan@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=maps2007.codfw.wmnet |
[production] |
| 15:42 |
<hnowlan@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on maps2007.codfw.wmnet with reason: depooling and reimaging as buster replica |
[production] |
| 15:42 |
<hnowlan@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on maps2007.codfw.wmnet with reason: depooling and reimaging as buster replica |
[production] |
| 15:31 |
<moritzm> |
installing jackson-databind security updates |
[production] |
| 15:26 |
<moritzm> |
installing ruby-websocket-extensions security updates |
[production] |
| 15:02 |
<hnowlan> |
reenabling puppet on P{C:Postgresql::Slave} |
[production] |
| 14:59 |
<moritzm> |
restarting mw canaries to pick up libxml2 security update |
[production] |
| 14:57 |
<moritzm> |
installing libxml2 security updates on buster |
[production] |
| 14:46 |
<hnowlan> |
Disabling puppet on P{C:Postgresql::Slave} (netboxdb2001,puppetdb2002, most maps hosts) to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/700071 |
[production] |