|
2020-08-31
§
|
| 12:37 |
<oblivian@cumin2001> |
START - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep |
[production] |
| 12:14 |
<oblivian@cumin2001> |
END (PASS) - Cookbook sre.switchdc.services.02-restore-ttl (exit_code=0) |
[production] |
| 12:14 |
<oblivian@cumin2001> |
START - Cookbook sre.switchdc.services.02-restore-ttl |
[production] |
| 12:13 |
<oblivian@cumin2001> |
END (PASS) - Cookbook sre.switchdc.services.01-switch-dc (exit_code=0) |
[production] |
| 12:13 |
<oblivian@cumin2001> |
Switching services restbase-async: eqiad => codfw |
[production] |
| 12:13 |
<oblivian@cumin2001> |
START - Cookbook sre.switchdc.services.01-switch-dc |
[production] |
| 12:10 |
<oblivian@cumin2001> |
END (PASS) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=0) |
[production] |
| 12:05 |
<oblivian@cumin2001> |
START - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep |
[production] |
| 11:58 |
<elukey> |
reimage kafka-jumbo1001 to Buster |
[production] |
| 11:32 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.6/extensions/WikimediaEvents/modules/ext.wikimediaEvents/searchSatisfaction.js: 5d583d9550787a8e36c29ca841233615405fcb7e: Disable MediaSearch A/B test (duration: 00m 55s) |
[production] |
| 11:22 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 81f88fde2aad23a619047b1177a6188f51df11a9: Enable Signature button on Wikiproject for hywiki (T261550) (duration: 00m 54s) |
[production] |
| 11:22 |
<jbond42> |
removing old hiera version 1 and 3 backends |
[production] |
| 11:17 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: b74893fecdaae599077daad5b1219ad3b9bc7fc9: Enable sitenotice on mobile for closed wikis (T261357) (duration: 00m 56s) |
[production] |
| 11:02 |
<volans> |
upgraded spicerack to 0.0.41 on cumin hosts |
[production] |
| 10:27 |
<hnowlan@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . |
[production] |
| 09:51 |
<elukey> |
executed /srv/phab/phabricator/bin/remove destroy @klausman on phab1001 (following https://wikitech.wikimedia.org/wiki/Phabricator#Delete_a_user) to clear incosistent state of new account (wrong email address) |
[production] |
| 08:43 |
<moritzm> |
installing bind9 security updates on stretch/buster (client-side tools/libs only) |
[production] |
| 07:53 |
<volans> |
uploaded spicerack_0.0.41 to apt.wikimedia.org buster-wikimedia |
[production] |
| 07:30 |
<moritzm> |
installing squid security updates |
[production] |
| 07:24 |
<moritzm> |
installing openexr security updates on buster |
[production] |
| 07:12 |
<marostegui> |
Sanitize jawikivoyage on db2094:3325 and db1124:3325 T260482 |
[production] |
| 06:30 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 06:27 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 06:06 |
<elukey> |
reimage kafka-jumbo1005 to Debian Buster |
[production] |
| 05:21 |
<marostegui> |
Reload haproxy on dbproxy1017 and dbproxy1021 to test db1128 |
[production] |
|
2020-08-28
§
|
| 21:53 |
<ryankemper> |
`sudo systemctl reload nginx.service` on `cloudelastic100[5,6].wikimedia.org` to try to resolve certificate warning issues |
[production] |
| 19:11 |
<andrewbogott> |
rebooting cloudvirt1006. It's a spare, unused system but showing a bus error and icinga alerts; not worth saving if it needs saving |
[production] |
| 17:39 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 17:39 |
<mutante> |
shutting down mw2196 |
[production] |
| 17:37 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 16:40 |
<rzl> |
switchdc live test complete |
[production] |
| 16:36 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.08-update-tendril (exit_code=0) |
[production] |
| 16:35 |
<rzl@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.08-update-tendril |
[production] |
| 16:35 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) |
[production] |
| 16:34 |
<rzl@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.08-start-maintenance |
[production] |
| 16:33 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=0) |
[production] |
| 16:33 |
<rzl@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.08-restore-ttl |
[production] |
| 16:33 |
<rzl@cumin1001> |
END (FAIL) - Cookbook sre.switchdc.mediawiki.08-restore-ttl (exit_code=99) |
[production] |
| 16:33 |
<rzl@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.08-restore-ttl |
[production] |
| 16:29 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) |
[production] |
| 16:29 |
<rzl@cumin1001> |
[DRY-RUN] MediaWiki read-only period ends at: 2020-08-28 16:29:24.432463 |
[production] |
| 16:29 |
<rzl@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.07-set-readwrite |
[production] |
| 16:29 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0) |
[production] |
| 16:29 |
<rzl@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite |
[production] |
| 16:29 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions (exit_code=0) |
[production] |
| 16:29 |
<rzl@cumin1001> |
START - Cookbook sre.switchdc.mediawiki.05-invert-redis-sessions |
[production] |
| 16:29 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0) |
[production] |