|
2020-08-31
ยง
|
| 14:58 |
<ema> |
Traffic: depool eqiad from user traffic T243316 |
[production] |
| 14:38 |
<moritzm> |
installing rake security updates on stretch |
[production] |
| 14:33 |
<rzl@cumin1001> |
conftool action : set/pooled=true; selector: dnsdisc=kartotherian,name=eqiad |
[production] |
| 14:21 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.switchdc.services.01-switch-dc (exit_code=0) |
[production] |
| 14:20 |
<rzl@cumin1001> |
Switching services apertium, termbox, search, api-gateway, ores, sessionstore, eventgate-main, graphoid, eventstreams, wikifeeds, wdqs, parsoid, eventgate-logging-external, wdqs-internal, echostore, mathoid, mobileapps, proton, restbase, kartotherian, recommendation-api, eventgate-analytics-external, restbase-async, citoid, schema, cxserver, eventgate-analytics, zotero: eqiad => codfw |
[production] |
| 14:20 |
<rzl@cumin1001> |
START - Cookbook sre.switchdc.services.01-switch-dc |
[production] |
| 14:18 |
<rzl@cumin1001> |
END (PASS) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=0) |
[production] |
| 14:13 |
<rzl@cumin1001> |
START - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep |
[production] |
| 14:12 |
<rzl@cumin1001> |
END (FAIL) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=99) |
[production] |
| 14:11 |
<rzl@cumin1001> |
START - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep |
[production] |
| 13:41 |
<andrewbogott> |
dropping many databases from m5, as per T261152 |
[production] |
| 13:15 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:15 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:07 |
<marostegui> |
Failover m3 (phabricator) proxy from dbproxy1016 to dbproxy1020 - T261459 |
[production] |
| 13:05 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:02 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:54 |
<oblivian@cumin2001> |
END (PASS) - Cookbook sre.switchdc.services.02-restore-ttl (exit_code=0) |
[production] |
| 12:54 |
<oblivian@cumin2001> |
START - Cookbook sre.switchdc.services.02-restore-ttl |
[production] |
| 12:53 |
<oblivian@cumin2001> |
END (PASS) - Cookbook sre.switchdc.services.01-switch-dc (exit_code=0) |
[production] |
| 12:53 |
<oblivian@cumin2001> |
Switching services parsoid: eqiad => codfw |
[production] |
| 12:53 |
<oblivian@cumin2001> |
START - Cookbook sre.switchdc.services.01-switch-dc |
[production] |
| 12:53 |
<oblivian@cumin2001> |
END (PASS) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=0) |
[production] |
| 12:47 |
<oblivian@cumin2001> |
START - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep |
[production] |
| 12:45 |
<oblivian@cumin2001> |
END (PASS) - Cookbook sre.switchdc.services.02-restore-ttl (exit_code=0) |
[production] |
| 12:45 |
<oblivian@cumin2001> |
START - Cookbook sre.switchdc.services.02-restore-ttl |
[production] |
| 12:44 |
<oblivian@cumin2001> |
END (PASS) - Cookbook sre.switchdc.services.01-switch-dc (exit_code=0) |
[production] |
| 12:44 |
<oblivian@cumin2001> |
Switching services restbase-async: eqiad => codfw |
[production] |
| 12:44 |
<oblivian@cumin2001> |
START - Cookbook sre.switchdc.services.01-switch-dc |
[production] |
| 12:43 |
<oblivian@cumin2001> |
END (PASS) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=0) |
[production] |
| 12:37 |
<oblivian@cumin2001> |
START - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep |
[production] |
| 12:14 |
<oblivian@cumin2001> |
END (PASS) - Cookbook sre.switchdc.services.02-restore-ttl (exit_code=0) |
[production] |
| 12:14 |
<oblivian@cumin2001> |
START - Cookbook sre.switchdc.services.02-restore-ttl |
[production] |
| 12:13 |
<oblivian@cumin2001> |
END (PASS) - Cookbook sre.switchdc.services.01-switch-dc (exit_code=0) |
[production] |
| 12:13 |
<oblivian@cumin2001> |
Switching services restbase-async: eqiad => codfw |
[production] |
| 12:13 |
<oblivian@cumin2001> |
START - Cookbook sre.switchdc.services.01-switch-dc |
[production] |
| 12:10 |
<oblivian@cumin2001> |
END (PASS) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=0) |
[production] |
| 12:05 |
<oblivian@cumin2001> |
START - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep |
[production] |
| 11:58 |
<elukey> |
reimage kafka-jumbo1001 to Buster |
[production] |
| 11:32 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.6/extensions/WikimediaEvents/modules/ext.wikimediaEvents/searchSatisfaction.js: 5d583d9550787a8e36c29ca841233615405fcb7e: Disable MediaSearch A/B test (duration: 00m 55s) |
[production] |
| 11:22 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 81f88fde2aad23a619047b1177a6188f51df11a9: Enable Signature button on Wikiproject for hywiki (T261550) (duration: 00m 54s) |
[production] |
| 11:22 |
<jbond42> |
removing old hiera version 1 and 3 backends |
[production] |
| 11:17 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: b74893fecdaae599077daad5b1219ad3b9bc7fc9: Enable sitenotice on mobile for closed wikis (T261357) (duration: 00m 56s) |
[production] |
| 11:02 |
<volans> |
upgraded spicerack to 0.0.41 on cumin hosts |
[production] |
| 10:27 |
<hnowlan@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . |
[production] |
| 09:51 |
<elukey> |
executed /srv/phab/phabricator/bin/remove destroy @klausman on phab1001 (following https://wikitech.wikimedia.org/wiki/Phabricator#Delete_a_user) to clear incosistent state of new account (wrong email address) |
[production] |
| 08:43 |
<moritzm> |
installing bind9 security updates on stretch/buster (client-side tools/libs only) |
[production] |
| 07:53 |
<volans> |
uploaded spicerack_0.0.41 to apt.wikimedia.org buster-wikimedia |
[production] |
| 07:30 |
<moritzm> |
installing squid security updates |
[production] |
| 07:24 |
<moritzm> |
installing openexr security updates on buster |
[production] |
| 07:12 |
<marostegui> |
Sanitize jawikivoyage on db2094:3325 and db1124:3325 T260482 |
[production] |