3051-3100 of 10000 results (29ms)
2020-08-31 §
14:11 <rzl@cumin1001> START - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep [production]
13:41 <andrewbogott> dropping many databases from m5, as per T261152 [production]
13:15 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:15 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
13:07 <marostegui> Failover m3 (phabricator) proxy from dbproxy1016 to dbproxy1020 - T261459 [production]
13:05 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:02 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
12:54 <oblivian@cumin2001> END (PASS) - Cookbook sre.switchdc.services.02-restore-ttl (exit_code=0) [production]
12:54 <oblivian@cumin2001> START - Cookbook sre.switchdc.services.02-restore-ttl [production]
12:53 <oblivian@cumin2001> END (PASS) - Cookbook sre.switchdc.services.01-switch-dc (exit_code=0) [production]
12:53 <oblivian@cumin2001> Switching services parsoid: eqiad => codfw [production]
12:53 <oblivian@cumin2001> START - Cookbook sre.switchdc.services.01-switch-dc [production]
12:53 <oblivian@cumin2001> END (PASS) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=0) [production]
12:47 <oblivian@cumin2001> START - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep [production]
12:45 <oblivian@cumin2001> END (PASS) - Cookbook sre.switchdc.services.02-restore-ttl (exit_code=0) [production]
12:45 <oblivian@cumin2001> START - Cookbook sre.switchdc.services.02-restore-ttl [production]
12:44 <oblivian@cumin2001> END (PASS) - Cookbook sre.switchdc.services.01-switch-dc (exit_code=0) [production]
12:44 <oblivian@cumin2001> Switching services restbase-async: eqiad => codfw [production]
12:44 <oblivian@cumin2001> START - Cookbook sre.switchdc.services.01-switch-dc [production]
12:43 <oblivian@cumin2001> END (PASS) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=0) [production]
12:37 <oblivian@cumin2001> START - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep [production]
12:14 <oblivian@cumin2001> END (PASS) - Cookbook sre.switchdc.services.02-restore-ttl (exit_code=0) [production]
12:14 <oblivian@cumin2001> START - Cookbook sre.switchdc.services.02-restore-ttl [production]
12:13 <oblivian@cumin2001> END (PASS) - Cookbook sre.switchdc.services.01-switch-dc (exit_code=0) [production]
12:13 <oblivian@cumin2001> Switching services restbase-async: eqiad => codfw [production]
12:13 <oblivian@cumin2001> START - Cookbook sre.switchdc.services.01-switch-dc [production]
12:10 <oblivian@cumin2001> END (PASS) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=0) [production]
12:05 <oblivian@cumin2001> START - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep [production]
11:58 <elukey> reimage kafka-jumbo1001 to Buster [production]
11:32 <urbanecm@deploy1001> Synchronized php-1.36.0-wmf.6/extensions/WikimediaEvents/modules/ext.wikimediaEvents/searchSatisfaction.js: 5d583d9550787a8e36c29ca841233615405fcb7e: Disable MediaSearch A/B test (duration: 00m 55s) [production]
11:22 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 81f88fde2aad23a619047b1177a6188f51df11a9: Enable Signature button on Wikiproject for hywiki (T261550) (duration: 00m 54s) [production]
11:22 <jbond42> removing old hiera version 1 and 3 backends [production]
11:17 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: b74893fecdaae599077daad5b1219ad3b9bc7fc9: Enable sitenotice on mobile for closed wikis (T261357) (duration: 00m 56s) [production]
11:02 <volans> upgraded spicerack to 0.0.41 on cumin hosts [production]
10:27 <hnowlan@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
09:51 <elukey> executed /srv/phab/phabricator/bin/remove destroy @klausman on phab1001 (following https://wikitech.wikimedia.org/wiki/Phabricator#Delete_a_user) to clear incosistent state of new account (wrong email address) [production]
08:43 <moritzm> installing bind9 security updates on stretch/buster (client-side tools/libs only) [production]
07:53 <volans> uploaded spicerack_0.0.41 to apt.wikimedia.org buster-wikimedia [production]
07:30 <moritzm> installing squid security updates [production]
07:24 <moritzm> installing openexr security updates on buster [production]
07:12 <marostegui> Sanitize jawikivoyage on db2094:3325 and db1124:3325 T260482 [production]
06:30 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
06:27 <elukey@cumin1001> START - Cookbook sre.hosts.downtime [production]
06:06 <elukey> reimage kafka-jumbo1005 to Debian Buster [production]
05:21 <marostegui> Reload haproxy on dbproxy1017 and dbproxy1021 to test db1128 [production]
2020-08-30 §
16:13 <herron> restarted eqiad v5 logstashes [production]
2020-08-29 §
18:05 <Amir1> end of ladsgroup@mwmaint1002:~$ foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https (T261451) [production]
17:45 <Amir1> start of ladsgroup@mwmaint1002:~$ foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https (T261451) [production]
2020-08-28 §
21:53 <ryankemper> `sudo systemctl reload nginx.service` on `cloudelastic100[5,6].wikimedia.org` to try to resolve certificate warning issues [production]
19:11 <andrewbogott> rebooting cloudvirt1006. It's a spare, unused system but showing a bus error and icinga alerts; not worth saving if it needs saving [production]