201-250 of 10000 results (19ms)
2021-06-22 §
08:49 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1166', diff saved to https://phabricator.wikimedia.org/P16680 and previous config saved to /var/cache/conftool/dbconfig/20210622-084915-marostegui.json [production]
08:35 <marostegui@cumin1001> dbctl commit (dc=all): 'db1169 (re)pooling @ 25%: Repool db1169 after schema change', diff saved to https://phabricator.wikimedia.org/P16679 and previous config saved to /var/cache/conftool/dbconfig/20210622-083545-root.json [production]
07:53 <joe> uploaded wmf-certificates package to buster-wikimedia/main, T284417 [production]
07:28 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1169 T283499', diff saved to https://phabricator.wikimedia.org/P16678 and previous config saved to /var/cache/conftool/dbconfig/20210622-072828-marostegui.json [production]
06:43 <dcausse> repool wdqs1005 [production]
06:16 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1100.eqiad.wmnet with reason: REIMAGE [production]
06:14 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on db1100.eqiad.wmnet with reason: REIMAGE [production]
05:06 <marostegui> Stop replication on old s5 master ( db1100) - T284529 [production]
05:06 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool old master running 10.1 T284529', diff saved to https://phabricator.wikimedia.org/P16677 and previous config saved to /var/cache/conftool/dbconfig/20210622-050602-marostegui.json [production]
05:01 <marostegui@cumin1001> dbctl commit (dc=all): 'Promote db1130 to s5 master and set section read-write T284529', diff saved to https://phabricator.wikimedia.org/P16676 and previous config saved to /var/cache/conftool/dbconfig/20210622-050123-root.json [production]
05:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Set s5 eqiad as read-only for maintenance - T284529', diff saved to https://phabricator.wikimedia.org/P16675 and previous config saved to /var/cache/conftool/dbconfig/20210622-050036-root.json [production]
05:00 <marostegui> Starting s5 eqiad failover from db1100 to db1130 - T284529 [production]
04:20 <marostegui> Start topology changes for s5 switchover T284529 [production]
04:19 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Master switchover s5 T284529 [production]
04:19 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 24 hosts with reason: Master switchover s5 T284529 [production]
04:11 <eileen> process-control config revision is 4ab72c1033 [production]
01:02 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ganeti2026.codfw.wmnet with reason: REIMAGE [production]
01:00 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2026.codfw.wmnet with reason: REIMAGE [production]
00:35 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ganeti2025.codfw.wmnet with reason: REIMAGE [production]
00:33 <pt1979@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2025.codfw.wmnet with reason: REIMAGE [production]
2021-06-21 §
23:16 <krinkle@deploy1002> Synchronized wmf-config/mc.php: I13646a5557c9 (duration: 00m 55s) [production]
23:12 <krinkle@deploy1002> Synchronized wmf-config/CommonSettings.php: I302a71d19604a45c54c (duration: 00m 56s) [production]
23:08 <krinkle@deploy1002> Synchronized wmf-config/CommonSettings.php: Idcac4d0345059f84 (duration: 00m 56s) [production]
23:05 <krinkle@deploy1002> Synchronized wmf-config/mc.php: I877a3ee1bbffb1e0 (duration: 00m 57s) [production]
23:04 <krinkle@deploy1002> Synchronized wmf-config/mc.php: Icc2676c23b808d (duration: 00m 56s) [production]
22:57 <krinkle@deploy1002> Synchronized wmf-config/mc.php: Iea94283c53 (duration: 00m 57s) [production]
22:53 <krinkle@deploy1002> Synchronized wmf-config/CommonSettings.php: Iea94283c53 (duration: 00m 57s) [production]
22:42 <eileen> civicrm revision changed from 0fca489063 to 629bd3b7b7, config revision is 2aed6ff89b [production]
22:41 <urbanecm> [urbanecm@mwmaint1002 ~]$ mwscript namespaceDupes.php --wiki=viwiki --fix # T284868 # P16674 [production]
22:13 <eileen> civicrm revision changed from acbcce94a2 to 0fca489063, config revision is 2aed6ff89b [production]
21:11 <sbassett> Deployed security patch for T285190 [production]
19:19 <sukhe@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:05:00 on doh1001.wikimedia.org with reason: temporarily depooling host [production]
19:19 <sukhe@cumin1001> START - Cookbook sre.hosts.downtime for 0:05:00 on doh1001.wikimedia.org with reason: temporarily depooling host [production]
18:41 <ppchelko@deploy1002> Synchronized wmf-config/wikitech.php: Replace uses of AbstractBlock::getTarget() T284141 (duration: 00m 58s) [production]
18:30 <urbanecm@deploy1002> Synchronized wmf-config/PoolCounterSettings.php: af61f1a9ffc4703ad9514eac96655c3dd9d491d8: Add pool counter for automated search requests (T284479) (duration: 00m 59s) [production]
18:30 <ebernhardson@deploy1002> Finished deploy [wikimedia/discovery/analytics@40b4b2f]: T273854 Airflow dag to extract and process sparql queries (duration: 07m 11s) [production]
18:29 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: f7db2b94d62c01cd62ec88d25675d3349d9c1b98: Enable wikilove on hewikisource (T284864) (duration: 00m 56s) [production]
18:26 <brennen> gitlab1001: running ansible for copying latest backup to dedicated folder (T274463) [production]
18:24 <urbanecm> [urbanecm@mwmaint1002 ~]$ mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=hewikisource wikilove # T284864 [production]
18:23 <urbanecm> Correction: [urbanecm@mwmaint1002 ~]$ mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=hiwikisource wikilove # T284864 [production]
18:23 <urbanecm> [urbanecm@mwmaint1002 ~]$ mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=hiwikisource # T284864 [production]
18:22 <ebernhardson@deploy1002> Started deploy [wikimedia/discovery/analytics@40b4b2f]: T273854 Airflow dag to extract and process sparql queries [production]
18:22 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: dd0fecb89da39611cdf18584626508741cf369e9: Rename Portal and Portal talk namespaces on viwiki (T284868) (duration: 00m 56s) [production]
18:16 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 5d8b9df1d6c77cae8fa18c343e5f1ec139d603b0: Disable Education Program namespaces in enwiki (T285193) (duration: 00m 58s) [production]
18:14 <urbanecm@deploy1002> Synchronized wmf-config/abusefilter.php: 5a51dd2540cd620cec60b9921b57c8ab185df029: Add `managechangetags` to the `abusefilter` group on eswiki (T285167) (duration: 00m 56s) [production]
18:11 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 219dd5b79d59b98d13e34f7d6e22a41ae1dd9cbb: eswiki AbuseFilter config changes (T284797; 2/2) (duration: 00m 56s) [production]
18:10 <urbanecm@deploy1002> Synchronized wmf-config/abusefilter.php: 219dd5b79d59b98d13e34f7d6e22a41ae1dd9cbb: eswiki AbuseFilter config changes (T284797; 1/2) (duration: 01m 07s) [production]
17:40 <ebernhardson> post-deploy restart airflow-webserver and airflow-scheduler on an-airflow1001 [production]
17:32 <ebernhardson@deploy1002> Finished deploy [wikimedia/discovery/analytics@2337592]: airflow: expect eventgate canary events in all dcs (duration: 04m 24s) [production]
17:27 <ebernhardson@deploy1002> Started deploy [wikimedia/discovery/analytics@2337592]: airflow: expect eventgate canary events in all dcs [production]