|
2021-06-22
§
|
| 06:14 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on db1100.eqiad.wmnet with reason: REIMAGE |
[production] |
| 05:06 |
<marostegui> |
Stop replication on old s5 master ( db1100) - T284529 |
[production] |
| 05:06 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool old master running 10.1 T284529', diff saved to https://phabricator.wikimedia.org/P16677 and previous config saved to /var/cache/conftool/dbconfig/20210622-050602-marostegui.json |
[production] |
| 05:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Promote db1130 to s5 master and set section read-write T284529', diff saved to https://phabricator.wikimedia.org/P16676 and previous config saved to /var/cache/conftool/dbconfig/20210622-050123-root.json |
[production] |
| 05:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Set s5 eqiad as read-only for maintenance - T284529', diff saved to https://phabricator.wikimedia.org/P16675 and previous config saved to /var/cache/conftool/dbconfig/20210622-050036-root.json |
[production] |
| 05:00 |
<marostegui> |
Starting s5 eqiad failover from db1100 to db1130 - T284529 |
[production] |
| 04:20 |
<marostegui> |
Start topology changes for s5 switchover T284529 |
[production] |
| 04:19 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Master switchover s5 T284529 |
[production] |
| 04:19 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 24 hosts with reason: Master switchover s5 T284529 |
[production] |
| 04:11 |
<eileen> |
process-control config revision is 4ab72c1033 |
[production] |
| 01:02 |
<pt1979@cumin2002> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ganeti2026.codfw.wmnet with reason: REIMAGE |
[production] |
| 01:00 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2026.codfw.wmnet with reason: REIMAGE |
[production] |
| 00:35 |
<pt1979@cumin2002> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on ganeti2025.codfw.wmnet with reason: REIMAGE |
[production] |
| 00:33 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti2025.codfw.wmnet with reason: REIMAGE |
[production] |
|
2021-06-21
§
|
| 23:16 |
<krinkle@deploy1002> |
Synchronized wmf-config/mc.php: I13646a5557c9 (duration: 00m 55s) |
[production] |
| 23:12 |
<krinkle@deploy1002> |
Synchronized wmf-config/CommonSettings.php: I302a71d19604a45c54c (duration: 00m 56s) |
[production] |
| 23:08 |
<krinkle@deploy1002> |
Synchronized wmf-config/CommonSettings.php: Idcac4d0345059f84 (duration: 00m 56s) |
[production] |
| 23:05 |
<krinkle@deploy1002> |
Synchronized wmf-config/mc.php: I877a3ee1bbffb1e0 (duration: 00m 57s) |
[production] |
| 23:04 |
<krinkle@deploy1002> |
Synchronized wmf-config/mc.php: Icc2676c23b808d (duration: 00m 56s) |
[production] |
| 22:57 |
<krinkle@deploy1002> |
Synchronized wmf-config/mc.php: Iea94283c53 (duration: 00m 57s) |
[production] |
| 22:53 |
<krinkle@deploy1002> |
Synchronized wmf-config/CommonSettings.php: Iea94283c53 (duration: 00m 57s) |
[production] |
| 22:42 |
<eileen> |
civicrm revision changed from 0fca489063 to 629bd3b7b7, config revision is 2aed6ff89b |
[production] |
| 22:41 |
<urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript namespaceDupes.php --wiki=viwiki --fix # T284868 # P16674 |
[production] |
| 22:13 |
<eileen> |
civicrm revision changed from acbcce94a2 to 0fca489063, config revision is 2aed6ff89b |
[production] |
| 21:11 |
<sbassett> |
Deployed security patch for T285190 |
[production] |
| 19:19 |
<sukhe@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:05:00 on doh1001.wikimedia.org with reason: temporarily depooling host |
[production] |
| 19:19 |
<sukhe@cumin1001> |
START - Cookbook sre.hosts.downtime for 0:05:00 on doh1001.wikimedia.org with reason: temporarily depooling host |
[production] |
| 18:41 |
<ppchelko@deploy1002> |
Synchronized wmf-config/wikitech.php: Replace uses of AbstractBlock::getTarget() T284141 (duration: 00m 58s) |
[production] |
| 18:30 |
<urbanecm@deploy1002> |
Synchronized wmf-config/PoolCounterSettings.php: af61f1a9ffc4703ad9514eac96655c3dd9d491d8: Add pool counter for automated search requests (T284479) (duration: 00m 59s) |
[production] |
| 18:30 |
<ebernhardson@deploy1002> |
Finished deploy [wikimedia/discovery/analytics@40b4b2f]: T273854 Airflow dag to extract and process sparql queries (duration: 07m 11s) |
[production] |
| 18:29 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: f7db2b94d62c01cd62ec88d25675d3349d9c1b98: Enable wikilove on hewikisource (T284864) (duration: 00m 56s) |
[production] |
| 18:26 |
<brennen> |
gitlab1001: running ansible for copying latest backup to dedicated folder (T274463) |
[production] |
| 18:24 |
<urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=hewikisource wikilove # T284864 |
[production] |
| 18:23 |
<urbanecm> |
Correction: [urbanecm@mwmaint1002 ~]$ mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=hiwikisource wikilove # T284864 |
[production] |
| 18:23 |
<urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=hiwikisource # T284864 |
[production] |
| 18:22 |
<ebernhardson@deploy1002> |
Started deploy [wikimedia/discovery/analytics@40b4b2f]: T273854 Airflow dag to extract and process sparql queries |
[production] |
| 18:22 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: dd0fecb89da39611cdf18584626508741cf369e9: Rename Portal and Portal talk namespaces on viwiki (T284868) (duration: 00m 56s) |
[production] |
| 18:16 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 5d8b9df1d6c77cae8fa18c343e5f1ec139d603b0: Disable Education Program namespaces in enwiki (T285193) (duration: 00m 58s) |
[production] |
| 18:14 |
<urbanecm@deploy1002> |
Synchronized wmf-config/abusefilter.php: 5a51dd2540cd620cec60b9921b57c8ab185df029: Add `managechangetags` to the `abusefilter` group on eswiki (T285167) (duration: 00m 56s) |
[production] |
| 18:11 |
<urbanecm@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: 219dd5b79d59b98d13e34f7d6e22a41ae1dd9cbb: eswiki AbuseFilter config changes (T284797; 2/2) (duration: 00m 56s) |
[production] |
| 18:10 |
<urbanecm@deploy1002> |
Synchronized wmf-config/abusefilter.php: 219dd5b79d59b98d13e34f7d6e22a41ae1dd9cbb: eswiki AbuseFilter config changes (T284797; 1/2) (duration: 01m 07s) |
[production] |
| 17:40 |
<ebernhardson> |
post-deploy restart airflow-webserver and airflow-scheduler on an-airflow1001 |
[production] |
| 17:32 |
<ebernhardson@deploy1002> |
Finished deploy [wikimedia/discovery/analytics@2337592]: airflow: expect eventgate canary events in all dcs (duration: 04m 24s) |
[production] |
| 17:27 |
<ebernhardson@deploy1002> |
Started deploy [wikimedia/discovery/analytics@2337592]: airflow: expect eventgate canary events in all dcs |
[production] |
| 16:47 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 16:32 |
<pt1979@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 15:46 |
<papaul> |
poweroff elastic2043 for maintenance |
[production] |
| 15:25 |
<hashar> |
Updated operations-puppet-tests-buster-docker Jenkins job to use latest Docker image https://gerrit.wikimedia.org/r/c/integration/config/+/700648 |
[production] |
| 15:06 |
<hnowlan@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host maps1009.eqiad.wmnet |
[production] |
| 15:02 |
<klausman@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve1004.eqiad.wmnet |
[production] |