|
2026-03-02
§
|
| 08:07 |
<kgraessle@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1240672|Enable revert risk filters for first batch of wikis: < 1000 monthly edits (T411485)]] |
[production] |
| 08:05 |
<fabfur@cumin1003> |
START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_magru and A:cp - 3.0 upgrade () |
[production] |
| 08:05 |
<fabfur> |
start upgrading haproxy to 3.0 on A:cp-upload_magru (T417253) |
[production] |
| 07:52 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P89449 and previous config saved to /var/cache/conftool/dbconfig/20260302-075252-marostegui.json |
[production] |
| 07:52 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P89448 and previous config saved to /var/cache/conftool/dbconfig/20260302-075241-marostegui.json |
[production] |
| 07:37 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1172 (T418465)', diff saved to https://phabricator.wikimedia.org/P89447 and previous config saved to /var/cache/conftool/dbconfig/20260302-073745-marostegui.json |
[production] |
| 07:37 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P89446 and previous config saved to /var/cache/conftool/dbconfig/20260302-073732-marostegui.json |
[production] |
| 07:22 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2154 (T418465)', diff saved to https://phabricator.wikimedia.org/P89445 and previous config saved to /var/cache/conftool/dbconfig/20260302-072224-marostegui.json |
[production] |
| 07:20 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db1172 (T418465)', diff saved to https://phabricator.wikimedia.org/P89444 and previous config saved to /var/cache/conftool/dbconfig/20260302-072058-marostegui.json |
[production] |
| 07:20 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1172.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:05 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1171.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:05 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1167 (T418465)', diff saved to https://phabricator.wikimedia.org/P89443 and previous config saved to /var/cache/conftool/dbconfig/20260302-070523-marostegui.json |
[production] |
| 07:05 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2154 (T418465)', diff saved to https://phabricator.wikimedia.org/P89442 and previous config saved to /var/cache/conftool/dbconfig/20260302-070512-marostegui.json |
[production] |
| 07:05 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2154.codfw.wmnet with reason: Maintenance |
[production] |
| 07:04 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2152 (T418465)', diff saved to https://phabricator.wikimedia.org/P89441 and previous config saved to /var/cache/conftool/dbconfig/20260302-070447-marostegui.json |
[production] |
| 07:01 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1244: After schema change |
[production] |
| 06:50 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P89439 and previous config saved to /var/cache/conftool/dbconfig/20260302-065014-marostegui.json |
[production] |
| 06:49 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2152', diff saved to https://phabricator.wikimedia.org/P89438 and previous config saved to /var/cache/conftool/dbconfig/20260302-064938-marostegui.json |
[production] |
| 06:35 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P89436 and previous config saved to /var/cache/conftool/dbconfig/20260302-063506-marostegui.json |
[production] |
| 06:34 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2152', diff saved to https://phabricator.wikimedia.org/P89435 and previous config saved to /var/cache/conftool/dbconfig/20260302-063430-marostegui.json |
[production] |
| 06:19 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1167 (T418465)', diff saved to https://phabricator.wikimedia.org/P89433 and previous config saved to /var/cache/conftool/dbconfig/20260302-061957-marostegui.json |
[production] |
| 06:19 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2152 (T418465)', diff saved to https://phabricator.wikimedia.org/P89432 and previous config saved to /var/cache/conftool/dbconfig/20260302-061922-marostegui.json |
[production] |
| 06:17 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2240.codfw.wmnet with reason: Maintenance |
[production] |
| 06:16 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.pool pool db1244: After schema change |
[production] |
| 06:15 |
<marostegui@dns1004> |
END - running authdns-update |
[production] |
| 06:14 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depool db2240 T418080', diff saved to https://phabricator.wikimedia.org/P89430 and previous config saved to /var/cache/conftool/dbconfig/20260302-061428-marostegui.json |
[production] |
| 06:13 |
<marostegui@dns1004> |
START - running authdns-update |
[production] |
| 06:13 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Promote db2179 to s4 primary and set section read-write T418080', diff saved to https://phabricator.wikimedia.org/P89429 and previous config saved to /var/cache/conftool/dbconfig/20260302-061316-marostegui.json |
[production] |
| 06:12 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Set s4 codfw as read-only for maintenance - T418080', diff saved to https://phabricator.wikimedia.org/P89428 and previous config saved to /var/cache/conftool/dbconfig/20260302-061252-marostegui.json |
[production] |
| 06:06 |
<marostegui> |
Starting s4 codfw failover from db2240 to db2179 - T418080 |
[production] |
| 06:03 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 42 hosts with reason: Primary switchover s4 T418080 |
[production] |
| 06:03 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Set db2179 with weight 0 T418080', diff saved to https://phabricator.wikimedia.org/P89427 and previous config saved to /var/cache/conftool/dbconfig/20260302-060317-marostegui.json |
[production] |
| 06:03 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db1167 (T418465)', diff saved to https://phabricator.wikimedia.org/P89426 and previous config saved to /var/cache/conftool/dbconfig/20260302-060317-marostegui.json |
[production] |
| 06:03 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:02 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1167.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:02 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2152 (T418465)', diff saved to https://phabricator.wikimedia.org/P89425 and previous config saved to /var/cache/conftool/dbconfig/20260302-060245-marostegui.json |
[production] |
| 06:02 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2152.codfw.wmnet with reason: Maintenance |
[production] |
| 05:58 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1181.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:58 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2220.codfw.wmnet with reason: Maintenance |
[production] |
| 02:14 |
<mwpresync@deploy2002> |
Finished scap build-images: Publishing wmf/next image (duration: 13m 13s) |
[production] |
| 02:00 |
<mwpresync@deploy2002> |
Started scap build-images: Publishing wmf/next image |
[production] |
| 00:50 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance |
[production] |
| 00:49 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1253 (T418465)', diff saved to https://phabricator.wikimedia.org/P89424 and previous config saved to /var/cache/conftool/dbconfig/20260302-004950-marostegui.json |
[production] |
| 00:34 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P89423 and previous config saved to /var/cache/conftool/dbconfig/20260302-003441-marostegui.json |
[production] |
| 00:19 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P89422 and previous config saved to /var/cache/conftool/dbconfig/20260302-001933-marostegui.json |
[production] |
| 00:04 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1253 (T418465)', diff saved to https://phabricator.wikimedia.org/P89421 and previous config saved to /var/cache/conftool/dbconfig/20260302-000425-marostegui.json |
[production] |
| 00:02 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db1253 (T418465)', diff saved to https://phabricator.wikimedia.org/P89420 and previous config saved to /var/cache/conftool/dbconfig/20260302-000208-marostegui.json |
[production] |
| 00:02 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1253.eqiad.wmnet with reason: Maintenance |
[production] |
| 00:01 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1236 (T418465)', diff saved to https://phabricator.wikimedia.org/P89419 and previous config saved to /var/cache/conftool/dbconfig/20260302-000143-marostegui.json |
[production] |