|
2026-02-04
ยง
|
| 10:44 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host cloudcumin2001.codfw.wmnet |
[production] |
| 10:40 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2216 (T415786)', diff saved to https://phabricator.wikimedia.org/P88639 and previous config saved to /var/cache/conftool/dbconfig/20260204-104035-marostegui.json |
[production] |
| 10:39 |
<moritzm> |
upgrade cloudcumin2001 to bookworm T403153 |
[production] |
| 10:36 |
<hashar@deploy2002> |
rebuilt and synchronized wikiversions files: group0 to 1.46.0-wmf.14 refs T413805 |
[production] |
| 10:29 |
<hashar> |
Rolling back to group0 due to an issue with OAuth on metawiki # T413805 |
[production] |
| 10:25 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P88638 and previous config saved to /var/cache/conftool/dbconfig/20260204-102527-marostegui.json |
[production] |
| 10:11 |
<hashar> |
Restarted CI Jenkins |
[production] |
| 10:10 |
<hashar> |
Gerrit is back |
[production] |
| 10:10 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P88637 and previous config saved to /var/cache/conftool/dbconfig/20260204-101018-marostegui.json |
[production] |
| 10:06 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2150 (T415786)', diff saved to https://phabricator.wikimedia.org/P88636 and previous config saved to /var/cache/conftool/dbconfig/20260204-100638-marostegui.json |
[production] |
| 10:06 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2150.codfw.wmnet with reason: Maintenance |
[production] |
| 10:06 |
<hashar> |
Restarting Gerrit instances |
[production] |
| 09:55 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2216 (T415786)', diff saved to https://phabricator.wikimedia.org/P88635 and previous config saved to /var/cache/conftool/dbconfig/20260204-095510-marostegui.json |
[production] |
| 09:38 |
<moritzm> |
installing openssl security updates |
[production] |
| 09:37 |
<hashar@deploy2002> |
rebuilt and synchronized wikiversions files: group1 to 1.46.0-wmf.14 refs T413805 |
[production] |
| 09:34 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db1251 (T415786)', diff saved to https://phabricator.wikimedia.org/P88634 and previous config saved to /var/cache/conftool/dbconfig/20260204-093421-marostegui.json |
[production] |
| 09:34 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1251.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:10 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:10 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1253 (T415786)', diff saved to https://phabricator.wikimedia.org/P88632 and previous config saved to /var/cache/conftool/dbconfig/20260204-091015-marostegui.json |
[production] |
| 08:55 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P88631 and previous config saved to /var/cache/conftool/dbconfig/20260204-085506-marostegui.json |
[production] |
| 08:39 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P88630 and previous config saved to /var/cache/conftool/dbconfig/20260204-083958-marostegui.json |
[production] |
| 08:24 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1253 (T415786)', diff saved to https://phabricator.wikimedia.org/P88629 and previous config saved to /var/cache/conftool/dbconfig/20260204-082450-marostegui.json |
[production] |
| 08:23 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2216 (T415786)', diff saved to https://phabricator.wikimedia.org/P88628 and previous config saved to /var/cache/conftool/dbconfig/20260204-082324-marostegui.json |
[production] |
| 08:23 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2216.codfw.wmnet with reason: Maintenance |
[production] |
| 08:23 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2212 (T415786)', diff saved to https://phabricator.wikimedia.org/P88627 and previous config saved to /var/cache/conftool/dbconfig/20260204-082259-marostegui.json |
[production] |
| 08:08 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1240.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:07 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2212', diff saved to https://phabricator.wikimedia.org/P88626 and previous config saved to /var/cache/conftool/dbconfig/20260204-080751-marostegui.json |
[production] |
| 07:52 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2212', diff saved to https://phabricator.wikimedia.org/P88625 and previous config saved to /var/cache/conftool/dbconfig/20260204-075243-marostegui.json |
[production] |
| 07:39 |
<ryankemper@cumin2002> |
END (FAIL) - Cookbook sre.hadoop.reboot-workers (exit_code=99) for Hadoop analytics cluster |
[production] |
| 07:37 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2212 (T415786)', diff saved to https://phabricator.wikimedia.org/P88624 and previous config saved to /var/cache/conftool/dbconfig/20260204-073735-marostegui.json |
[production] |
| 07:35 |
<marostegui> |
Deploy schema change on db2204 (old s2 codfw master) T415786 |
[production] |
| 07:34 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2204.codfw.wmnet with reason: Schema change |
[production] |
| 07:26 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db1253 (T415786)', diff saved to https://phabricator.wikimedia.org/P88623 and previous config saved to /var/cache/conftool/dbconfig/20260204-072658-marostegui.json |
[production] |
| 07:26 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1253.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:26 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1231 (T415786)', diff saved to https://phabricator.wikimedia.org/P88622 and previous config saved to /var/cache/conftool/dbconfig/20260204-072632-marostegui.json |
[production] |
| 07:11 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P88621 and previous config saved to /var/cache/conftool/dbconfig/20260204-071124-marostegui.json |
[production] |
| 06:56 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P88620 and previous config saved to /var/cache/conftool/dbconfig/20260204-065616-marostegui.json |
[production] |
| 06:41 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1239.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:41 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1235 (T415786)', diff saved to https://phabricator.wikimedia.org/P88619 and previous config saved to /var/cache/conftool/dbconfig/20260204-064118-marostegui.json |
[production] |
| 06:41 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1231 (T415786)', diff saved to https://phabricator.wikimedia.org/P88618 and previous config saved to /var/cache/conftool/dbconfig/20260204-064107-marostegui.json |
[production] |
| 06:31 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P88617 and previous config saved to /var/cache/conftool/dbconfig/20260204-063103-marostegui.json |
[production] |
| 06:20 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P88616 and previous config saved to /var/cache/conftool/dbconfig/20260204-062055-marostegui.json |
[production] |
| 06:18 |
<marostegui@dns1006> |
END - running authdns-update |
[production] |
| 06:17 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depool db2204 T416300', diff saved to https://phabricator.wikimedia.org/P88615 and previous config saved to /var/cache/conftool/dbconfig/20260204-061739-marostegui.json |
[production] |
| 06:17 |
<marostegui@dns1006> |
START - running authdns-update |
[production] |
| 06:16 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Promote db2207 to s2 primary and set section read-write T416300', diff saved to https://phabricator.wikimedia.org/P88614 and previous config saved to /var/cache/conftool/dbconfig/20260204-061637-marostegui.json |
[production] |
| 06:16 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Set s2 codfw as read-only for maintenance - T416300', diff saved to https://phabricator.wikimedia.org/P88613 and previous config saved to /var/cache/conftool/dbconfig/20260204-061613-marostegui.json |
[production] |
| 06:12 |
<marostegui> |
Starting s2 codfw failover from db2204 to db2207 - T416300 |
[production] |
| 06:11 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s2 T416300 |
[production] |
| 06:11 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Set db2207 with weight 0 T416300', diff saved to https://phabricator.wikimedia.org/P88612 and previous config saved to /var/cache/conftool/dbconfig/20260204-061122-marostegui.json |
[production] |