|
2025-07-28
ยง
|
| 11:22 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2211 (T399728)', diff saved to https://phabricator.wikimedia.org/P80085 and previous config saved to /var/cache/conftool/dbconfig/20250728-112254-fceratto.json |
[production] |
| 11:19 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db2211 (T399728)', diff saved to https://phabricator.wikimedia.org/P80084 and previous config saved to /var/cache/conftool/dbconfig/20250728-111958-fceratto.json |
[production] |
| 11:19 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2211.codfw.wmnet with reason: Maintenance |
[production] |
| 11:17 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2201.codfw.wmnet with reason: Maintenance |
[production] |
| 11:17 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2192 (T399728)', diff saved to https://phabricator.wikimedia.org/P80083 and previous config saved to /var/cache/conftool/dbconfig/20250728-111730-fceratto.json |
[production] |
| 11:15 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P80082 and previous config saved to /var/cache/conftool/dbconfig/20250728-111524-marostegui.json |
[production] |
| 11:15 |
<elukey@cumin1003> |
START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 11:11 |
<elukey@cumin1003> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 11:02 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P80081 and previous config saved to /var/cache/conftool/dbconfig/20250728-110222-fceratto.json |
[production] |
| 11:00 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P80080 and previous config saved to /var/cache/conftool/dbconfig/20250728-110016-marostegui.json |
[production] |
| 10:53 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2162 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P80079 and previous config saved to /var/cache/conftool/dbconfig/20250728-105323-root.json |
[production] |
| 10:52 |
<elukey@cumin1003> |
START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 10:47 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P80078 and previous config saved to /var/cache/conftool/dbconfig/20250728-104715-fceratto.json |
[production] |
| 10:45 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1203 (T399249)', diff saved to https://phabricator.wikimedia.org/P80077 and previous config saved to /var/cache/conftool/dbconfig/20250728-104508-marostegui.json |
[production] |
| 10:44 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1203 (T399249)', diff saved to https://phabricator.wikimedia.org/P80076 and previous config saved to /var/cache/conftool/dbconfig/20250728-104400-marostegui.json |
[production] |
| 10:43 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1203.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:43 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1192 (T399249)', diff saved to https://phabricator.wikimedia.org/P80075 and previous config saved to /var/cache/conftool/dbconfig/20250728-104337-marostegui.json |
[production] |
| 10:41 |
<elukey@cumin1003> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 10:40 |
<elukey@cumin1003> |
START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 10:39 |
<elukey@cumin1003> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 10:38 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2162 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P80074 and previous config saved to /var/cache/conftool/dbconfig/20250728-103817-root.json |
[production] |
| 10:32 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2192 (T399728)', diff saved to https://phabricator.wikimedia.org/P80073 and previous config saved to /var/cache/conftool/dbconfig/20250728-103208-fceratto.json |
[production] |
| 10:29 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db2192 (T399728)', diff saved to https://phabricator.wikimedia.org/P80072 and previous config saved to /var/cache/conftool/dbconfig/20250728-102918-fceratto.json |
[production] |
| 10:29 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2192.codfw.wmnet with reason: Maintenance |
[production] |
| 10:28 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2178 (T399728)', diff saved to https://phabricator.wikimedia.org/P80071 and previous config saved to /var/cache/conftool/dbconfig/20250728-102856-fceratto.json |
[production] |
| 10:28 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P80070 and previous config saved to /var/cache/conftool/dbconfig/20250728-102830-marostegui.json |
[production] |
| 10:25 |
<btullis@deploy1003> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply |
[production] |
| 10:25 |
<btullis@deploy1003> |
helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply |
[production] |
| 10:24 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2196 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P80069 and previous config saved to /var/cache/conftool/dbconfig/20250728-102444-root.json |
[production] |
| 10:24 |
<elukey@cumin1003> |
START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 10:24 |
<elukey@cumin1003> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve1012.eqiad.wmnet with OS trixie |
[production] |
| 10:23 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2162 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P80068 and previous config saved to /var/cache/conftool/dbconfig/20250728-102311-root.json |
[production] |
| 10:23 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2218 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P80067 and previous config saved to /var/cache/conftool/dbconfig/20250728-102300-root.json |
[production] |
| 10:17 |
<btullis@deploy1003> |
Finished scap build-images: Updating mediawiki-cli image for T400383 (duration: 16m 00s) |
[production] |
| 10:13 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P80066 and previous config saved to /var/cache/conftool/dbconfig/20250728-101348-fceratto.json |
[production] |
| 10:13 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P80065 and previous config saved to /var/cache/conftool/dbconfig/20250728-101322-marostegui.json |
[production] |
| 10:09 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2196 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P80064 and previous config saved to /var/cache/conftool/dbconfig/20250728-100938-root.json |
[production] |
| 10:09 |
<elukey@cumin1003> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve1012.eqiad.wmnet with reason: host reimage |
[production] |
| 10:08 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2162 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P80063 and previous config saved to /var/cache/conftool/dbconfig/20250728-100806-root.json |
[production] |
| 10:07 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2218 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P80062 and previous config saved to /var/cache/conftool/dbconfig/20250728-100754-root.json |
[production] |
| 10:05 |
<elukey@cumin1003> |
START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve1012.eqiad.wmnet with reason: host reimage |
[production] |
| 10:05 |
<elukey@cumin1003> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 10:02 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db2162 T400599', diff saved to https://phabricator.wikimedia.org/P80061 and previous config saved to /var/cache/conftool/dbconfig/20250728-100243-marostegui.json |
[production] |
| 10:02 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Promote db2241 to x3 primary T400599', diff saved to https://phabricator.wikimedia.org/P80060 and previous config saved to /var/cache/conftool/dbconfig/20250728-100208-root.json |
[production] |
| 10:01 |
<marostegui> |
Starting x3 codfw failover from db2162 to db2241 - T400599 |
[production] |
| 10:01 |
<btullis@deploy1003> |
Started scap build-images: Updating mediawiki-cli image for T400383 |
[production] |
| 09:58 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P80059 and previous config saved to /var/cache/conftool/dbconfig/20250728-095841-fceratto.json |
[production] |
| 09:58 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1192 (T399249)', diff saved to https://phabricator.wikimedia.org/P80058 and previous config saved to /var/cache/conftool/dbconfig/20250728-095815-marostegui.json |
[production] |
| 09:56 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1192 (T399249)', diff saved to https://phabricator.wikimedia.org/P80057 and previous config saved to /var/cache/conftool/dbconfig/20250728-095601-marostegui.json |
[production] |
| 09:55 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1192.eqiad.wmnet with reason: Maintenance |
[production] |