|
2025-07-28
ยง
|
| 10:25 |
<btullis@deploy1003> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply |
[production] |
| 10:25 |
<btullis@deploy1003> |
helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply |
[production] |
| 10:24 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2196 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P80069 and previous config saved to /var/cache/conftool/dbconfig/20250728-102444-root.json |
[production] |
| 10:24 |
<elukey@cumin1003> |
START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 10:24 |
<elukey@cumin1003> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-serve1012.eqiad.wmnet with OS trixie |
[production] |
| 10:23 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2162 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P80068 and previous config saved to /var/cache/conftool/dbconfig/20250728-102311-root.json |
[production] |
| 10:23 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2218 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P80067 and previous config saved to /var/cache/conftool/dbconfig/20250728-102300-root.json |
[production] |
| 10:17 |
<btullis@deploy1003> |
Finished scap build-images: Updating mediawiki-cli image for T400383 (duration: 16m 00s) |
[production] |
| 10:13 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P80066 and previous config saved to /var/cache/conftool/dbconfig/20250728-101348-fceratto.json |
[production] |
| 10:13 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P80065 and previous config saved to /var/cache/conftool/dbconfig/20250728-101322-marostegui.json |
[production] |
| 10:09 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2196 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P80064 and previous config saved to /var/cache/conftool/dbconfig/20250728-100938-root.json |
[production] |
| 10:09 |
<elukey@cumin1003> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve1012.eqiad.wmnet with reason: host reimage |
[production] |
| 10:08 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2162 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P80063 and previous config saved to /var/cache/conftool/dbconfig/20250728-100806-root.json |
[production] |
| 10:07 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2218 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P80062 and previous config saved to /var/cache/conftool/dbconfig/20250728-100754-root.json |
[production] |
| 10:05 |
<elukey@cumin1003> |
START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve1012.eqiad.wmnet with reason: host reimage |
[production] |
| 10:05 |
<elukey@cumin1003> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 10:02 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool db2162 T400599', diff saved to https://phabricator.wikimedia.org/P80061 and previous config saved to /var/cache/conftool/dbconfig/20250728-100243-marostegui.json |
[production] |
| 10:02 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Promote db2241 to x3 primary T400599', diff saved to https://phabricator.wikimedia.org/P80060 and previous config saved to /var/cache/conftool/dbconfig/20250728-100208-root.json |
[production] |
| 10:01 |
<marostegui> |
Starting x3 codfw failover from db2162 to db2241 - T400599 |
[production] |
| 10:01 |
<btullis@deploy1003> |
Started scap build-images: Updating mediawiki-cli image for T400383 |
[production] |
| 09:58 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P80059 and previous config saved to /var/cache/conftool/dbconfig/20250728-095841-fceratto.json |
[production] |
| 09:58 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1192 (T399249)', diff saved to https://phabricator.wikimedia.org/P80058 and previous config saved to /var/cache/conftool/dbconfig/20250728-095815-marostegui.json |
[production] |
| 09:56 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1192 (T399249)', diff saved to https://phabricator.wikimedia.org/P80057 and previous config saved to /var/cache/conftool/dbconfig/20250728-095601-marostegui.json |
[production] |
| 09:55 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1192.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:55 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 16 hosts with reason: Primary switchover x3 T400599 |
[production] |
| 09:55 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1178 (T399249)', diff saved to https://phabricator.wikimedia.org/P80056 and previous config saved to /var/cache/conftool/dbconfig/20250728-095539-marostegui.json |
[production] |
| 09:54 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2196 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P80055 and previous config saved to /var/cache/conftool/dbconfig/20250728-095432-root.json |
[production] |
| 09:53 |
<elukey@cumin1003> |
START - Cookbook sre.hosts.reimage for host ml-serve1012.eqiad.wmnet with OS trixie |
[production] |
| 09:52 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2218 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P80054 and previous config saved to /var/cache/conftool/dbconfig/20250728-095249-root.json |
[production] |
| 09:51 |
<wmbot~dcaro@acme> |
START - Cookbook wmcs.openstack.cloudvirt.vm_console |
[admin-monitoring] |
| 09:51 |
<wmbot~dcaro@acme> |
END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99) |
[admin-monitoring] |
| 09:51 |
<wmbot~dcaro@acme> |
START - Cookbook wmcs.openstack.cloudvirt.vm_console |
[admin-monitoring] |
| 09:44 |
<wmbot~dcaro@acme> |
END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) |
[admin-monitoring] |
| 09:43 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2178 (T399728)', diff saved to https://phabricator.wikimedia.org/P80053 and previous config saved to /var/cache/conftool/dbconfig/20250728-094333-fceratto.json |
[production] |
| 09:41 |
<wmbot~dcaro@acme> |
START - Cookbook wmcs.openstack.cloudvirt.vm_console |
[admin-monitoring] |
| 09:41 |
<wmbot~dcaro@acme> |
END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99) |
[admin-monitoring] |
| 09:41 |
<wmbot~dcaro@acme> |
START - Cookbook wmcs.openstack.cloudvirt.vm_console |
[admin-monitoring] |
| 09:40 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P80052 and previous config saved to /var/cache/conftool/dbconfig/20250728-094031-marostegui.json |
[production] |
| 09:39 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db2178 (T399728)', diff saved to https://phabricator.wikimedia.org/P80051 and previous config saved to /var/cache/conftool/dbconfig/20250728-093945-fceratto.json |
[production] |
| 09:39 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2178.codfw.wmnet with reason: Maintenance |
[production] |
| 09:39 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2196 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P80050 and previous config saved to /var/cache/conftool/dbconfig/20250728-093926-root.json |
[production] |
| 09:39 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2171 (T399728)', diff saved to https://phabricator.wikimedia.org/P80049 and previous config saved to /var/cache/conftool/dbconfig/20250728-093922-fceratto.json |
[production] |
| 09:39 |
<wmbot~dcaro@acme> |
END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) |
[admin-monitoring] |
| 09:37 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db2218 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P80048 and previous config saved to /var/cache/conftool/dbconfig/20250728-093743-root.json |
[production] |
| 09:35 |
<wmbot~multichill@tools-bastion-12> |
Started the coord-from-exif-mysql job (T399417) |
[tools.multichill] |
| 09:34 |
<wmbot~multichill@tools-bastion-12> |
Web service giving 504. Restart no change. Created T400595 |
[tools.heritage] |
| 09:34 |
<wmbot~dcaro@acme> |
START - Cookbook wmcs.openstack.cloudvirt.vm_console |
[admin-monitoring] |
| 09:33 |
<wmbot~dcaro@acme> |
END (FAIL) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=99) |
[admin-monitoring] |
| 09:33 |
<wmbot~dcaro@acme> |
START - Cookbook wmcs.openstack.cloudvirt.vm_console |
[admin-monitoring] |
| 09:30 |
<elukey@cumin1003> |
START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |