|
2025-12-01
ยง
|
| 20:20 |
<urbanecm@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1213557|Introduce HTML confirmation email (T396155)]], [[gerrit:1213558|ConfirmEmailHooks: Do not run when UserEmailConfirmationUseHTML is true (T396155)]] |
[production] |
| 20:13 |
<jhathaway@cumin2002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on sretest2001.codfw.wmnet with reason: T383173 |
[production] |
| 20:10 |
<taavi@cumin1003> |
END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-low-traffic-eqiad |
[production] |
| 20:09 |
<taavi@cumin1003> |
START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-low-traffic-eqiad |
[production] |
| 20:08 |
<taavi@cumin1003> |
END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-secondary-eqiad |
[production] |
| 20:08 |
<mutante> |
upgrading envoyproxy on contint1002; phab1004; T405808 |
[production] |
| 20:04 |
<taavi@cumin1003> |
START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-secondary-eqiad |
[production] |
| 20:04 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2178 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P86256 and previous config saved to /var/cache/conftool/dbconfig/20251201-200359-marostegui.json |
[production] |
| 20:03 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2178.codfw.wmnet with reason: Maintenance |
[production] |
| 20:03 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2171 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P86255 and previous config saved to /var/cache/conftool/dbconfig/20251201-200335-marostegui.json |
[production] |
| 20:02 |
<mutante> |
updating envoyproxy from 1.29.x to 1.32.x on phabricator prod host |
[production] |
| 19:49 |
<cdobbins@cumin2002> |
END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) rebooting P{lvs6003*} and A:liberica |
[production] |
| 19:48 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P86254 and previous config saved to /var/cache/conftool/dbconfig/20251201-194828-marostegui.json |
[production] |
| 19:46 |
<cdobbins@cumin2002> |
START - Cookbook sre.loadbalancer.admin rebooting P{lvs6003*} and A:liberica |
[production] |
| 19:33 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P86253 and previous config saved to /var/cache/conftool/dbconfig/20251201-193320-marostegui.json |
[production] |
| 19:28 |
<cdobbins@cumin2002> |
END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) rebooting P{lvs6003*} and A:liberica |
[production] |
| 19:25 |
<cdobbins@cumin2002> |
START - Cookbook sre.loadbalancer.admin rebooting P{lvs6003*} and A:liberica |
[production] |
| 19:18 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2171 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P86252 and previous config saved to /var/cache/conftool/dbconfig/20251201-191812-marostegui.json |
[production] |
| 19:14 |
<cdobbins@cumin2002> |
END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) rebooting P{lvs6003*} and A:liberica |
[production] |
| 19:11 |
<cdobbins@cumin2002> |
START - Cookbook sre.loadbalancer.admin rebooting P{lvs6003*} and A:liberica |
[production] |
| 19:03 |
<cdobbins@cumin2002> |
END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) rebooting P{lvs6003*} and A:liberica |
[production] |
| 19:00 |
<cdobbins@cumin2002> |
START - Cookbook sre.loadbalancer.admin rebooting P{lvs6003*} and A:liberica |
[production] |
| 18:44 |
<andrew@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudweb1003.wikimedia.org with OS trixie |
[production] |
| 18:24 |
<andrew@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudweb1003.wikimedia.org with reason: host reimage |
[production] |
| 18:18 |
<andrew@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cloudweb1003.wikimedia.org with reason: host reimage |
[production] |
| 18:05 |
<andrew@cumin2002> |
START - Cookbook sre.hosts.reimage for host cloudweb1003.wikimedia.org with OS trixie |
[production] |
| 18:03 |
<fceratto@deploy2002> |
helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . |
[production] |
| 18:02 |
<taavi@cumin1003> |
END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-secondary-eqiad |
[production] |
| 18:01 |
<taavi@cumin1003> |
START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-secondary-eqiad |
[production] |
| 18:00 |
<taavi@cumin1003> |
END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-low-traffic-eqiad |
[production] |
| 17:59 |
<taavi@cumin1003> |
START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-low-traffic-eqiad |
[production] |
| 17:56 |
<fceratto@deploy2002> |
helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . |
[production] |
| 17:45 |
<taavi@cumin1003> |
conftool action : set/pooled=no; selector: cluster=cloudweb,name=cloudweb1003.wikimedia.org |
[production] |
| 17:43 |
<taavi@cumin1003> |
conftool action : set/pooled=inactive; selector: cluster=cloudweb,name=cloudweb1003.wikimedia.org |
[production] |
| 17:39 |
<andrew@cumin2002> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudweb1003.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL |
[production] |
| 17:39 |
<bd808@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1208478|labswiki: Enable sitenotice on mobile (T410702)]] (duration: 06m 49s) |
[production] |
| 17:39 |
<tappof> |
"thanos-store: set cutoff days to 1" reverted on titan2001 (4/4) T410152 |
[production] |
| 17:35 |
<bd808@deploy2002> |
bd808: Continuing with sync |
[production] |
| 17:34 |
<bd808@deploy2002> |
bd808: Backport for [[gerrit:1208478|labswiki: Enable sitenotice on mobile (T410702)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 17:32 |
<bd808@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1208478|labswiki: Enable sitenotice on mobile (T410702)]] |
[production] |
| 17:32 |
<andrew@cumin2002> |
START - Cookbook sre.hosts.provision for host cloudweb1003.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL |
[production] |
| 17:31 |
<andrew@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudweb1004.wikimedia.org with OS trixie |
[production] |
| 17:17 |
<tappof> |
"thanos-store: set cutoff days to 1" reverted on titan2002 (3/4) T410152 |
[production] |
| 17:08 |
<hnowlan@deploy2002> |
Finished deploy [restbase/deploy@19cb647]: Add new wikis to restbase T408352 T408344 (duration: 16m 16s) |
[production] |
| 16:59 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db1157 (T411163 T411164)', diff saved to https://phabricator.wikimedia.org/P86251 and previous config saved to /var/cache/conftool/dbconfig/20251201-165902-marostegui.json |
[production] |
| 16:58 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1157.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:58 |
<cdobbins@cumin2002> |
END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) rebooting P{lvs6003*} and A:liberica |
[production] |
| 16:55 |
<cdobbins@cumin2002> |
START - Cookbook sre.loadbalancer.admin rebooting P{lvs6003*} and A:liberica |
[production] |
| 16:52 |
<hnowlan@deploy2002> |
Started deploy [restbase/deploy@19cb647]: Add new wikis to restbase T408352 T408344 |
[production] |
| 16:48 |
<andrew@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudweb1004.wikimedia.org with reason: host reimage |
[production] |