|
2025-11-12
ยง
|
| 12:15 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 35%: Moved it to es7', diff saved to https://phabricator.wikimedia.org/P85281 and previous config saved to /var/cache/conftool/dbconfig/20251112-121557-root.json |
[production] |
| 12:14 |
<topranks> |
shut down link from ssw1-d8-eqiad ethernet-1/28 <-> asw2-c7-eqiad et-7/0/49 to observe results T409800 |
[production] |
| 12:14 |
<kartik@deploy2002> |
helmfile [staging] DONE helmfile.d/services/cxserver: apply |
[production] |
| 12:14 |
<kartik@deploy2002> |
helmfile [staging] START helmfile.d/services/cxserver: apply |
[production] |
| 12:09 |
<mvolz@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/citoid: apply |
[production] |
| 12:09 |
<mvolz@deploy2002> |
helmfile [eqiad] START helmfile.d/services/citoid: apply |
[production] |
| 12:08 |
<mvolz@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/citoid: apply |
[production] |
| 12:07 |
<mvolz@deploy2002> |
helmfile [codfw] START helmfile.d/services/citoid: apply |
[production] |
| 12:06 |
<mvolz@deploy2002> |
helmfile [staging] DONE helmfile.d/services/citoid: apply |
[production] |
| 12:05 |
<mvolz@deploy2002> |
helmfile [staging] START helmfile.d/services/citoid: apply |
[production] |
| 12:00 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 30%: Moved it to es7', diff saved to https://phabricator.wikimedia.org/P85280 and previous config saved to /var/cache/conftool/dbconfig/20251112-120051-root.json |
[production] |
| 11:45 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 25%: Moved it to es7', diff saved to https://phabricator.wikimedia.org/P85279 and previous config saved to /var/cache/conftool/dbconfig/20251112-114545-root.json |
[production] |
| 11:44 |
<cgoubert@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply |
[production] |
| 11:44 |
<cgoubert@deploy2002> |
helmfile [codfw] START helmfile.d/services/rest-gateway: apply |
[production] |
| 11:43 |
<cgoubert@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply |
[production] |
| 11:43 |
<cgoubert@deploy2002> |
helmfile [eqiad] START helmfile.d/services/rest-gateway: apply |
[production] |
| 11:41 |
<jmm@cumin1003> |
END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cumin2002.codfw.wmnet |
[production] |
| 11:39 |
<jmm@cumin1003> |
END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host cumin2002.codfw.wmnet |
[production] |
| 11:36 |
<moritzm> |
migrated cumin2002 to nftables T389380 |
[production] |
| 11:30 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 20%: Moved it to es7', diff saved to https://phabricator.wikimedia.org/P85278 and previous config saved to /var/cache/conftool/dbconfig/20251112-113040-root.json |
[production] |
| 11:26 |
<jmm@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host cumin2002.codfw.wmnet |
[production] |
| 11:25 |
<jmm@cumin1003> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cumin2002.codfw.wmnet |
[production] |
| 11:24 |
<jmm@cumin1003> |
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cumin2002.codfw.wmnet |
[production] |
| 11:22 |
<topranks> |
will not shut just yet will log again when about to do so T409800 |
[production] |
| 11:18 |
<topranks> |
shut down link from ssw1-d8-eqiad ethernet-1/28 <-> asw2-c7-eqiad et-7/0/49 to observe results T409800 |
[production] |
| 11:18 |
<jmm@cumin1003> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cumin2002.codfw.wmnet |
[production] |
| 11:17 |
<cmooney@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw2-c-eqiad,ssw1-d8-eqiad with reason: shutting down one leg of LAG from ssw1-d8-eqiad to asw2-c7-eqiad |
[production] |
| 11:17 |
<aklapper@deploy2002> |
rebuilt and synchronized wikiversions files: group0 to 1.46.0-wmf.2 refs T408272 |
[production] |
| 11:15 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 10%: Moved it to es7', diff saved to https://phabricator.wikimedia.org/P85277 and previous config saved to /var/cache/conftool/dbconfig/20251112-111534-root.json |
[production] |
| 11:08 |
<cgoubert@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply |
[production] |
| 11:07 |
<cgoubert@deploy2002> |
helmfile [codfw] START helmfile.d/services/rest-gateway: apply |
[production] |
| 11:07 |
<cgoubert@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply |
[production] |
| 11:06 |
<cgoubert@deploy2002> |
helmfile [eqiad] START helmfile.d/services/rest-gateway: apply |
[production] |
| 11:06 |
<mvolz@deploy1003> |
helmfile [staging] DONE helmfile.d/services/citoid: apply |
[production] |
| 11:04 |
<mvolz@deploy1003> |
helmfile [staging] START helmfile.d/services/citoid: apply |
[production] |
| 11:00 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 9%: Moved it to es7', diff saved to https://phabricator.wikimedia.org/P85275 and previous config saved to /var/cache/conftool/dbconfig/20251112-110028-root.json |
[production] |
| 10:50 |
<cgoubert@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply |
[production] |
| 10:50 |
<cgoubert@deploy2002> |
helmfile [eqiad] START helmfile.d/services/rest-gateway: apply |
[production] |
| 10:49 |
<aklapper@deploy2002> |
rebuilt and synchronized wikiversions files: group1 to 1.46.0-wmf.2 refs T408272 |
[production] |
| 10:45 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 8%: Moved it to es7', diff saved to https://phabricator.wikimedia.org/P85273 and previous config saved to /var/cache/conftool/dbconfig/20251112-104522-root.json |
[production] |
| 10:39 |
<kevinbazira@deploy2002> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' . |
[production] |
| 10:39 |
<cgoubert@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply |
[production] |
| 10:38 |
<cgoubert@deploy2002> |
helmfile [eqiad] START helmfile.d/services/rest-gateway: apply |
[production] |
| 10:37 |
<ladsgroup@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1204258|Revert "pagers: Make history pager work with Postgres" (T409831)]] (duration: 08m 34s) |
[production] |
| 10:34 |
<cgoubert@deploy2002> |
helmfile [staging] DONE helmfile.d/services/rest-gateway: apply |
[production] |
| 10:34 |
<cgoubert@deploy2002> |
helmfile [staging] START helmfile.d/services/rest-gateway: apply |
[production] |
| 10:32 |
<ladsgroup@deploy2002> |
ladsgroup: Continuing with sync |
[production] |
| 10:30 |
<ladsgroup@deploy2002> |
ladsgroup: Backport for [[gerrit:1204258|Revert "pagers: Make history pager work with Postgres" (T409831)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 10:30 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 7%: Moved it to es7', diff saved to https://phabricator.wikimedia.org/P85272 and previous config saved to /var/cache/conftool/dbconfig/20251112-103016-root.json |
[production] |
| 10:28 |
<ladsgroup@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1204258|Revert "pagers: Make history pager work with Postgres" (T409831)]] |
[production] |