|
2025-11-12
ยง
|
| 11:44 |
<cgoubert@deploy2002> |
helmfile [codfw] START helmfile.d/services/rest-gateway: apply |
[production] |
| 11:43 |
<cgoubert@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply |
[production] |
| 11:43 |
<cgoubert@deploy2002> |
helmfile [eqiad] START helmfile.d/services/rest-gateway: apply |
[production] |
| 11:41 |
<jmm@cumin1003> |
END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cumin2002.codfw.wmnet |
[production] |
| 11:39 |
<jmm@cumin1003> |
END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host cumin2002.codfw.wmnet |
[production] |
| 11:36 |
<moritzm> |
migrated cumin2002 to nftables T389380 |
[production] |
| 11:30 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 20%: Moved it to es7', diff saved to https://phabricator.wikimedia.org/P85278 and previous config saved to /var/cache/conftool/dbconfig/20251112-113040-root.json |
[production] |
| 11:26 |
<jmm@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host cumin2002.codfw.wmnet |
[production] |
| 11:25 |
<jmm@cumin1003> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cumin2002.codfw.wmnet |
[production] |
| 11:24 |
<jmm@cumin1003> |
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cumin2002.codfw.wmnet |
[production] |
| 11:22 |
<topranks> |
will not shut just yet will log again when about to do so T409800 |
[production] |
| 11:18 |
<topranks> |
shut down link from ssw1-d8-eqiad ethernet-1/28 <-> asw2-c7-eqiad et-7/0/49 to observe results T409800 |
[production] |
| 11:18 |
<jmm@cumin1003> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cumin2002.codfw.wmnet |
[production] |
| 11:17 |
<cmooney@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw2-c-eqiad,ssw1-d8-eqiad with reason: shutting down one leg of LAG from ssw1-d8-eqiad to asw2-c7-eqiad |
[production] |
| 11:17 |
<aklapper@deploy2002> |
rebuilt and synchronized wikiversions files: group0 to 1.46.0-wmf.2 refs T408272 |
[production] |
| 11:15 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 10%: Moved it to es7', diff saved to https://phabricator.wikimedia.org/P85277 and previous config saved to /var/cache/conftool/dbconfig/20251112-111534-root.json |
[production] |
| 11:08 |
<cgoubert@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply |
[production] |
| 11:07 |
<cgoubert@deploy2002> |
helmfile [codfw] START helmfile.d/services/rest-gateway: apply |
[production] |
| 11:07 |
<cgoubert@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply |
[production] |
| 11:06 |
<cgoubert@deploy2002> |
helmfile [eqiad] START helmfile.d/services/rest-gateway: apply |
[production] |
| 11:06 |
<mvolz@deploy1003> |
helmfile [staging] DONE helmfile.d/services/citoid: apply |
[production] |
| 11:04 |
<mvolz@deploy1003> |
helmfile [staging] START helmfile.d/services/citoid: apply |
[production] |
| 11:00 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 9%: Moved it to es7', diff saved to https://phabricator.wikimedia.org/P85275 and previous config saved to /var/cache/conftool/dbconfig/20251112-110028-root.json |
[production] |
| 10:50 |
<cgoubert@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply |
[production] |
| 10:50 |
<cgoubert@deploy2002> |
helmfile [eqiad] START helmfile.d/services/rest-gateway: apply |
[production] |
| 10:49 |
<aklapper@deploy2002> |
rebuilt and synchronized wikiversions files: group1 to 1.46.0-wmf.2 refs T408272 |
[production] |
| 10:45 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 8%: Moved it to es7', diff saved to https://phabricator.wikimedia.org/P85273 and previous config saved to /var/cache/conftool/dbconfig/20251112-104522-root.json |
[production] |
| 10:39 |
<kevinbazira@deploy2002> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revision-models' for release 'main' . |
[production] |
| 10:39 |
<cgoubert@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply |
[production] |
| 10:38 |
<cgoubert@deploy2002> |
helmfile [eqiad] START helmfile.d/services/rest-gateway: apply |
[production] |
| 10:37 |
<ladsgroup@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1204258|Revert "pagers: Make history pager work with Postgres" (T409831)]] (duration: 08m 34s) |
[production] |
| 10:34 |
<cgoubert@deploy2002> |
helmfile [staging] DONE helmfile.d/services/rest-gateway: apply |
[production] |
| 10:34 |
<cgoubert@deploy2002> |
helmfile [staging] START helmfile.d/services/rest-gateway: apply |
[production] |
| 10:32 |
<ladsgroup@deploy2002> |
ladsgroup: Continuing with sync |
[production] |
| 10:30 |
<ladsgroup@deploy2002> |
ladsgroup: Backport for [[gerrit:1204258|Revert "pagers: Make history pager work with Postgres" (T409831)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 10:30 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 7%: Moved it to es7', diff saved to https://phabricator.wikimedia.org/P85272 and previous config saved to /var/cache/conftool/dbconfig/20251112-103016-root.json |
[production] |
| 10:28 |
<ladsgroup@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1204258|Revert "pagers: Make history pager work with Postgres" (T409831)]] |
[production] |
| 10:26 |
<jynus@cumin1003> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 8 hosts |
[production] |
| 10:26 |
<jynus@cumin1003> |
START - Cookbook sre.hosts.remove-downtime for 8 hosts |
[production] |
| 10:18 |
<kharlan@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1203841|hCaptcha: Set fallback for ConfirmEditTriggersCaptcha (T409736)]] (duration: 07m 51s) |
[production] |
| 10:15 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 6%: Moved it to es7', diff saved to https://phabricator.wikimedia.org/P85271 and previous config saved to /var/cache/conftool/dbconfig/20251112-101510-root.json |
[production] |
| 10:14 |
<kharlan@deploy2002> |
kharlan: Continuing with sync |
[production] |
| 10:12 |
<kharlan@deploy2002> |
kharlan: Backport for [[gerrit:1203841|hCaptcha: Set fallback for ConfirmEditTriggersCaptcha (T409736)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 10:10 |
<kharlan@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1203841|hCaptcha: Set fallback for ConfirmEditTriggersCaptcha (T409736)]] |
[production] |
| 10:05 |
<kharlan@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1204112|Throttler: Use SecurityLogContext]] (duration: 10m 35s) |
[production] |
| 10:03 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Add db1264 to x1 depooled T407941', diff saved to https://phabricator.wikimedia.org/P85270 and previous config saved to /var/cache/conftool/dbconfig/20251112-100346-marostegui.json |
[production] |
| 10:01 |
<kharlan@deploy2002> |
kharlan: Continuing with sync |
[production] |
| 10:00 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 5%: Moved it to es7', diff saved to https://phabricator.wikimedia.org/P85268 and previous config saved to /var/cache/conftool/dbconfig/20251112-100004-root.json |
[production] |
| 09:57 |
<kharlan@deploy2002> |
kharlan: Backport for [[gerrit:1204112|Throttler: Use SecurityLogContext]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 09:55 |
<kharlan@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1204112|Throttler: Use SecurityLogContext]] |
[production] |