|
2025-11-12
ยง
|
| 12:46 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 45%: Moved it to es7', diff saved to https://phabricator.wikimedia.org/P85285 and previous config saved to /var/cache/conftool/dbconfig/20251112-124609-root.json |
[production] |
| 12:43 |
<fceratto@cumin1002> |
END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2230.codfw.wmnet onto db-test2001.codfw.wmnet |
[production] |
| 12:41 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1013,1022].eqiad.wmnet with reason: Cloning clouddb1022:s3 |
[production] |
| 12:41 |
<marostegui@cumin1003> |
conftool action : set/pooled=no; selector: name=clouddb1013.eqiad.wmnet,service=s3 |
[production] |
| 12:31 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 40%: Moved it to es7', diff saved to https://phabricator.wikimedia.org/P85282 and previous config saved to /var/cache/conftool/dbconfig/20251112-123103-root.json |
[production] |
| 12:30 |
<kart_> |
Updated cxserver to 2025-11-12-114324-production (T408515) |
[production] |
| 12:21 |
<kartik@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/cxserver: apply |
[production] |
| 12:20 |
<kartik@deploy2002> |
helmfile [eqiad] START helmfile.d/services/cxserver: apply |
[production] |
| 12:20 |
<kartik@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/cxserver: apply |
[production] |
| 12:19 |
<kartik@deploy2002> |
helmfile [codfw] START helmfile.d/services/cxserver: apply |
[production] |
| 12:15 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 35%: Moved it to es7', diff saved to https://phabricator.wikimedia.org/P85281 and previous config saved to /var/cache/conftool/dbconfig/20251112-121557-root.json |
[production] |
| 12:14 |
<topranks> |
shut down link from ssw1-d8-eqiad ethernet-1/28 <-> asw2-c7-eqiad et-7/0/49 to observe results T409800 |
[production] |
| 12:14 |
<kartik@deploy2002> |
helmfile [staging] DONE helmfile.d/services/cxserver: apply |
[production] |
| 12:14 |
<kartik@deploy2002> |
helmfile [staging] START helmfile.d/services/cxserver: apply |
[production] |
| 12:09 |
<mvolz@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/citoid: apply |
[production] |
| 12:09 |
<mvolz@deploy2002> |
helmfile [eqiad] START helmfile.d/services/citoid: apply |
[production] |
| 12:08 |
<mvolz@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/citoid: apply |
[production] |
| 12:07 |
<mvolz@deploy2002> |
helmfile [codfw] START helmfile.d/services/citoid: apply |
[production] |
| 12:06 |
<mvolz@deploy2002> |
helmfile [staging] DONE helmfile.d/services/citoid: apply |
[production] |
| 12:05 |
<mvolz@deploy2002> |
helmfile [staging] START helmfile.d/services/citoid: apply |
[production] |
| 12:00 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 30%: Moved it to es7', diff saved to https://phabricator.wikimedia.org/P85280 and previous config saved to /var/cache/conftool/dbconfig/20251112-120051-root.json |
[production] |
| 11:45 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 25%: Moved it to es7', diff saved to https://phabricator.wikimedia.org/P85279 and previous config saved to /var/cache/conftool/dbconfig/20251112-114545-root.json |
[production] |
| 11:44 |
<cgoubert@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply |
[production] |
| 11:44 |
<cgoubert@deploy2002> |
helmfile [codfw] START helmfile.d/services/rest-gateway: apply |
[production] |
| 11:43 |
<cgoubert@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply |
[production] |
| 11:43 |
<cgoubert@deploy2002> |
helmfile [eqiad] START helmfile.d/services/rest-gateway: apply |
[production] |
| 11:41 |
<jmm@cumin1003> |
END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cumin2002.codfw.wmnet |
[production] |
| 11:39 |
<jmm@cumin1003> |
END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host cumin2002.codfw.wmnet |
[production] |
| 11:36 |
<moritzm> |
migrated cumin2002 to nftables T389380 |
[production] |
| 11:30 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 20%: Moved it to es7', diff saved to https://phabricator.wikimedia.org/P85278 and previous config saved to /var/cache/conftool/dbconfig/20251112-113040-root.json |
[production] |
| 11:26 |
<jmm@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host cumin2002.codfw.wmnet |
[production] |
| 11:25 |
<jmm@cumin1003> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cumin2002.codfw.wmnet |
[production] |
| 11:24 |
<jmm@cumin1003> |
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cumin2002.codfw.wmnet |
[production] |
| 11:22 |
<topranks> |
will not shut just yet will log again when about to do so T409800 |
[production] |
| 11:18 |
<topranks> |
shut down link from ssw1-d8-eqiad ethernet-1/28 <-> asw2-c7-eqiad et-7/0/49 to observe results T409800 |
[production] |
| 11:18 |
<jmm@cumin1003> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cumin2002.codfw.wmnet |
[production] |
| 11:17 |
<cmooney@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw2-c-eqiad,ssw1-d8-eqiad with reason: shutting down one leg of LAG from ssw1-d8-eqiad to asw2-c7-eqiad |
[production] |
| 11:17 |
<aklapper@deploy2002> |
rebuilt and synchronized wikiversions files: group0 to 1.46.0-wmf.2 refs T408272 |
[production] |
| 11:15 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 10%: Moved it to es7', diff saved to https://phabricator.wikimedia.org/P85277 and previous config saved to /var/cache/conftool/dbconfig/20251112-111534-root.json |
[production] |
| 11:08 |
<cgoubert@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply |
[production] |
| 11:07 |
<cgoubert@deploy2002> |
helmfile [codfw] START helmfile.d/services/rest-gateway: apply |
[production] |
| 11:07 |
<cgoubert@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply |
[production] |
| 11:06 |
<cgoubert@deploy2002> |
helmfile [eqiad] START helmfile.d/services/rest-gateway: apply |
[production] |
| 11:06 |
<mvolz@deploy1003> |
helmfile [staging] DONE helmfile.d/services/citoid: apply |
[production] |
| 11:04 |
<mvolz@deploy1003> |
helmfile [staging] START helmfile.d/services/citoid: apply |
[production] |
| 11:00 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 9%: Moved it to es7', diff saved to https://phabricator.wikimedia.org/P85275 and previous config saved to /var/cache/conftool/dbconfig/20251112-110028-root.json |
[production] |
| 10:50 |
<cgoubert@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply |
[production] |
| 10:50 |
<cgoubert@deploy2002> |
helmfile [eqiad] START helmfile.d/services/rest-gateway: apply |
[production] |
| 10:49 |
<aklapper@deploy2002> |
rebuilt and synchronized wikiversions files: group1 to 1.46.0-wmf.2 refs T408272 |
[production] |
| 10:45 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 8%: Moved it to es7', diff saved to https://phabricator.wikimedia.org/P85273 and previous config saved to /var/cache/conftool/dbconfig/20251112-104522-root.json |
[production] |