|
2023-01-04
ยง
|
| 12:41 |
<cgoubert@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 12:41 |
<urbanecm@deploy1002> |
Started scap: Creating guwwikiquote (T321247) |
[production] |
| 12:40 |
<claime> |
Rolling reboot of api_appserver hosts in codfw paused for https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20230104T1200 |
[production] |
| 12:38 |
<urbanecm@deploy1002> |
Finished scap: Creating aswikiquote (T321246) (duration: 07m 49s) |
[production] |
| 12:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2124 (re)pooling @ 75%: After cloning db2151', diff saved to https://phabricator.wikimedia.org/P42770 and previous config saved to /var/cache/conftool/dbconfig/20230104-123825-root.json |
[production] |
| 12:35 |
<btullis@cumin1001> |
START - Cookbook sre.hosts.reimage for host cephosd1001.eqiad.wmnet with OS bullseye |
[production] |
| 12:30 |
<urbanecm@deploy1002> |
Started scap: Creating aswikiquote (T321246) |
[production] |
| 12:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P42769 and previous config saved to /var/cache/conftool/dbconfig/20230104-122857-marostegui.json |
[production] |
| 12:27 |
<cgoubert@cumin1001> |
END (ERROR) - Cookbook sre.hosts.reboot-cluster (exit_code=97) |
[production] |
| 12:26 |
<urbanecm@deploy1002> |
Finished scap: Backport for [[gerrit:874880|Add namespace translations in Wayuu (T321881)]], [[gerrit:874879|Add namespace translations in Wayuu (T321881)]] (duration: 10m 36s) |
[production] |
| 12:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2124 (re)pooling @ 50%: After cloning db2151', diff saved to https://phabricator.wikimedia.org/P42768 and previous config saved to /var/cache/conftool/dbconfig/20230104-122320-root.json |
[production] |
| 12:18 |
<urbanecm@deploy1002> |
urbanecm and urbanecm: Backport for [[gerrit:874880|Add namespace translations in Wayuu (T321881)]], [[gerrit:874879|Add namespace translations in Wayuu (T321881)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet |
[production] |
| 12:16 |
<urbanecm@deploy1002> |
Started scap: Backport for [[gerrit:874880|Add namespace translations in Wayuu (T321881)]], [[gerrit:874879|Add namespace translations in Wayuu (T321881)]] |
[production] |
| 12:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P42767 and previous config saved to /var/cache/conftool/dbconfig/20230104-121350-marostegui.json |
[production] |
| 12:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2124 (re)pooling @ 25%: After cloning db2151', diff saved to https://phabricator.wikimedia.org/P42766 and previous config saved to /var/cache/conftool/dbconfig/20230104-120815-root.json |
[production] |
| 11:58 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2109 (T326011)', diff saved to https://phabricator.wikimedia.org/P42765 and previous config saved to /var/cache/conftool/dbconfig/20230104-115844-marostegui.json |
[production] |
| 11:53 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2124 (re)pooling @ 10%: After cloning db2151', diff saved to https://phabricator.wikimedia.org/P42764 and previous config saved to /var/cache/conftool/dbconfig/20230104-115310-root.json |
[production] |
| 11:50 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db2109 (T326011)', diff saved to https://phabricator.wikimedia.org/P42763 and previous config saved to /var/cache/conftool/dbconfig/20230104-115011-marostegui.json |
[production] |
| 11:50 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2109.codfw.wmnet with reason: Maintenance |
[production] |
| 11:49 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on db2109.codfw.wmnet with reason: Maintenance |
[production] |
| 11:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2124 (re)pooling @ 5%: After cloning db2151', diff saved to https://phabricator.wikimedia.org/P42761 and previous config saved to /var/cache/conftool/dbconfig/20230104-113805-root.json |
[production] |
| 11:33 |
<jmm@cumin2002> |
END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) for host puppetdb2003.codfw.wmnet |
[production] |
| 11:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Add db2151 to dbctl depooled T326206', diff saved to https://phabricator.wikimedia.org/P42759 and previous config saved to /var/cache/conftool/dbconfig/20230104-112801-marostegui.json |
[production] |
| 11:23 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db2124 (re)pooling @ 1%: After cloning db2151', diff saved to https://phabricator.wikimedia.org/P42758 and previous config saved to /var/cache/conftool/dbconfig/20230104-112300-root.json |
[production] |
| 11:02 |
<vgutierrez> |
testing HAProxy 2.4.20 in cp4037 and cp4045 |
[production] |
| 10:56 |
<vgutierrez> |
(apt1001) import HAproxy 2.4.20 from third-party repo for buster and bullseye |
[production] |
| 10:49 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging AKhatun out of all services on: 1098 hosts |
[production] |
| 10:48 |
<jmm@cumin2002> |
START - Cookbook sre.idm.logout Logging AKhatun out of all services on: 1098 hosts |
[production] |
| 10:48 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging AKhatun out of all services on: 894 hosts |
[production] |
| 10:47 |
<jmm@cumin2002> |
START - Cookbook sre.idm.logout Logging AKhatun out of all services on: 894 hosts |
[production] |
| 10:37 |
<cgoubert@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 10:37 |
<cgoubert@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 10:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db2124 T326206', diff saved to https://phabricator.wikimedia.org/P42756 and previous config saved to /var/cache/conftool/dbconfig/20230104-103109-marostegui.json |
[production] |
| 10:29 |
<cgoubert@cumin1001> |
START - Cookbook sre.hosts.reboot-cluster |
[production] |
| 10:29 |
<claime> |
Rolling reboot of api_appserver hosts in codfw |
[production] |
| 10:24 |
<cgoubert@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0) |
[production] |
| 10:14 |
<cgoubert@cumin1001> |
START - Cookbook sre.hosts.reboot-cluster |
[production] |
| 10:14 |
<claime> |
Rolling reboot of mwdebug hosts in eqiad |
[production] |
| 10:13 |
<cgoubert@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0) |
[production] |
| 10:04 |
<cgoubert@cumin1001> |
START - Cookbook sre.hosts.reboot-cluster |
[production] |
| 10:04 |
<marostegui> |
dbmaint eqiad deploy schema change on s5 T326011 |
[production] |
| 10:04 |
<claime> |
Rolling reboot of mwdebug hosts in codfw |
[production] |
| 10:04 |
<filippo@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-web: apply |
[production] |
| 10:04 |
<filippo@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-web: apply |
[production] |
| 10:04 |
<filippo@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-web: apply |
[production] |
| 10:03 |
<filippo@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-web: apply |
[production] |
| 10:03 |
<filippo@deploy1002> |
helmfile [eqiad] [main] DONE helmfile.d/services/mw-jobrunner : sync |
[production] |
| 10:03 |
<filippo@deploy1002> |
helmfile [eqiad] [canary] DONE helmfile.d/services/mw-jobrunner : sync |
[production] |
| 10:03 |
<filippo@deploy1002> |
helmfile [eqiad] [main] START helmfile.d/services/mw-jobrunner : sync |
[production] |
| 10:03 |
<filippo@deploy1002> |
helmfile [eqiad] [canary] START helmfile.d/services/mw-jobrunner : sync |
[production] |