|
2024-01-22
ยง
|
| 17:44 |
<bking@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host elastic2088.codfw.wmnet with OS bullseye |
[production] |
| 17:42 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P55235 and previous config saved to /var/cache/conftool/dbconfig/20240122-174249-marostegui.json |
[production] |
| 17:38 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2155 (T352010)', diff saved to https://phabricator.wikimedia.org/P55234 and previous config saved to /var/cache/conftool/dbconfig/20240122-173840-ladsgroup.json |
[production] |
| 17:27 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1178 (T354336)', diff saved to https://phabricator.wikimedia.org/P55233 and previous config saved to /var/cache/conftool/dbconfig/20240122-172743-marostegui.json |
[production] |
| 17:26 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1178 (T354336)', diff saved to https://phabricator.wikimedia.org/P55232 and previous config saved to /var/cache/conftool/dbconfig/20240122-172635-marostegui.json |
[production] |
| 17:26 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1178.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:26 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1178.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:26 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1177 (T354336)', diff saved to https://phabricator.wikimedia.org/P55231 and previous config saved to /var/cache/conftool/dbconfig/20240122-172612-marostegui.json |
[production] |
| 17:17 |
<akosiaris> |
draining kubestage2001, uncordoning kubestage2002 to allow it to receive the pods. T355437 |
[production] |
| 17:11 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P55230 and previous config saved to /var/cache/conftool/dbconfig/20240122-171106-marostegui.json |
[production] |
| 17:05 |
<vgutierrez> |
restore HAProxy tune.bufsize = 16684 in cp3066 - T354424 |
[production] |
| 16:56 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P55229 and previous config saved to /var/cache/conftool/dbconfig/20240122-165559-marostegui.json |
[production] |
| 16:53 |
<vgutierrez> |
testing HAProxy tune.bufsize = 32768 in cp3066 - T354424 |
[production] |
| 16:46 |
<dcausse@deploy2002> |
Finished deploy [airflow-dags/search@dcf08b2]: (no justification provided) (duration: 00m 31s) |
[production] |
| 16:46 |
<dcausse@deploy2002> |
Started deploy [airflow-dags/search@dcf08b2]: (no justification provided) |
[production] |
| 16:42 |
<Daimona> |
T353459 Running mwscript /home/daimona/GenerateInvitationList.php to test the script before it reaches production |
[production] |
| 16:40 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1177 (T354336)', diff saved to https://phabricator.wikimedia.org/P55228 and previous config saved to /var/cache/conftool/dbconfig/20240122-164053-marostegui.json |
[production] |
| 16:39 |
<hnowlan@cumin1002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1495.eqiad.wmnet with OS bullseye |
[production] |
| 16:38 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1177 (T354336)', diff saved to https://phabricator.wikimedia.org/P55227 and previous config saved to /var/cache/conftool/dbconfig/20240122-163844-marostegui.json |
[production] |
| 16:38 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1177.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:38 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1177.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:38 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1172 (T354336)', diff saved to https://phabricator.wikimedia.org/P55226 and previous config saved to /var/cache/conftool/dbconfig/20240122-163822-marostegui.json |
[production] |
| 16:38 |
<cgoubert@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply |
[production] |
| 16:38 |
<cgoubert@deploy2002> |
helmfile [codfw] START helmfile.d/services/mw-api-int: apply |
[production] |
| 16:38 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2155 (T352010)', diff saved to https://phabricator.wikimedia.org/P55225 and previous config saved to /var/cache/conftool/dbconfig/20240122-163808-ladsgroup.json |
[production] |
| 16:38 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance |
[production] |
| 16:38 |
<cgoubert@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply |
[production] |
| 16:38 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance |
[production] |
| 16:37 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2155.codfw.wmnet with reason: Maintenance |
[production] |
| 16:37 |
<cgoubert@deploy2002> |
helmfile [eqiad] START helmfile.d/services/mw-api-int: apply |
[production] |
| 16:37 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2155.codfw.wmnet with reason: Maintenance |
[production] |
| 16:37 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2147 (T352010)', diff saved to https://phabricator.wikimedia.org/P55224 and previous config saved to /var/cache/conftool/dbconfig/20240122-163729-ladsgroup.json |
[production] |
| 16:31 |
<hnowlan@cumin1002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1486.eqiad.wmnet with OS bullseye |
[production] |
| 16:29 |
<klausman@deploy2002> |
helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 16:29 |
<klausman@deploy2002> |
helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 16:23 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P55222 and previous config saved to /var/cache/conftool/dbconfig/20240122-162315-marostegui.json |
[production] |
| 16:22 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P55221 and previous config saved to /var/cache/conftool/dbconfig/20240122-162223-ladsgroup.json |
[production] |
| 16:14 |
<hnowlan@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1495.eqiad.wmnet with reason: host reimage |
[production] |
| 16:12 |
<hnowlan@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1486.eqiad.wmnet with reason: host reimage |
[production] |
| 16:09 |
<hnowlan@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1495.eqiad.wmnet with reason: host reimage |
[production] |
| 16:08 |
<hnowlan@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1486.eqiad.wmnet with reason: host reimage |
[production] |
| 16:08 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P55220 and previous config saved to /var/cache/conftool/dbconfig/20240122-160809-marostegui.json |
[production] |
| 16:07 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P55219 and previous config saved to /var/cache/conftool/dbconfig/20240122-160716-ladsgroup.json |
[production] |
| 15:56 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1165 (re)pooling @ 100%: Upgrade to 10.6.16 and bookworm', diff saved to https://phabricator.wikimedia.org/P55218 and previous config saved to /var/cache/conftool/dbconfig/20240122-155607-root.json |
[production] |
| 15:55 |
<hnowlan@cumin1002> |
START - Cookbook sre.hosts.reimage for host mw1495.eqiad.wmnet with OS bullseye |
[production] |
| 15:55 |
<hnowlan@cumin1002> |
START - Cookbook sre.hosts.reimage for host mw1486.eqiad.wmnet with OS bullseye |
[production] |
| 15:53 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1172 (T354336)', diff saved to https://phabricator.wikimedia.org/P55217 and previous config saved to /var/cache/conftool/dbconfig/20240122-155302-marostegui.json |
[production] |
| 15:52 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2147 (T352010)', diff saved to https://phabricator.wikimedia.org/P55216 and previous config saved to /var/cache/conftool/dbconfig/20240122-155210-ladsgroup.json |
[production] |
| 15:51 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db1172 (T354336)', diff saved to https://phabricator.wikimedia.org/P55215 and previous config saved to /var/cache/conftool/dbconfig/20240122-155154-marostegui.json |
[production] |
| 15:51 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1172.eqiad.wmnet with reason: Maintenance |
[production] |