|
2023-08-21
ยง
|
| 14:15 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T344589)', diff saved to https://phabricator.wikimedia.org/P50627 and previous config saved to /var/cache/conftool/dbconfig/20230821-141504-ladsgroup.json |
[production] |
| 14:14 |
<hnowlan@cumin1001> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: hnowlan: updating records for reuse of thumbor servers for k8s nodes T343996 T343993 - hnowlan@cumin1001" |
[production] |
| 14:14 |
<sukhe@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host lvs3010.esams.wmnet |
[production] |
| 14:13 |
<sukhe@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host lvs3009.esams.wmnet |
[production] |
| 14:11 |
<sukhe@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host lvs3008.esams.wmnet |
[production] |
| 14:11 |
<hnowlan@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 14:10 |
<sukhe@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on lvs[3008-3009].esams.wmnet with reason: rebooting for microcode updates |
[production] |
| 14:10 |
<sukhe@cumin2002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on lvs[3008-3009].esams.wmnet with reason: rebooting for microcode updates |
[production] |
| 14:09 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2123', diff saved to https://phabricator.wikimedia.org/P50626 and previous config saved to /var/cache/conftool/dbconfig/20230821-140901-ladsgroup.json |
[production] |
| 14:04 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P50625 and previous config saved to /var/cache/conftool/dbconfig/20230821-140433-ladsgroup.json |
[production] |
| 14:02 |
<Amir1> |
killed cebwiki's dump gen on snapshot1010 |
[production] |
| 14:01 |
<cgoubert@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-misc: apply |
[production] |
| 14:01 |
<cgoubert@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-misc: apply |
[production] |
| 14:01 |
<cgoubert@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-misc: apply |
[production] |
| 14:01 |
<cgoubert@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-misc: apply |
[production] |
| 13:59 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P50624 and previous config saved to /var/cache/conftool/dbconfig/20230821-135958-ladsgroup.json |
[production] |
| 13:55 |
<cgoubert@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-misc: apply |
[production] |
| 13:55 |
<claime> |
Re-enforcing limits and requests for mw-misc - T342748 |
[production] |
| 13:55 |
<cgoubert@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-misc: apply |
[production] |
| 13:53 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2123', diff saved to https://phabricator.wikimedia.org/P50623 and previous config saved to /var/cache/conftool/dbconfig/20230821-135355-ladsgroup.json |
[production] |
| 13:53 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kafka-stretch2002.codfw.wmnet |
[production] |
| 13:49 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1213:3315 (T342617)', diff saved to https://phabricator.wikimedia.org/P50622 and previous config saved to /var/cache/conftool/dbconfig/20230821-134949-ladsgroup.json |
[production] |
| 13:49 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1213.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:49 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1213.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:49 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1172 (T342617)', diff saved to https://phabricator.wikimedia.org/P50621 and previous config saved to /var/cache/conftool/dbconfig/20230821-134926-ladsgroup.json |
[production] |
| 13:48 |
<cgoubert@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-web: apply |
[production] |
| 13:47 |
<cgoubert@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-web: apply |
[production] |
| 13:47 |
<btullis@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host kafka-stretch2002.codfw.wmnet |
[production] |
| 13:47 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kafka-stretch2001.codfw.wmnet |
[production] |
| 13:46 |
<cgoubert@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-misc: apply |
[production] |
| 13:46 |
<cgoubert@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-misc: apply |
[production] |
| 13:45 |
<cgoubert@deploy1002> |
helmfile [codfw] [main] DONE helmfile.d/services/mw-jobrunner : sync |
[production] |
| 13:45 |
<cgoubert@deploy1002> |
helmfile [codfw] [canary] DONE helmfile.d/services/mw-jobrunner : sync |
[production] |
| 13:45 |
<cgoubert@deploy1002> |
helmfile [codfw] [canary] START helmfile.d/services/mw-jobrunner : sync |
[production] |
| 13:45 |
<cgoubert@deploy1002> |
helmfile [codfw] [main] START helmfile.d/services/mw-jobrunner : sync |
[production] |
| 13:45 |
<cgoubert@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply |
[production] |
| 13:44 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1144:3314', diff saved to https://phabricator.wikimedia.org/P50620 and previous config saved to /var/cache/conftool/dbconfig/20230821-134452-ladsgroup.json |
[production] |
| 13:44 |
<cgoubert@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply |
[production] |
| 13:44 |
<cgoubert@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply |
[production] |
| 13:43 |
<cgoubert@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-api-ext: apply |
[production] |
| 13:43 |
<cgoubert@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 13:42 |
<cgoubert@deploy1002> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 13:42 |
<claime> |
Enabling memory limit autocompute for all mw-on-k8s deployments - T342748 |
[production] |
| 13:41 |
<btullis@cumin1001> |
START - Cookbook sre.k8s.reboot-nodes rolling reboot on A:dse-k8s-worker |
[production] |
| 13:40 |
<btullis@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host kafka-stretch2001.codfw.wmnet |
[production] |
| 13:39 |
<stevemunene@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1112.eqiad.wmnet with OS bullseye |
[production] |
| 13:38 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2123 (T344589)', diff saved to https://phabricator.wikimedia.org/P50618 and previous config saved to /var/cache/conftool/dbconfig/20230821-133849-ladsgroup.json |
[production] |
| 13:35 |
<stevemunene@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1111.eqiad.wmnet with OS bullseye |
[production] |
| 13:31 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2123 (T344589)', diff saved to https://phabricator.wikimedia.org/P50617 and previous config saved to /var/cache/conftool/dbconfig/20230821-133113-ladsgroup.json |
[production] |
| 13:31 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2123.codfw.wmnet with reason: Maintenance |
[production] |