|
2022-03-29
ยง
|
| 17:04 |
<klausman@cumin2002> |
END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) for host ml-staging2001.codfw.wmnet |
[production] |
| 17:04 |
<klausman@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ml-staging2001.codfw.wmnet |
[production] |
| 17:03 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host ores2003.codfw.wmnet |
[production] |
| 17:03 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores2002.codfw.wmnet |
[production] |
| 17:00 |
<bking@cumin1001> |
START - Cookbook sre.wdqs.reboot |
[production] |
| 17:00 |
<aokoth@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.reboot-vm (exit_code=99) for VM gitlab2001.wikimedia.org |
[production] |
| 16:55 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host ores2002.codfw.wmnet |
[production] |
| 16:54 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P23586 and previous config saved to /var/cache/conftool/dbconfig/20220329-165411-ladsgroup.json |
[production] |
| 16:51 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores2001.codfw.wmnet |
[production] |
| 16:45 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host ores2001.codfw.wmnet |
[production] |
| 16:39 |
<hashar@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/blubberoid: apply |
[production] |
| 16:39 |
<hashar@deploy1002> |
helmfile [codfw] START helmfile.d/services/blubberoid: apply |
[production] |
| 16:39 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P23584 and previous config saved to /var/cache/conftool/dbconfig/20220329-163906-ladsgroup.json |
[production] |
| 16:39 |
<hashar@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/blubberoid: apply |
[production] |
| 16:38 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:38 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:38 |
<hashar@deploy1002> |
helmfile [eqiad] START helmfile.d/services/blubberoid: apply |
[production] |
| 16:38 |
<hashar@deploy1002> |
helmfile [staging] DONE helmfile.d/services/blubberoid: apply |
[production] |
| 16:37 |
<hashar@deploy1002> |
helmfile [staging] START helmfile.d/services/blubberoid: apply |
[production] |
| 16:35 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1098:3316 (T297189)', diff saved to https://phabricator.wikimedia.org/P23583 and previous config saved to /var/cache/conftool/dbconfig/20220329-163503-marostegui.json |
[production] |
| 16:35 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1098.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:35 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1098.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:34 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1180 (T297189)', diff saved to https://phabricator.wikimedia.org/P23582 and previous config saved to /var/cache/conftool/dbconfig/20220329-163455-marostegui.json |
[production] |
| 16:24 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T298565)', diff saved to https://phabricator.wikimedia.org/P23581 and previous config saved to /var/cache/conftool/dbconfig/20220329-162401-ladsgroup.json |
[production] |
| 16:21 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1105:3312 (T298565)', diff saved to https://phabricator.wikimedia.org/P23580 and previous config saved to /var/cache/conftool/dbconfig/20220329-162153-ladsgroup.json |
[production] |
| 16:21 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:21 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance |
[production] |
| 16:21 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1182 (T298565)', diff saved to https://phabricator.wikimedia.org/P23579 and previous config saved to /var/cache/conftool/dbconfig/20220329-162146-ladsgroup.json |
[production] |
| 16:21 |
<bking@cumin1001> |
END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99) |
[production] |
| 16:19 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P23578 and previous config saved to /var/cache/conftool/dbconfig/20220329-161950-marostegui.json |
[production] |
| 16:19 |
<aokoth@cumin1001> |
START - Cookbook sre.ganeti.reboot-vm for VM gitlab2001.wikimedia.org |
[production] |
| 16:17 |
<aokoth@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.reboot-vm (exit_code=99) for VM gitlab2001.wikimedia.org |
[production] |
| 16:17 |
<aokoth@cumin1001> |
START - Cookbook sre.ganeti.reboot-vm for VM gitlab2001.wikimedia.org |
[production] |
| 16:06 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P23577 and previous config saved to /var/cache/conftool/dbconfig/20220329-160640-ladsgroup.json |
[production] |
| 16:04 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P23576 and previous config saved to /var/cache/conftool/dbconfig/20220329-160446-marostegui.json |
[production] |
| 15:54 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 10 hosts with reason: Maintenance |
[production] |
| 15:54 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 12:00:00 on 10 hosts with reason: Maintenance |
[production] |
| 15:54 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance |
[production] |
| 15:54 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db2121.codfw.wmnet with reason: Maintenance |
[production] |
| 15:54 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1158 (T298565)', diff saved to https://phabricator.wikimedia.org/P23575 and previous config saved to /var/cache/conftool/dbconfig/20220329-155415-ladsgroup.json |
[production] |
| 15:51 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P23574 and previous config saved to /var/cache/conftool/dbconfig/20220329-155135-ladsgroup.json |
[production] |
| 15:49 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1180 (T297189)', diff saved to https://phabricator.wikimedia.org/P23573 and previous config saved to /var/cache/conftool/dbconfig/20220329-154941-marostegui.json |
[production] |
| 15:47 |
<jayme@deploy1002> |
Finished deploy [restbase/deploy@0848b15] (dev-cluster): (no justification provided) (duration: 00m 18s) |
[production] |
| 15:47 |
<jayme@deploy1002> |
Started deploy [restbase/deploy@0848b15] (dev-cluster): (no justification provided) |
[production] |
| 15:46 |
<jayme> |
updated scap to 4.5.0 on canary hosts - T304134 |
[production] |
| 15:43 |
<jayme> |
imported scap 4.5.0 to strets-/buster-/bullseye-wikimedia - T304134 |
[production] |
| 15:39 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P23572 and previous config saved to /var/cache/conftool/dbconfig/20220329-153910-ladsgroup.json |
[production] |
| 15:36 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1182 (T298565)', diff saved to https://phabricator.wikimedia.org/P23571 and previous config saved to /var/cache/conftool/dbconfig/20220329-153630-ladsgroup.json |
[production] |
| 15:34 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1182 (T298565)', diff saved to https://phabricator.wikimedia.org/P23570 and previous config saved to /var/cache/conftool/dbconfig/20220329-153423-ladsgroup.json |
[production] |
| 15:34 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance |
[production] |