|
2022-03-29
ยง
|
| 18:57 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host idp-test1001.wikimedia.org |
[production] |
| 18:57 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T298565)', diff saved to https://phabricator.wikimedia.org/P23602 and previous config saved to /var/cache/conftool/dbconfig/20220329-185733-ladsgroup.json |
[production] |
| 18:55 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1105:3312 (T298565)', diff saved to https://phabricator.wikimedia.org/P23601 and previous config saved to /var/cache/conftool/dbconfig/20220329-185526-ladsgroup.json |
[production] |
| 18:55 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:55 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1105.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:54 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1129 (T298565)', diff saved to https://phabricator.wikimedia.org/P23600 and previous config saved to /var/cache/conftool/dbconfig/20220329-185454-ladsgroup.json |
[production] |
| 18:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P23599 and previous config saved to /var/cache/conftool/dbconfig/20220329-184720-marostegui.json |
[production] |
| 18:45 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:45 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:39 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P23598 and previous config saved to /var/cache/conftool/dbconfig/20220329-183949-ladsgroup.json |
[production] |
| 18:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1135 (T300775)', diff saved to https://phabricator.wikimedia.org/P23597 and previous config saved to /var/cache/conftool/dbconfig/20220329-183215-marostegui.json |
[production] |
| 18:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depooling db1131 (T297189)', diff saved to https://phabricator.wikimedia.org/P23596 and previous config saved to /var/cache/conftool/dbconfig/20220329-183041-marostegui.json |
[production] |
| 18:30 |
<marostegui@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1131.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:30 |
<marostegui@cumin1001> |
START - Cookbook sre.hosts.downtime for 8:00:00 on db1131.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T297189)', diff saved to https://phabricator.wikimedia.org/P23595 and previous config saved to /var/cache/conftool/dbconfig/20220329-183034-marostegui.json |
[production] |
| 18:24 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1129', diff saved to https://phabricator.wikimedia.org/P23594 and previous config saved to /var/cache/conftool/dbconfig/20220329-182444-ladsgroup.json |
[production] |
| 18:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P23593 and previous config saved to /var/cache/conftool/dbconfig/20220329-181529-marostegui.json |
[production] |
| 18:09 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1129 (T298565)', diff saved to https://phabricator.wikimedia.org/P23592 and previous config saved to /var/cache/conftool/dbconfig/20220329-180938-ladsgroup.json |
[production] |
| 18:05 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:05 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:02 |
<moritzm> |
restarting fpm on mw canaries to pick up new libtiff |
[production] |
| 18:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3316', diff saved to https://phabricator.wikimedia.org/P23591 and previous config saved to /var/cache/conftool/dbconfig/20220329-180023-marostegui.json |
[production] |
| 17:47 |
<moritzm> |
installing tiff security updates |
[production] |
| 17:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1098:3316 (T297189)', diff saved to https://phabricator.wikimedia.org/P23590 and previous config saved to /var/cache/conftool/dbconfig/20220329-174518-marostegui.json |
[production] |
| 17:29 |
<mutante> |
gitlab2001 - systemctl reset-failed |
[production] |
| 17:23 |
<mutante> |
gitlab2001 - did not come back from reboot via cookbook. logged in via console. then "s/ens5/ens13" in /etc/network/interfaces ; reboot ; issue was like T272555 and others |
[production] |
| 17:22 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:22 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1171.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:13 |
<razzi@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on an-tool1005.eqiad.wmnet with reason: Testing deploy of superset 1.4.2 to staging |
[production] |
| 17:13 |
<razzi@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on an-tool1005.eqiad.wmnet with reason: Testing deploy of superset 1.4.2 to staging |
[production] |
| 17:11 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores2003.codfw.wmnet |
[production] |
| 17:09 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1129 (T298565)', diff saved to https://phabricator.wikimedia.org/P23589 and previous config saved to /var/cache/conftool/dbconfig/20220329-170924-ladsgroup.json |
[production] |
| 17:09 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:09 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1129.eqiad.wmnet with reason: Maintenance |
[production] |
| 17:09 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1105:3312 (T298565)', diff saved to https://phabricator.wikimedia.org/P23588 and previous config saved to /var/cache/conftool/dbconfig/20220329-170916-ladsgroup.json |
[production] |
| 17:04 |
<klausman@cumin2002> |
END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) for host ml-staging2001.codfw.wmnet |
[production] |
| 17:04 |
<klausman@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ml-staging2001.codfw.wmnet |
[production] |
| 17:03 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host ores2003.codfw.wmnet |
[production] |
| 17:03 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores2002.codfw.wmnet |
[production] |
| 17:00 |
<bking@cumin1001> |
START - Cookbook sre.wdqs.reboot |
[production] |
| 17:00 |
<aokoth@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.reboot-vm (exit_code=99) for VM gitlab2001.wikimedia.org |
[production] |
| 16:55 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host ores2002.codfw.wmnet |
[production] |
| 16:54 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P23586 and previous config saved to /var/cache/conftool/dbconfig/20220329-165411-ladsgroup.json |
[production] |
| 16:51 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ores2001.codfw.wmnet |
[production] |
| 16:45 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host ores2001.codfw.wmnet |
[production] |
| 16:39 |
<hashar@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/blubberoid: apply |
[production] |
| 16:39 |
<hashar@deploy1002> |
helmfile [codfw] START helmfile.d/services/blubberoid: apply |
[production] |
| 16:39 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1105:3312', diff saved to https://phabricator.wikimedia.org/P23584 and previous config saved to /var/cache/conftool/dbconfig/20220329-163906-ladsgroup.json |
[production] |
| 16:39 |
<hashar@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/blubberoid: apply |
[production] |
| 16:38 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1003.eqiad.wmnet with reason: Maintenance |
[production] |