|
2023-08-28
ยง
|
| 13:30 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on es1030.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:30 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance es1026 (T344589)', diff saved to https://phabricator.wikimedia.org/P51629 and previous config saved to /var/cache/conftool/dbconfig/20230828-133016-ladsgroup.json |
[production] |
| 13:27 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2153 (T343718)', diff saved to https://phabricator.wikimedia.org/P51628 and previous config saved to /var/cache/conftool/dbconfig/20230828-132724-ladsgroup.json |
[production] |
| 13:27 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2153.codfw.wmnet with reason: Maintenance |
[production] |
| 13:27 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2153.codfw.wmnet with reason: Maintenance |
[production] |
| 13:27 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2146 (T343718)', diff saved to https://phabricator.wikimedia.org/P51627 and previous config saved to /var/cache/conftool/dbconfig/20230828-132703-ladsgroup.json |
[production] |
| 13:26 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1168 (T344589)', diff saved to https://phabricator.wikimedia.org/P51626 and previous config saved to /var/cache/conftool/dbconfig/20230828-132655-ladsgroup.json |
[production] |
| 13:26 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1168.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:26 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2124 (T344589)', diff saved to https://phabricator.wikimedia.org/P51625 and previous config saved to /var/cache/conftool/dbconfig/20230828-132648-ladsgroup.json |
[production] |
| 13:26 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2124.codfw.wmnet with reason: Maintenance |
[production] |
| 13:26 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1168.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:26 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1165 (T344589)', diff saved to https://phabricator.wikimedia.org/P51624 and previous config saved to /var/cache/conftool/dbconfig/20230828-132632-ladsgroup.json |
[production] |
| 13:26 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2124.codfw.wmnet with reason: Maintenance |
[production] |
| 13:26 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2117 (T344589)', diff saved to https://phabricator.wikimedia.org/P51623 and previous config saved to /var/cache/conftool/dbconfig/20230828-132623-ladsgroup.json |
[production] |
| 13:24 |
<fabfur> |
enable puppet and start pybal on lvs5004 (T344587) |
[production] |
| 13:23 |
<fabfur@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs5004.eqsin.wmnet |
[production] |
| 13:22 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P51622 and previous config saved to /var/cache/conftool/dbconfig/20230828-132250-ladsgroup.json |
[production] |
| 13:22 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2025.codfw.wmnet |
[production] |
| 13:20 |
<fabfur@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host lvs5004.eqsin.wmnet |
[production] |
| 13:19 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2024.codfw.wmnet |
[production] |
| 13:18 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2024.codfw.wmnet |
[production] |
| 13:15 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance es1026', diff saved to https://phabricator.wikimedia.org/P51621 and previous config saved to /var/cache/conftool/dbconfig/20230828-131510-ladsgroup.json |
[production] |
| 13:14 |
<urbanecm@deploy1002> |
Finished scap: Backport for [[gerrit:952808|Revert "ltwiki: Disable Growth features" (T344013)]] (duration: 09m 04s) |
[production] |
| 13:12 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2024.codfw.wmnet |
[production] |
| 13:11 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P51620 and previous config saved to /var/cache/conftool/dbconfig/20230828-131157-ladsgroup.json |
[production] |
| 13:11 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P51619 and previous config saved to /var/cache/conftool/dbconfig/20230828-131125-ladsgroup.json |
[production] |
| 13:11 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2024.codfw.wmnet |
[production] |
| 13:11 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P51618 and previous config saved to /var/cache/conftool/dbconfig/20230828-131117-ladsgroup.json |
[production] |
| 13:08 |
<urbanecm@deploy1002> |
urbanecm: Continuing with sync |
[production] |
| 13:08 |
<jmm@cumin2002> |
END (FAIL) - Cookbook sre.ganeti.drain-node (exit_code=99) for draining ganeti node ganeti2023.codfw.wmnet |
[production] |
| 13:07 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2023.codfw.wmnet |
[production] |
| 13:07 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P51617 and previous config saved to /var/cache/conftool/dbconfig/20230828-130744-ladsgroup.json |
[production] |
| 13:06 |
<urbanecm@deploy1002> |
urbanecm: Backport for [[gerrit:952808|Revert "ltwiki: Disable Growth features" (T344013)]] synced to the testservers mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option) |
[production] |
| 13:04 |
<urbanecm@deploy1002> |
Started scap: Backport for [[gerrit:952808|Revert "ltwiki: Disable Growth features" (T344013)]] |
[production] |
| 13:02 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2022.codfw.wmnet |
[production] |
| 13:02 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2022.codfw.wmnet |
[production] |
| 13:01 |
<bblack> |
esams cp clusters: rolling restarts of varnish-frontend ~1h apart over the next ~8h, to apply memory sizing change from: https://gerrit.wikimedia.org/r/c/operations/puppet/+/952555/ |
[production] |
| 13:01 |
<fabfur> |
disable puppet and stop pybal on lvs5004 for reboot (T344587) |
[production] |
| 13:00 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1139.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:00 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1139.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:00 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1135 (T343718)', diff saved to https://phabricator.wikimedia.org/P51616 and previous config saved to /var/cache/conftool/dbconfig/20230828-130012-ladsgroup.json |
[production] |
| 13:00 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance es1026', diff saved to https://phabricator.wikimedia.org/P51615 and previous config saved to /var/cache/conftool/dbconfig/20230828-130004-ladsgroup.json |
[production] |
| 12:56 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P51614 and previous config saved to /var/cache/conftool/dbconfig/20230828-125651-ladsgroup.json |
[production] |
| 12:56 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P51613 and previous config saved to /var/cache/conftool/dbconfig/20230828-125619-ladsgroup.json |
[production] |
| 12:56 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2117', diff saved to https://phabricator.wikimedia.org/P51612 and previous config saved to /var/cache/conftool/dbconfig/20230828-125610-ladsgroup.json |
[production] |
| 12:55 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2022.codfw.wmnet |
[production] |
| 12:52 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1221 (T344589)', diff saved to https://phabricator.wikimedia.org/P51611 and previous config saved to /var/cache/conftool/dbconfig/20230828-125237-ladsgroup.json |
[production] |
| 12:49 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2022.codfw.wmnet |
[production] |
| 12:46 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host puppetserver1002 |
[production] |
| 12:45 |
<jbond@cumin1001> |
START - Cookbook sre.network.configure-switch-interfaces for host puppetserver1002 |
[production] |