|
2022-10-24
ยง
|
| 10:01 |
<mvernon@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host ms-be1049.eqiad.wmnet |
[production] |
| 09:59 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2145', diff saved to https://phabricator.wikimedia.org/P35999 and previous config saved to /var/cache/conftool/dbconfig/20221024-095946-ladsgroup.json |
[production] |
| 09:58 |
<mvernon@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1048.eqiad.wmnet |
[production] |
| 09:57 |
<mvernon@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host ms-be2052.codfw.wmnet |
[production] |
| 09:56 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1106', diff saved to https://phabricator.wikimedia.org/P35998 and previous config saved to /var/cache/conftool/dbconfig/20221024-095559-ladsgroup.json |
[production] |
| 09:53 |
<mvernon@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2051.codfw.wmnet |
[production] |
| 09:44 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2145', diff saved to https://phabricator.wikimedia.org/P35997 and previous config saved to /var/cache/conftool/dbconfig/20221024-094440-ladsgroup.json |
[production] |
| 09:43 |
<mvernon@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host ms-be1048.eqiad.wmnet |
[production] |
| 09:41 |
<cgoubert@cumin1001> |
conftool action : set/pooled=yes; selector: dc=codfw,cluster=kubernetes,service=kubemaster,name=kubemaster2002.codfw.wmnet |
[production] |
| 09:41 |
<mvernon@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host ms-be2051.codfw.wmnet |
[production] |
| 09:41 |
<cgoubert@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kubemaster2002.codfw.wmnet |
[production] |
| 09:40 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1106 (T321312)', diff saved to https://phabricator.wikimedia.org/P35996 and previous config saved to /var/cache/conftool/dbconfig/20221024-094052-ladsgroup.json |
[production] |
| 09:35 |
<cgoubert@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host kubemaster2002.codfw.wmnet |
[production] |
| 09:35 |
<cgoubert@cumin1001> |
conftool action : set/pooled=no; selector: dc=codfw,cluster=kubernetes,service=kubemaster,name=kubemaster2002.codfw.wmnet |
[production] |
| 09:34 |
<cgoubert@cumin1001> |
conftool action : set/pooled=yes; selector: dc=codfw,cluster=kubernetes,service=kubemaster,name=kubemaster2001.codfw.wmnet |
[production] |
| 09:34 |
<cgoubert@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kubemaster2001.codfw.wmnet |
[production] |
| 09:29 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2145 (T321312)', diff saved to https://phabricator.wikimedia.org/P35995 and previous config saved to /var/cache/conftool/dbconfig/20221024-092933-ladsgroup.json |
[production] |
| 09:27 |
<cgoubert@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host kubemaster2001.codfw.wmnet |
[production] |
| 09:27 |
<cgoubert@cumin1001> |
conftool action : set/pooled=no; selector: dc=codfw,cluster=kubernetes,service=kubemaster,name=kubemaster2001.codfw.wmnet |
[production] |
| 09:25 |
<cgoubert@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kubernetes2016.codfw.wmnet |
[production] |
| 09:23 |
<oblivian@deploy1002> |
Finished scap: Backport for [[gerrit:839499|Stop assigning the PHP_ENGINE cookie (T271736)]] (duration: 04m 59s) |
[production] |
| 09:23 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2145 (T321312)', diff saved to https://phabricator.wikimedia.org/P35994 and previous config saved to /var/cache/conftool/dbconfig/20221024-092310-ladsgroup.json |
[production] |
| 09:23 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2145.codfw.wmnet with reason: Maintenance |
[production] |
| 09:22 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2145.codfw.wmnet with reason: Maintenance |
[production] |
| 09:20 |
<cgoubert@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host kubernetes2016.codfw.wmnet |
[production] |
| 09:19 |
<cgoubert@cumin1001> |
conftool action : set/pooled=no; selector: dc=codfw,cluster=kubernetes,service=kubesvc,name=kubernetes2016.codfw.wmnet |
[production] |
| 09:19 |
<cgoubert@cumin1001> |
conftool action : set/pooled=yes; selector: dc=codfw,cluster=kubernetes,service=kubesvc,name=kubernetes2015.codfw.wmnet |
[production] |
| 09:18 |
<oblivian@deploy1002> |
oblivian and oblivian: Backport for [[gerrit:839499|Stop assigning the PHP_ENGINE cookie (T271736)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet |
[production] |
| 09:18 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2141.codfw.wmnet with reason: Maintenance |
[production] |
| 09:18 |
<cgoubert@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kubernetes2015.codfw.wmnet |
[production] |
| 09:18 |
<oblivian@deploy1002> |
Started scap: Backport for [[gerrit:839499|Stop assigning the PHP_ENGINE cookie (T271736)]] |
[production] |
| 09:18 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2141.codfw.wmnet with reason: Maintenance |
[production] |
| 09:18 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2130 (T321312)', diff saved to https://phabricator.wikimedia.org/P35993 and previous config saved to /var/cache/conftool/dbconfig/20221024-091801-ladsgroup.json |
[production] |
| 09:17 |
<claime> |
kubernetes2015:~$ sudo systemctl reset-failed ifup@ens13.service T273026 |
[production] |
| 09:15 |
<mvernon@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host ms-be2051.codfw.wmnet |
[production] |
| 09:12 |
<cgoubert@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host kubernetes2015.codfw.wmnet |
[production] |
| 09:11 |
<cgoubert@cumin1001> |
conftool action : set/pooled=no; selector: dc=codfw,cluster=kubernetes,service=kubesvc,name=kubernetes2015.codfw.wmnet |
[production] |
| 09:10 |
<mvernon@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1047.eqiad.wmnet |
[production] |
| 09:09 |
<claime> |
Starting october reboots of lingering wikikube codfw hosts |
[production] |
| 09:08 |
<cgoubert@cumin1001> |
conftool action : set/pooled=yes; selector: dc=eqiad,cluster=kubernetes,service=kubemaster,name=kubemaster1002.eqiad.wmnet |
[production] |
| 09:08 |
<cgoubert@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kubemaster1002.eqiad.wmnet |
[production] |
| 09:03 |
<mvernon@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host ms-be1047.eqiad.wmnet |
[production] |
| 09:02 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2130', diff saved to https://phabricator.wikimedia.org/P35991 and previous config saved to /var/cache/conftool/dbconfig/20221024-090255-ladsgroup.json |
[production] |
| 09:01 |
<cgoubert@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host kubemaster1002.eqiad.wmnet |
[production] |
| 09:01 |
<cgoubert@cumin1001> |
conftool action : set/pooled=no; selector: dc=eqiad,cluster=kubernetes,service=kubemaster,name=kubemaster1002.eqiad.wmnet |
[production] |
| 09:01 |
<mvernon@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1046.eqiad.wmnet |
[production] |
| 09:01 |
<mvernon@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host ms-be2051.codfw.wmnet |
[production] |
| 09:00 |
<mvernon@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be2050.codfw.wmnet |
[production] |
| 09:00 |
<cgoubert@cumin1001> |
conftool action : set/pooled=yes; selector: dc=eqiad,cluster=kubernetes,service=kubemaster,name=kubemaster1001.eqiad.wmnet |
[production] |
| 08:59 |
<cgoubert@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kubemaster1001.eqiad.wmnet |
[production] |