|
2022-10-20
ยง
|
| 22:12 |
<wm-bot2> |
Safe rebooting 'cloudvirt1027.eqiad.wmnet'. - cookbook ran by andrew@bullseye |
[admin] |
| 22:11 |
<wm-bot2> |
Draining 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@bullseye |
[admin] |
| 22:11 |
<wm-bot2> |
Safe rebooting 'cloudvirt1029.eqiad.wmnet'. - cookbook ran by andrew@bullseye |
[admin] |
| 22:11 |
<wm-bot2> |
Draining 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@bullseye |
[admin] |
| 22:11 |
<wm-bot2> |
Safe rebooting 'cloudvirt1030.eqiad.wmnet'. - cookbook ran by andrew@bullseye |
[admin] |
| 22:11 |
<wm-bot2> |
Draining 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@bullseye |
[admin] |
| 22:11 |
<wm-bot2> |
Safe rebooting 'cloudvirt1032.eqiad.wmnet'. - cookbook ran by andrew@bullseye |
[admin] |
| 22:10 |
<wm-bot> |
<anticomposite> ./stewardbots/StewardBot/manage.sh restart # Ping timeout not noticed by bot |
[tools.stewardbots] |
| 22:10 |
<wm-bot2> |
Safe reboot of 'cloudvirt1031.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye |
[admin] |
| 22:10 |
<wm-bot2> |
Unset cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye |
[admin] |
| 22:09 |
<wm-bot> |
<anticomposite> ./SULWatcher/manage.sh restart # all bots down |
[tools.stewardbots] |
| 22:09 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2119', diff saved to https://phabricator.wikimedia.org/P35762 and previous config saved to /var/cache/conftool/dbconfig/20221020-220924-ladsgroup.json |
[production] |
| 22:07 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P35761 and previous config saved to /var/cache/conftool/dbconfig/20221020-220722-ladsgroup.json |
[production] |
| 22:06 |
<wm-bot2> |
Drained 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@bullseye |
[admin] |
| 21:58 |
<wm-bot2> |
Safe reboot of 'cloudvirt1034.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye |
[admin] |
| 21:58 |
<wm-bot2> |
Unset cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye |
[admin] |
| 21:58 |
<wm-bot2> |
Safe reboot of 'cloudvirt1035.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye |
[admin] |
| 21:58 |
<wm-bot2> |
Unset cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye |
[admin] |
| 21:58 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti4005.mgmt.ulsfo.wmnet with reboot policy FORCED |
[production] |
| 21:57 |
<sukhe@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4037.ulsfo.wmnet with OS buster |
[production] |
| 21:55 |
<ryankemper@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: eqiad cluster reboot - ryankemper@cumin1001 - T321310 |
[production] |
| 21:55 |
<wm-bot2> |
Safe reboot of 'cloudvirt1033.eqiad.wmnet' finished successfully. - cookbook ran by andrew@bullseye |
[admin] |
| 21:55 |
<wm-bot2> |
Unset cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. - cookbook ran by andrew@bullseye |
[admin] |
| 21:54 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2119 (T321312)', diff saved to https://phabricator.wikimedia.org/P35760 and previous config saved to /var/cache/conftool/dbconfig/20221020-215418-ladsgroup.json |
[production] |
| 21:54 |
<wm-bot2> |
Drained 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@bullseye |
[admin] |
| 21:54 |
<wm-bot2> |
Drained 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@bullseye |
[admin] |
| 21:52 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1147', diff saved to https://phabricator.wikimedia.org/P35759 and previous config saved to /var/cache/conftool/dbconfig/20221020-215216-ladsgroup.json |
[production] |
| 21:51 |
<wm-bot2> |
Drained 'cloudvirt1033.eqiad.wmnet'. - cookbook ran by andrew@bullseye |
[admin] |
| 21:47 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2119 (T321312)', diff saved to https://phabricator.wikimedia.org/P35758 and previous config saved to /var/cache/conftool/dbconfig/20221020-214750-ladsgroup.json |
[production] |
| 21:47 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2119.codfw.wmnet with reason: Maintenance |
[production] |
| 21:47 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2119.codfw.wmnet with reason: Maintenance |
[production] |
| 21:47 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2110 (T321312)', diff saved to https://phabricator.wikimedia.org/P35757 and previous config saved to /var/cache/conftool/dbconfig/20221020-214725-ladsgroup.json |
[production] |
| 21:40 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.provision for host ganeti4005.mgmt.ulsfo.wmnet with reboot policy FORCED |
[production] |
| 21:38 |
<wm-bot2> |
Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance (downtime id: 0b75b44c-1efe-4edf-8f5e-42a67e8d3b13, use this to unset). - cookbook ran by andrew@bullseye |
[admin] |
| 21:38 |
<wm-bot2> |
Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance (downtime id: aeb9c3a6-961a-4873-85d4-929248aebb8b, use this to unset). - cookbook ran by andrew@bullseye |
[admin] |
| 21:38 |
<wm-bot2> |
Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance (downtime id: eb14cc04-3293-4cf8-a46f-1f87d8b0bcc4, use this to unset). - cookbook ran by andrew@bullseye |
[admin] |
| 21:37 |
<wm-bot2> |
Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance (downtime id: 84a3cd99-2643-495a-86b6-7ae80b11a30d, use this to unset). - cookbook ran by andrew@bullseye |
[admin] |
| 21:37 |
<wm-bot2> |
Draining 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@bullseye |
[admin] |
| 21:37 |
<wm-bot2> |
Safe rebooting 'cloudvirt1031.eqiad.wmnet'. - cookbook ran by andrew@bullseye |
[admin] |
| 21:37 |
<wm-bot2> |
Draining 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@bullseye |
[admin] |
| 21:37 |
<wm-bot2> |
Safe rebooting 'cloudvirt1039.eqiad.wmnet'. - cookbook ran by andrew@bullseye |
[admin] |
| 21:37 |
<wm-bot2> |
Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@bullseye |
[admin] |
| 21:37 |
<wm-bot2> |
Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@bullseye |
[admin] |
| 21:37 |
<wm-bot2> |
Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@bullseye |
[admin] |
| 21:37 |
<wm-bot2> |
Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@bullseye |
[admin] |
| 21:37 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1147 (T321312)', diff saved to https://phabricator.wikimedia.org/P35756 and previous config saved to /var/cache/conftool/dbconfig/20221020-213709-ladsgroup.json |
[production] |
| 21:36 |
<wm-bot2> |
Draining 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@bullseye |
[admin] |
| 21:36 |
<wm-bot2> |
Safe rebooting 'cloudvirt1035.eqiad.wmnet'. - cookbook ran by andrew@bullseye |
[admin] |
| 21:36 |
<wm-bot2> |
Draining 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@bullseye |
[admin] |
| 21:36 |
<wm-bot2> |
Safe rebooting 'cloudvirt1034.eqiad.wmnet'. - cookbook ran by andrew@bullseye |
[admin] |