|
2023-05-15
ยง
|
| 17:00 |
<dcaro> |
rebooting tools-sgegrid-master (T316544) |
[tools] |
| 16:55 |
<dcaro> |
rebooting tools-sgeexec-10-20 (T316544) |
[tools] |
| 16:53 |
<dcaro> |
rebooting tools-sgeweblight-10-18 (T316544) |
[tools] |
| 16:53 |
<dcaro> |
rebooting tools-sgeweblight-10-25 (T316544) |
[tools] |
| 16:53 |
<dcaro> |
rebooting tools-sgeweblight-10-20 (T316544) |
[tools] |
| 16:52 |
<dcaro> |
rebooting tools-sgeweblight-10-21 (T316544) |
[tools] |
| 16:52 |
<dcaro> |
rebooting tools-sgeexec-10-22 (T316544) |
[tools] |
| 16:51 |
<dcaro> |
rebooting tools-sgeweblight-10-28 (T316544) |
[tools] |
| 16:50 |
<dcaro> |
rebooting tools-sgeexec-10-17 (T316544) |
[tools] |
| 16:48 |
<dcaro> |
rebooting tools-sgeexec-10-21 (T316544) |
[tools] |
| 16:47 |
<dcaro> |
rebooting tools-sgeexec-10-19 (T316544) |
[tools] |
| 16:45 |
<dcaro> |
rebooting tools-sgeexec-10-8 (T316544) |
[tools] |
| 16:45 |
<dcaro> |
rebooting tools-sgeweblight-10-24 (T316544) |
[tools] |
| 16:44 |
<dcaro> |
rebooting tools-sgewebgen-10-2 (T316544) |
[tools] |
| 16:44 |
<dcaro> |
rebooting tools-sgeweblight-10-16 (T316544) |
[tools] |
| 16:43 |
<dcaro> |
rebooting tools-sgeweblight-10-30 (T316544) |
[tools] |
| 16:43 |
<dcaro> |
rebooting tools-sgeexec-10-18 (T316544) |
[tools] |
| 16:42 |
<dcaro> |
rebooting tools-sgeexec-10-16 (T316544) |
[tools] |
| 16:42 |
<dcaro> |
rebooting tools-sgeexec-10-14 (T316544) |
[tools] |
| 16:41 |
<dcaro> |
rebooting tools-sgeweblight-10-32 (T316544) |
[tools] |
| 16:40 |
<dcaro> |
rebooting tools-sgeweblight-10-22 (T316544) |
[tools] |
| 16:39 |
<dcaro> |
rebooting tools-sgeweblight-10-17 (T316544) |
[tools] |
| 16:32 |
<dcaro> |
rebooting tools-sgeexec-10-13.tools.eqiad1.wikimedia.cloud (T316544) |
[tools] |
| 16:23 |
<dcaro> |
rebooting tools-sgeweblight-10-26 (T316544) |
[tools] |
| 16:15 |
<bd808> |
Hard reboot of tools-sgebastion-11 via Horizon (done circa 16:11Z) |
[tools] |
| 16:14 |
<arturo> |
rebooted a bunch of nodes to cleanup D procs and high load avg because NFS outage (result of T316544) |
[tools] |
| 15:00 |
<aokoth@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on vrts2001.codfw.wmnet with reason: Setup Incomplete |
[production] |
| 15:00 |
<aokoth@cumin1001> |
START - Cookbook sre.hosts.downtime for 14 days, 0:00:00 on vrts2001.codfw.wmnet with reason: Setup Incomplete |
[production] |
| 14:58 |
<hauskater> |
Dropped akwiki and nawiki from CVNBot10 as closed wikis. On-wiki lists require an update. |
[cvn] |
| 14:28 |
<wm-bot2> |
Drained cloudvirt1034.eqiad.wmnet - cookbook ran by andrew@bullseye |
[admin] |
| 14:28 |
<wm-bot2> |
Set cloudvirt cloudvirt1034.eqiad.wmnet maintenance (downtime id: 96ce2ed0-3aff-4d04-be0b-e16513070617, use this to unset) - cookbook ran by andrew@bullseye |
[admin] |
| 14:27 |
<wm-bot2> |
Draining cloudvirt1034.eqiad.wmnet - cookbook ran by andrew@bullseye |
[admin] |
| 14:24 |
<bking@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on wdqs2021.codfw.wmnet with reason: testing transferpy cookbook |
[production] |
| 14:24 |
<bking@cumin1001> |
START - Cookbook sre.hosts.downtime for 14 days, 0:00:00 on wdqs2021.codfw.wmnet with reason: testing transferpy cookbook |
[production] |
| 14:23 |
<wm-bot2> |
Drained cloudvirt1027.eqiad.wmnet (T316544) - cookbook ran by dcaro@vulcanus |
[admin] |
| 14:21 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest1001.eqiad.wmnet with OS bullseye |
[production] |
| 14:20 |
<klausman@deploy1002> |
helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. |
[production] |
| 14:20 |
<klausman@deploy1002> |
helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. |
[production] |
| 14:17 |
<wm-bot2> |
Set cloudvirt cloudvirt1027.eqiad.wmnet maintenance (downtime id: 110176f8-04d5-4110-bb7d-1ab272bd8be2, use this to unset) (T316544) - cookbook ran by dcaro@vulcanus |
[admin] |
| 14:17 |
<klausman@deploy1002> |
helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 14:16 |
<wm-bot2> |
Draining cloudvirt1027.eqiad.wmnet (T316544) - cookbook ran by dcaro@vulcanus |
[admin] |
| 14:16 |
<klausman@deploy1002> |
helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 14:13 |
<wm-bot2> |
Set cloudvirt cloudvirt1033.eqiad.wmnet maintenance (downtime id: c6e92e13-49f4-4db3-8a13-8692ccfd3bc9, use this to unset) - cookbook ran by andrew@bullseye |
[admin] |
| 14:12 |
<wm-bot2> |
Draining cloudvirt1033.eqiad.wmnet - cookbook ran by andrew@bullseye |
[admin] |
| 14:12 |
<wm-bot2> |
Drained cloudvirt1035.eqiad.wmnet (T316544) - cookbook ran by dcaro@vulcanus |
[admin] |
| 14:11 |
<wm-bot2> |
Set cloudvirt cloudvirt1033.eqiad.wmnet maintenance (downtime id: fa730dec-848f-45fb-9eda-e74bd874c5c9, use this to unset) - cookbook ran by andrew@bullseye |
[admin] |
| 14:10 |
<wm-bot2> |
Draining cloudvirt1033.eqiad.wmnet - cookbook ran by andrew@bullseye |
[admin] |
| 14:06 |
<wm-bot2> |
Restarting openstack services on cloudbackup2001: ['cinder-backup'] - cookbook ran by andrew@bullseye |
[admin] |
| 14:06 |
<wm-bot2> |
Restarting openstack services on cloudcontrol1006: ['cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye |
[admin] |
| 14:06 |
<wm-bot2> |
Restarting openstack services on cloudcontrol1007: ['cinder-volume', 'cinder-scheduler'] - cookbook ran by andrew@bullseye |
[admin] |