1701-1750 of 10000 results (27ms)
2023-05-05 ยง
16:10 <andrew@cumin1001> START - Cookbook sre.hosts.decommission for hosts cloudvirt1024.eqiad.wmnet [production]
16:10 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs2011.codfw.wmnet with OS bullseye [production]
16:10 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host lvs2011.codfw.wmnet with OS bullseye [production]
16:08 <andrew@cumin1001> START - Cookbook sre.hosts.decommission for hosts cloudvirt1023.eqiad.wmnet [production]
16:07 <wm-bot2> Drained cloudvirt1024.eqiad.wmnet (T336064) - cookbook ran by andrew@bullseye [admin]
16:06 <btullis@cumin1001> END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) [production]
16:06 <btullis@cumin1001> Added views for new wiki: zhwiki T334041 [production]
16:03 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs2011.codfw.wmnet with OS bullseye [production]
16:03 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host lvs2011.codfw.wmnet with OS bullseye [production]
16:03 <wm-bot2> Set cloudvirt cloudvirt1024.eqiad.wmnet maintenance (downtime id: 95995009-09d6-496e-8cd2-0cfac93d3cf7, use this to unset) (T336064) - cookbook ran by andrew@bullseye [admin]
16:02 <wm-bot2> Draining cloudvirt1024.eqiad.wmnet (T336064) - cookbook ran by andrew@bullseye [admin]
16:01 <wm-bot2> Drained cloudvirt1023.eqiad.wmnet (T336064) - cookbook ran by andrew@bullseye [admin]
16:00 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lvs2011.codfw.wmnet with OS bullseye [production]
16:00 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host lvs2011.codfw.wmnet with OS bullseye [production]
15:51 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts cloudvirt1020.eqiad.wmnet [production]
15:51 <andrew@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:51 <wm-bot2> Set cloudvirt cloudvirt1023.eqiad.wmnet maintenance (downtime id: 53c46cae-00af-4664-97ff-266b393335bb, use this to unset) (T336064) - cookbook ran by andrew@bullseye [admin]
15:50 <andrew@cumin1001> START - Cookbook sre.dns.netbox [production]
15:50 <wm-bot2> Draining cloudvirt1023.eqiad.wmnet (T336064) - cookbook ran by andrew@bullseye [admin]
15:49 <wm-bot2> Set cloudvirt cloudvirt1024.eqiad.wmnet maintenance (downtime id: 528ea4f6-8088-475e-937f-098ffba861b6, use this to unset) - cookbook ran by andrew@bullseye [admin]
15:48 <btullis@cumin1001> END (PASS) - Cookbook sre.presto.reboot-workers (exit_code=0) for Presto analytics cluster: Reboot Presto nodes [production]
15:47 <wm-bot2> Set cloudvirt cloudvirt1023.eqiad.wmnet maintenance (downtime id: 3ef85b5e-d9d9-4b24-901b-a3058a7d0615, use this to unset) - cookbook ran by andrew@bullseye [admin]
15:44 <andrewbogott> moved cloudvirt1023 and cloudvirt1024 from 'ceph' aggregate to 'maintenance' aggregate, prep for decom T336064 [admin]
15:44 <mforns> re-ran projectview_hourly DAG for 2023-05-05T13 [analytics]
15:44 <andrewbogott> moved cloudvirt1028 from 'localdisk' aggregate to 'maintenance' aggregate. Nothing new should be scheduled here, local storage should now move to cloudvirtlocal100x [admin]
15:42 <andrew@cumin1001> START - Cookbook sre.hosts.decommission for hosts cloudvirt1020.eqiad.wmnet [production]
15:41 <andrewbogott> moved cloudvirt1055 and cloudvirt1056 from 'spare' to 'ceph' aggregate. Prep for removing two obsolete cloudvirts, 1023 and 1024. T336064 [admin]
15:41 <btullis@cumin1001> START - Cookbook sre.wikireplicas.add-wiki [production]
15:41 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts cloudvirt1020.eqiad.wmnet [production]
15:41 <andrew@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:41 <andrew@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1020.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1001" [production]
15:40 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudvirt1019.eqiad.wmnet [production]
15:40 <andrew@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:40 <andrew@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudvirt1020.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1001" [production]
15:39 <andrew@cumin1001> START - Cookbook sre.dns.netbox [production]
15:37 <andrew@cumin1001> START - Cookbook sre.dns.netbox [production]
15:27 <andrew@cumin1001> START - Cookbook sre.hosts.decommission for hosts cloudvirt1019.eqiad.wmnet [production]
15:27 <andrew@cumin1001> START - Cookbook sre.hosts.decommission for hosts cloudvirt1020.eqiad.wmnet [production]
15:22 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1225.eqiad.wmnet with reason: Maintenance [production]
15:22 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1225.eqiad.wmnet with reason: Maintenance [production]
15:22 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1222 (T335845)', diff saved to https://phabricator.wikimedia.org/P47778 and previous config saved to /var/cache/conftool/dbconfig/20230505-152222-ladsgroup.json [production]
15:07 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P47777 and previous config saved to /var/cache/conftool/dbconfig/20230505-150716-ladsgroup.json [production]
15:06 <mforns> deployed airflow analytics [analytics]
15:06 <mforns@deploy1002> Finished deploy [airflow-dags/analytics@11fa4e1]: (no justification provided) (duration: 00m 13s) [production]
15:06 <mforns@deploy1002> Started deploy [airflow-dags/analytics@11fa4e1]: (no justification provided) [production]
14:52 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1222', diff saved to https://phabricator.wikimedia.org/P47776 and previous config saved to /var/cache/conftool/dbconfig/20230505-145209-ladsgroup.json [production]
14:37 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1222 (T335845)', diff saved to https://phabricator.wikimedia.org/P47774 and previous config saved to /var/cache/conftool/dbconfig/20230505-143703-ladsgroup.json [production]
14:30 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cuminunpriv1001.eqiad.wmnet [production]
14:29 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1222 (T335845)', diff saved to https://phabricator.wikimedia.org/P47773 and previous config saved to /var/cache/conftool/dbconfig/20230505-142940-ladsgroup.json [production]
14:29 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1222.eqiad.wmnet with reason: Maintenance [production]