6301-6350 of 10000 results (47ms)
2023-12-04 §
00:07 <andrewbogott> rebooting cloudcontrol1006 to recover from full disk error [admin]
2023-12-03 §
12:39 <wm-bot> <lucaswerkmeister> deployed da71d5402e (force English for wbformatvalue+wbformatentities, T345881 [tools.ranker]
09:05 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [admin]
09:05 <taavi@cloudcumin1001> START - Cookbook wmcs.openstack.restart_openstack [admin]
08:58 <taavi> webservice restart [toolsbeta.fourohfour]
2023-12-02 §
13:42 <bd808> Restarted bot. Was missing from IRC channels (possibly all channels). Likely fallout from Kubernetes cluster issues yesterday. [tools.bridgebot]
12:27 <taavi> powercycle cloudvirt1063 T352595 [admin]
11:28 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster [admin]
11:28 <taavi@cloudcumin1001> Added a new k8s worker tools-k8s-worker-97.tools.eqiad1.wikimedia.cloud to the cluster [admin]
11:18 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster [tools]
11:15 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-22 [tools]
11:06 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeexec-10-13, tools-sgeweblight-10-20 [tools]
11:02 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster [admin]
11:02 <taavi@cloudcumin1001> Added a new k8s worker tools-k8s-worker-96.tools.eqiad1.wikimedia.cloud to the cluster [admin]
10:50 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster [tools]
10:50 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster [admin]
10:50 <taavi@cloudcumin1001> Added a new k8s worker tools-k8s-worker-95.tools.eqiad1.wikimedia.cloud to the cluster [admin]
10:39 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster [tools]
01:51 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be1078.eqiad.wmnet with OS bullseye [production]
01:50 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be1079.eqiad.wmnet with OS bullseye [production]
01:50 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be1077.eqiad.wmnet with OS bullseye [production]
01:50 <jclark@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be1076.eqiad.wmnet with OS bullseye [production]
00:30 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host ms-be1078.eqiad.wmnet with OS bullseye [production]
00:30 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host ms-be1079.eqiad.wmnet with OS bullseye [production]
00:30 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host ms-be1077.eqiad.wmnet with OS bullseye [production]
00:30 <jclark@cumin1001> START - Cookbook sre.hosts.reimage for host ms-be1076.eqiad.wmnet with OS bullseye [production]
00:21 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster [admin]
00:21 <taavi@cloudcumin1001> Added a new k8s worker tools-k8s-worker-94.tools.eqiad1.wikimedia.cloud to the cluster [admin]
00:18 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster [admin]
00:18 <taavi@cloudcumin1001> Added a new k8s worker tools-k8s-worker-93.tools.eqiad1.wikimedia.cloud to the cluster [admin]
00:15 <jclark@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ms-be1076'] [production]
00:15 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster [admin]
00:15 <taavi@cloudcumin1001> Added a new k8s worker tools-k8s-worker-92.tools.eqiad1.wikimedia.cloud to the cluster [admin]
00:15 <jclark@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ms-be1076'] [production]
00:14 <jclark@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ms-be1076'] [production]
00:14 <jclark@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ms-be1076'] [production]
00:14 <jclark@cumin1001> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['ms-be1076'] [production]
00:14 <jclark@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ms-be1076'] [production]
00:14 <jclark@cumin1001> END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['ms-be1076'] [production]
00:14 <jclark@cumin1001> END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['ms-be1076'] [production]
00:14 <jclark@cumin1001> END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['ms-be1076'] [production]
00:14 <jclark@cumin1001> END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ['ms-be1076'] [production]
00:14 <jclark@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ms-be1076'] [production]
00:14 <jclark@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ms-be1076'] [production]
00:13 <jclark@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ms-be1076'] [production]
00:13 <jclark@cumin1001> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ms-be1076'] [production]
00:13 <jclark@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1079.mgmt.eqiad.wmnet with reboot policy FORCED [production]
00:13 <jclark@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1078.mgmt.eqiad.wmnet with reboot policy FORCED [production]
00:13 <jclark@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1077.mgmt.eqiad.wmnet with reboot policy FORCED [production]
00:12 <jclark@cumin1001> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be1076.mgmt.eqiad.wmnet with reboot policy FORCED [production]