1451-1500 of 6995 results (31ms)
2023-12-21 §
03:01 <andrewbogott> restarting mariadb on cloudcontrol1005, hoping to get Galera back in sync [admin]
2023-12-20 §
19:13 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [admin]
19:08 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.restart_openstack [admin]
2023-12-18 §
17:29 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [admin]
17:23 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.restart_openstack [admin]
17:23 <andrewbogott> restarting all eqiad1 openstack services after a rabbitmq upgrade/rebuild for T353646 [admin]
15:29 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [admin]
15:25 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.restart_openstack [admin]
2023-12-15 §
13:13 <dcaro> restarted nova-fullstack on codfw as it was stuck (and alerting through stale prometheus file) [admin]
2023-12-14 §
00:27 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=99) [admin]
00:26 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.set_maintenance [admin]
00:25 <andrewbogott> evacuating hosts from cloudvirt1063 and depooling. T353406 [admin]
2023-12-13 §
16:38 <taavi@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.scale_grid_exec (exit_code=99) [admin]
2023-12-12 §
21:01 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [admin]
20:55 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.restart_openstack [admin]
17:45 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster [admin]
17:45 <taavi@cloudcumin1001> Added a new k8s worker tools-k8s-worker-100.tools.eqiad1.wikimedia.cloud to the cluster [admin]
16:11 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster [admin]
16:11 <taavi@cloudcumin1001> Added a new k8s worker tools-k8s-worker-99.tools.eqiad1.wikimedia.cloud to the cluster [admin]
15:49 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster [admin]
15:49 <taavi@cloudcumin1001> Added a new k8s worker tools-k8s-worker-98.tools.eqiad1.wikimedia.cloud to the cluster [admin]
2023-12-10 §
18:37 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [admin]
18:33 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.restart_openstack [admin]
18:30 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [admin]
18:27 <andrewbogott> restarting all openstack API servers, hoping to make things a bit more responsive [admin]
18:25 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.restart_openstack [admin]
2023-12-08 §
12:00 <wm-bot2> dcaro@urcuchillay END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on etcd-discovery-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud (T353055) [admin]
11:58 <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.vps.refresh_puppet_certs on etcd-discovery-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud (T353055) [admin]
11:57 <wm-bot2> dcaro@urcuchillay END (ERROR) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=97) on etcd-discovery-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud [admin]
11:57 <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.vps.refresh_puppet_certs on etcd-discovery-1.cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud [admin]
09:38 <wm-bot2> dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) (T345084) [admin]
09:32 <dcaro> restarting nova and keystone as they are getting too slow (T345084) [admin]
09:32 <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.openstack.restart_openstack (T345084) [admin]
09:32 <wm-bot2> dcaro@urcuchillay END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) (T345084) [admin]
09:31 <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.openstack.restart_openstack (T345084) [admin]
2023-12-07 §
13:12 <dcaro> rebooting cloudcephosd1001 to make sure puppet7 migration went ok [admin]
2023-12-04 §
00:07 <andrewbogott> rebooting cloudcontrol1006 to recover from full disk error [admin]
2023-12-03 §
09:05 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [admin]
09:05 <taavi@cloudcumin1001> START - Cookbook wmcs.openstack.restart_openstack [admin]
2023-12-02 §
12:27 <taavi> powercycle cloudvirt1063 T352595 [admin]
11:28 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster [admin]
11:28 <taavi@cloudcumin1001> Added a new k8s worker tools-k8s-worker-97.tools.eqiad1.wikimedia.cloud to the cluster [admin]
11:02 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster [admin]
11:02 <taavi@cloudcumin1001> Added a new k8s worker tools-k8s-worker-96.tools.eqiad1.wikimedia.cloud to the cluster [admin]
10:50 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster [admin]
10:50 <taavi@cloudcumin1001> Added a new k8s worker tools-k8s-worker-95.tools.eqiad1.wikimedia.cloud to the cluster [admin]
00:21 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster [admin]
00:21 <taavi@cloudcumin1001> Added a new k8s worker tools-k8s-worker-94.tools.eqiad1.wikimedia.cloud to the cluster [admin]
00:18 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster [admin]
00:18 <taavi@cloudcumin1001> Added a new k8s worker tools-k8s-worker-93.tools.eqiad1.wikimedia.cloud to the cluster [admin]