651-700 of 5159 results (29ms)
2023-12-12 §
13:15 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [tools]
12:17 <wm-bot2> dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api (T352774) [tools]
12:16 <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api (T352774) [tools]
2023-12-11 §
15:36 <wm-bot2> dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api (T352774) [tools]
15:36 <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api (T352774) [tools]
13:43 <wm-bot2> dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api (T352774) [tools]
13:42 <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api (T352774) [tools]
13:28 <wm-bot2> dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission (T352774) [tools]
13:28 <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission (T352774) [tools]
2023-12-09 §
16:45 <dcaro> set toolsdb back as read-write [tools]
16:35 <andrewbogott> rebooting tools-db-1.tools.eqiad1.wikimedia.cloud yet again [tools]
07:23 <dcaro> set toolsdb back as read-write [tools]
00:54 <taavi> set toolsdb back as read-write [tools]
2023-12-08 §
11:03 <sstefanova@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api [tools]
11:03 <sstefanova@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api [tools]
2023-12-07 §
04:42 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-26 [tools]
2023-12-05 §
21:09 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) [tools]
21:09 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors [tools]
19:16 <andrewbogott> rebooting tools-sgeweblight-10-26.tools.eqiad1.wikimedia.cloud; can't log in even with root key [tools]
11:25 <wm-bot2> dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) [tools]
11:21 <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console [tools]
11:20 <wm-bot2> dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) [tools]
11:20 <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console [tools]
11:20 <wm-bot2> dcaro@urcuchillay END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) [tools]
11:20 <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console [tools]
11:20 <wm-bot2> dcaro@urcuchillay END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) [tools]
11:20 <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console [tools]
11:15 <wm-bot2> dcaro@urcuchillay END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) [tools]
11:15 <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.openstack.cloudvirt.vm_console [tools]
11:01 <dcaro> rebooting tools-sgeweblight-10-25 due to memory allocation issue (T352753) [tools]
04:51 <andrewbogott> rebooting tools-sgeweblight-10-27, tools-sgeweblight-10-17 and tools-sgeweblight-10-30; their filesystems seem locked up and I suspect NFS somehow [tools]
2023-12-04 §
09:15 <sstefanova@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api [tools]
09:15 <sstefanova@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api [tools]
2023-12-02 §
11:18 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster [tools]
11:15 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-22 [tools]
11:06 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeexec-10-13, tools-sgeweblight-10-20 [tools]
10:50 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster [tools]
10:39 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster [tools]
00:08 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster [tools]
00:08 <taavi@cloudcumin1001> END (ERROR) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=97) for a worker role in the tools cluster [tools]
00:07 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster [tools]
00:05 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster [tools]
00:04 <taavi@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster [tools]
00:04 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster [tools]
00:02 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster [tools]
2023-12-01 §
23:55 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster [tools]
23:52 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster [tools]
23:50 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster [tools]
22:51 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers [tools]
21:22 <andrewbogott> rebooting tools-sgeweblight-10-[18,21,32].tools.eqiad1.wikimedia.cloud to recover from nfs lockup [tools]