551-600 of 5111 results (31ms)
2023-12-20 §
10:01 <taavi> rebooting tools-sgeweblight-10-18, -24, -25, to get rid of a large number of jobs in deleting status [tools]
2023-12-19 §
15:39 <dhinus> restarting toolsdb to apply a config change T353093 [tools]
13:18 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api [tools]
13:17 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api [tools]
2023-12-18 §
16:15 <taavi> reboot tools-sgeexec-10-15, -23 due to stuck NFS processes [tools]
14:43 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [tools]
14:42 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [tools]
14:40 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [tools]
14:39 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [tools]
2023-12-16 §
22:01 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) [tools]
22:01 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors [tools]
20:54 <bd808> Rebuilding all containers to pick up lighttpd config fix and normal package updates (T293552) [tools]
08:14 <dhinus> restarting toolsdb with jemalloc [tools]
05:32 <andrewbogott> restarting mariadb on toolsdb-1 because it's just about to go oom (or possibly just did) [tools]
00:21 <dhinus> restarting toolsdb again as it's again low in free mem T353093 [tools]
2023-12-15 §
20:26 <andrewbogott> restarting toolsdb to avoid upcoming oom crash [tools]
16:49 <dhinus> restarting toolsdb before it's about to go OOM, enabling performance_schema for debugging [tools]
14:40 <dcaro> deploy toolforge-builds-cli 0.0.10 (T341067) [tools]
13:33 <wm-bot2> dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api (T341067) [tools]
13:32 <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api (T341067) [tools]
12:19 <dhinus> restarting toolsdb again to apply a config fix T353093 [tools]
10:48 <dhinus> restarting toolsdb to apply new config T353093 [tools]
2023-12-14 §
23:02 <andrewbogott> rebooting tools-db-1 yet again [tools]
17:42 <taavi> reboot tools-sgewebgen-10-3 [tools]
02:20 <andrewbogott> restarting tools-db-1, oomkiller killed mariadb [tools]
2023-12-13 §
19:53 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers [tools]
18:38 <andrewbogott> rebooting tools-nfs-2 server to resolve weird file locking issues [tools]
16:23 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.scale_grid_exec [tools]
14:23 <wm-bot2> dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-builder (T352774) [tools]
14:22 <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-builder (T352774) [tools]
14:22 <wm-bot2> dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api (T352774) [tools]
14:22 <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api (T352774) [tools]
13:54 <dcaro> deploy toolforge-builds-cli version 0.0.9 (with envvars support) [tools]
13:32 <wm-bot2> dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api (T338142) [tools]
13:31 <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api (T338142) [tools]
11:44 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) [tools]
11:44 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors [tools]
11:17 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-16 [tools]
10:48 <wm-bot2> dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission (T338142) [tools]
10:48 <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission (T338142) [tools]
09:49 <wm-bot2> dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api [tools]
09:49 <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api [tools]
2023-12-12 §
17:32 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster [tools]
17:29 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeexec-10-18 [tools]
17:27 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeexec-10-19 [tools]
17:24 <taavi> reboot tools-sgeexec-10-14 [tools]
15:58 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster [tools]
15:54 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-14, tools-sgeexec-10-8 [tools]
15:51 <taavi@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker role in the tools cluster [tools]
15:51 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster [tools]