| 2024-02-09
      
      § | 
    
  | 09:56 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster | [tools] | 
            
  | 09:56 | <taavi@cloudcumin1001> | Added a new k8s worker-nfs tools-k8s-worker-nfs-14.tools.eqiad1.wikimedia.cloud to the cluster | [tools] | 
            
  | 09:47 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 09:47 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-51 | [tools] | 
            
  | 09:46 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-51 | [tools] | 
            
  | 09:46 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-50 | [tools] | 
            
  | 09:46 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-50 | [tools] | 
            
  | 08:56 | <dcaro> | restart tools-k8s-worker-50 due to D some stuck processes | [tools] | 
            
  
    | 2024-02-08
      
      § | 
    
  | 13:03 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) | [tools] | 
            
  | 13:03 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors | [tools] | 
            
  | 09:46 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster | [tools] | 
            
  | 09:46 | <taavi@cloudcumin1001> | Added a new k8s worker-nfs tools-k8s-worker-nfs-13.tools.eqiad1.wikimedia.cloud to the cluster | [tools] | 
            
  | 09:35 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 09:34 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-49 | [tools] | 
            
  | 09:33 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-49 | [tools] | 
            
  | 09:33 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-48 | [tools] | 
            
  | 09:33 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-48 | [tools] | 
            
  | 09:32 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster | [tools] | 
            
  | 09:32 | <taavi@cloudcumin1001> | Added a new k8s worker-nfs tools-k8s-worker-nfs-12.tools.eqiad1.wikimedia.cloud to the cluster | [tools] | 
            
  | 09:23 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 09:22 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-47 | [tools] | 
            
  | 09:22 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-47 | [tools] | 
            
  | 09:22 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-46 | [tools] | 
            
  | 09:21 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-46 | [tools] | 
            
  | 09:21 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster | [tools] | 
            
  | 09:21 | <taavi@cloudcumin1001> | Added a new k8s worker-nfs tools-k8s-worker-nfs-11.tools.eqiad1.wikimedia.cloud to the cluster | [tools] | 
            
  | 09:13 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 09:11 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-45 | [tools] | 
            
  | 09:11 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-45 | [tools] | 
            
  | 09:10 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-44 | [tools] | 
            
  | 09:10 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-44 | [tools] | 
            
  | 09:10 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster | [tools] | 
            
  | 09:09 | <taavi@cloudcumin1001> | Added a new k8s worker-nfs tools-k8s-worker-nfs-10.tools.eqiad1.wikimedia.cloud to the cluster | [tools] | 
            
  | 09:00 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 08:59 | <taavi@cloudcumin1001> | END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster | [tools] | 
            
  | 08:58 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 08:58 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-43 | [tools] | 
            
  | 08:57 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-43 | [tools] | 
            
  | 08:57 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-42 | [tools] | 
            
  | 08:56 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-42 | [tools] | 
            
  
    | 2024-02-07
      
      § | 
    
  | 21:33 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for all workers | [tools] | 
            
  | 18:00 | <andrew@cloudcumin1001> | START - Cookbook wmcs.toolforge.k8s.reboot for all workers | [tools] | 
            
  | 17:58 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-9 | [tools] | 
            
  | 17:58 | <andrew@cloudcumin1001> | START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-9 | [tools] | 
            
  | 17:24 | <taavi@cloudcumin1001> | END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers | [tools] | 
            
  | 17:23 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.k8s.reboot for all workers | [tools] | 
            
  | 17:05 | <taavi@cloudcumin1001> | END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for all workers | [tools] | 
            
  | 17:05 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.k8s.reboot for all workers | [tools] | 
            
  | 17:03 | <taavi@cloudcumin1001> | END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for all workers | [tools] | 
            
  | 17:02 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.k8s.reboot for all workers | [tools] |