| 2024-02-09
      
      § | 
    
  | 18:03 | <andrewbogott> | updated the default security group, removing the 0.0.0.0/0 rule allowing port 22 access everywhere, replaced it with a 172.16.0.0/21 rule | [tools] | 
            
  | 13:06 | <taavi> | reboot tools-sgecron-2 due to high load | [tools] | 
            
  | 10:34 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config | [tools] | 
            
  | 10:34 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config | [tools] | 
            
  | 09:56 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster | [tools] | 
            
  | 09:56 | <taavi@cloudcumin1001> | Added a new k8s worker-nfs tools-k8s-worker-nfs-14.tools.eqiad1.wikimedia.cloud to the cluster | [tools] | 
            
  | 09:47 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 09:47 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-51 | [tools] | 
            
  | 09:46 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-51 | [tools] | 
            
  | 09:46 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-50 | [tools] | 
            
  | 09:46 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-50 | [tools] | 
            
  | 08:56 | <dcaro> | restart tools-k8s-worker-50 due to D some stuck processes | [tools] | 
            
  
    | 2024-02-08
      
      § | 
    
  | 13:03 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) | [tools] | 
            
  | 13:03 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors | [tools] | 
            
  | 09:46 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster | [tools] | 
            
  | 09:46 | <taavi@cloudcumin1001> | Added a new k8s worker-nfs tools-k8s-worker-nfs-13.tools.eqiad1.wikimedia.cloud to the cluster | [tools] | 
            
  | 09:35 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 09:34 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-49 | [tools] | 
            
  | 09:33 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-49 | [tools] | 
            
  | 09:33 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-48 | [tools] | 
            
  | 09:33 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-48 | [tools] | 
            
  | 09:32 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster | [tools] | 
            
  | 09:32 | <taavi@cloudcumin1001> | Added a new k8s worker-nfs tools-k8s-worker-nfs-12.tools.eqiad1.wikimedia.cloud to the cluster | [tools] | 
            
  | 09:23 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 09:22 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-47 | [tools] | 
            
  | 09:22 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-47 | [tools] | 
            
  | 09:22 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-46 | [tools] | 
            
  | 09:21 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-46 | [tools] | 
            
  | 09:21 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster | [tools] | 
            
  | 09:21 | <taavi@cloudcumin1001> | Added a new k8s worker-nfs tools-k8s-worker-nfs-11.tools.eqiad1.wikimedia.cloud to the cluster | [tools] | 
            
  | 09:13 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 09:11 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-45 | [tools] | 
            
  | 09:11 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-45 | [tools] | 
            
  | 09:10 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-44 | [tools] | 
            
  | 09:10 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-44 | [tools] | 
            
  | 09:10 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster | [tools] | 
            
  | 09:09 | <taavi@cloudcumin1001> | Added a new k8s worker-nfs tools-k8s-worker-nfs-10.tools.eqiad1.wikimedia.cloud to the cluster | [tools] | 
            
  | 09:00 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 08:59 | <taavi@cloudcumin1001> | END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster | [tools] | 
            
  | 08:58 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 08:58 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-43 | [tools] | 
            
  | 08:57 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-43 | [tools] | 
            
  | 08:57 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-42 | [tools] |