| 2024-02-12
      
      § | 
    
  | 12:56 | <taavi@cloudcumin1001> | Added a new k8s worker-nfs tools-k8s-worker-nfs-15.tools.eqiad1.wikimedia.cloud to the cluster | [tools] | 
            
  | 12:46 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 12:46 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-15 | [tools] | 
            
  | 12:45 | <taavi@cloudcumin1001> | START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-15 | [tools] | 
            
  | 12:44 | <taavi@cloudcumin1001> | END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster | [tools] | 
            
  | 12:37 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 12:37 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-53 | [tools] | 
            
  | 12:36 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-53 | [tools] | 
            
  | 12:36 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-52 | [tools] | 
            
  | 12:35 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-52 | [tools] | 
            
  | 10:51 | <sstefanova@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api | [tools] | 
            
  | 10:50 | <sstefanova@cloudcumin1001> | START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api | [tools] | 
            
  | 10:33 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers | [tools] | 
            
  | 10:33 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers | [tools] | 
            
  
    | 2024-02-09
      
      § | 
    
  | 18:03 | <andrewbogott> | updated the default security group, removing the 0.0.0.0/0 rule allowing port 22 access everywhere, replaced it with a 172.16.0.0/21 rule | [tools] | 
            
  | 13:06 | <taavi> | reboot tools-sgecron-2 due to high load | [tools] | 
            
  | 10:34 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component image-config | [tools] | 
            
  | 10:34 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.k8s.component.deploy for component image-config | [tools] | 
            
  | 09:56 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster | [tools] | 
            
  | 09:56 | <taavi@cloudcumin1001> | Added a new k8s worker-nfs tools-k8s-worker-nfs-14.tools.eqiad1.wikimedia.cloud to the cluster | [tools] | 
            
  | 09:47 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 09:47 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-51 | [tools] | 
            
  | 09:46 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-51 | [tools] | 
            
  | 09:46 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-50 | [tools] | 
            
  | 09:46 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-50 | [tools] | 
            
  | 08:56 | <dcaro> | restart tools-k8s-worker-50 due to D some stuck processes | [tools] | 
            
  
    | 2024-02-08
      
      § | 
    
  | 13:03 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.grid.cleanup_queue_errors (exit_code=0) | [tools] | 
            
  | 13:03 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.grid.cleanup_queue_errors | [tools] | 
            
  | 09:46 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster | [tools] | 
            
  | 09:46 | <taavi@cloudcumin1001> | Added a new k8s worker-nfs tools-k8s-worker-nfs-13.tools.eqiad1.wikimedia.cloud to the cluster | [tools] | 
            
  | 09:35 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 09:34 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-49 | [tools] | 
            
  | 09:33 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-49 | [tools] | 
            
  | 09:33 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-48 | [tools] | 
            
  | 09:33 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-48 | [tools] | 
            
  | 09:32 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster | [tools] | 
            
  | 09:32 | <taavi@cloudcumin1001> | Added a new k8s worker-nfs tools-k8s-worker-nfs-12.tools.eqiad1.wikimedia.cloud to the cluster | [tools] | 
            
  | 09:23 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 09:22 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-47 | [tools] | 
            
  | 09:22 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-47 | [tools] | 
            
  | 09:22 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-46 | [tools] | 
            
  | 09:21 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-46 | [tools] | 
            
  | 09:21 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster | [tools] | 
            
  | 09:21 | <taavi@cloudcumin1001> | Added a new k8s worker-nfs tools-k8s-worker-nfs-11.tools.eqiad1.wikimedia.cloud to the cluster | [tools] | 
            
  | 09:13 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 09:11 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-45 | [tools] | 
            
  | 09:11 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-45 | [tools] | 
            
  | 09:10 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-44 | [tools] |