| 2024-02-16
      
      § | 
    
  | 09:49 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 09:49 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-80 | [tools] | 
            
  | 09:49 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-80 | [tools] | 
            
  | 09:45 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster | [tools] | 
            
  | 09:45 | <taavi@cloudcumin1001> | Added a new k8s worker-nfs tools-k8s-worker-nfs-35.tools.eqiad1.wikimedia.cloud to the cluster | [tools] | 
            
  | 09:35 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 09:35 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-79 | [tools] | 
            
  | 09:34 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-79 | [tools] | 
            
  | 09:24 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster | [tools] | 
            
  | 09:24 | <taavi@cloudcumin1001> | Added a new k8s worker-nfs tools-k8s-worker-nfs-34.tools.eqiad1.wikimedia.cloud to the cluster | [tools] | 
            
  | 09:13 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 09:06 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-78 | [tools] | 
            
  | 09:05 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-78 | [tools] | 
            
  | 09:05 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster | [tools] | 
            
  | 09:05 | <taavi@cloudcumin1001> | Added a new k8s worker-nfs tools-k8s-worker-nfs-33.tools.eqiad1.wikimedia.cloud to the cluster | [tools] | 
            
  | 08:55 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 08:55 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-77 | [tools] | 
            
  | 08:54 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-77 | [tools] | 
            
  
    | 2024-02-15
      
      § | 
    
  | 13:03 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-ingress-4 | [tools] | 
            
  | 13:03 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-ingress-4 | [tools] | 
            
  | 13:02 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster | [tools] | 
            
  | 13:02 | <taavi@cloudcumin1001> | Added a new k8s worker-nfs tools-k8s-worker-nfs-32.tools.eqiad1.wikimedia.cloud to the cluster | [tools] | 
            
  | 12:51 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 12:51 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-76 | [tools] | 
            
  | 12:50 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-76 | [tools] | 
            
  | 12:44 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster | [tools] | 
            
  | 12:44 | <taavi@cloudcumin1001> | Added a new k8s worker-nfs tools-k8s-worker-nfs-31.tools.eqiad1.wikimedia.cloud to the cluster | [tools] | 
            
  | 12:34 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 12:34 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-75 | [tools] | 
            
  | 12:33 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-75 | [tools] | 
            
  | 11:37 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a ingress role in the tools cluster | [tools] | 
            
  | 11:37 | <taavi@cloudcumin1001> | Added a new k8s ingress tools-k8s-ingress-7.tools.eqiad1.wikimedia.cloud to the cluster | [tools] | 
            
  | 11:30 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster | [tools] | 
            
  | 11:30 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-ingress-7 | [tools] | 
            
  | 11:29 | <taavi@cloudcumin1001> | START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-ingress-7 | [tools] | 
            
  | 11:29 | <taavi@cloudcumin1001> | END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a ingress role in the tools cluster | [tools] | 
            
  | 11:24 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.add_k8s_node for a ingress role in the tools cluster | [tools] | 
            
  
    | 2024-02-14
      
      § | 
    
  | 19:32 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeweblight-10-17, tools-sgeweblight-10-30 | [tools] | 
            
  | 16:35 | <taavi> | kill jobs user 'wikishizhao' is running directly on the grid per https://wikitech.wikimedia.org/wiki/Help:Toolforge/Rules #3 | [tools] | 
            
  | 16:30 | <taavi> | reboot tools-sgeexec-10-23 due to high load | [tools] | 
            
  | 09:14 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud | [tools] | 
            
  | 09:13 | <taavi@cloudcumin1001> | START - Cookbook wmcs.vps.refresh_puppet_certs on tools-k8s-worker-nfs-25.tools.eqiad1.wikimedia.cloud | [tools] | 
            
  | 09:13 | <wmbot~dcaro@urcuchillay> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0) | [tools] | 
            
  | 09:07 | <wmbot~dcaro@urcuchillay> | START - Cookbook wmcs.openstack.cloudvirt.vm_console | [tools] | 
            
  | 09:07 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster | [tools] | 
            
  | 09:07 | <taavi@cloudcumin1001> | Added a new k8s worker-nfs tools-k8s-worker-nfs-30.tools.eqiad1.wikimedia.cloud to the cluster | [tools] | 
            
  | 08:56 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 08:56 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-74 | [tools] | 
            
  | 08:55 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-74 | [tools] | 
            
  | 08:54 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster | [tools] |