| 
      
        2025-01-13
      
      §
     | 
  
    
  | 21:35 | 
  <andrew@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-47 (T383625) | 
  [tools] | 
            
  | 21:31 | 
  <andrew@cloudcumin1001> | 
  START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-47 (T383625) | 
  [tools] | 
            
  | 21:30 | 
  <andrew@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38 (T383625) | 
  [tools] | 
            
  | 21:29 | 
  <andrew@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19 (T383238) | 
  [tools] | 
            
  | 21:25 | 
  <andrew@cloudcumin1001> | 
  START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38 (T383625) | 
  [tools] | 
            
  | 21:24 | 
  <andrew@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-74 (T383625) | 
  [tools] | 
            
  | 21:24 | 
  <andrew@cloudcumin1001> | 
  START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19 (T383238) | 
  [tools] | 
            
  | 21:20 | 
  <andrew@cloudcumin1001> | 
  START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-74 (T383625) | 
  [tools] | 
            
  | 21:19 | 
  <andrew@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 (T383625) | 
  [tools] | 
            
  | 21:18 | 
  <andrew@cloudcumin1001> | 
  START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 (T383625) | 
  [tools] | 
            
  | 21:18 | 
  <andrew@cloudcumin1001> | 
  END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-21 (T383238) | 
  [tools] | 
            
  | 21:15 | 
  <andrew@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-75 (T383625) | 
  [tools] | 
            
  | 21:14 | 
  <andrew@cloudcumin1001> | 
  START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 (T383625) | 
  [tools] | 
            
  | 21:14 | 
  <andrew@cloudcumin1001> | 
  START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 (T383238) | 
  [tools] | 
            
  | 21:14 | 
  <andrew@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-2 (T383238) | 
  [tools] | 
            
  | 21:14 | 
  <andrew@cloudcumin1001> | 
  END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-75 (T383238) | 
  [tools] | 
            
  | 21:13 | 
  <andrew@cloudcumin1001> | 
  START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-75 (T383238) | 
  [tools] | 
            
  | 21:10 | 
  <andrew@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-45 (T383625) | 
  [tools] | 
            
  | 21:08 | 
  <andrew@cloudcumin1001> | 
  START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-2 (T383238) | 
  [tools] | 
            
  | 21:08 | 
  <andrew@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-35 (T383238) | 
  [tools] | 
            
  | 21:05 | 
  <andrew@cloudcumin1001> | 
  START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-45 (T383625) | 
  [tools] | 
            
  | 21:03 | 
  <andrew@cloudcumin1001> | 
  START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-35 (T383238) | 
  [tools] | 
            
  | 21:03 | 
  <andrew@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-13 (T383238) | 
  [tools] | 
            
  | 20:58 | 
  <andrew@cloudcumin1001> | 
  START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-13 (T383238) | 
  [tools] | 
            
  | 20:58 | 
  <andrew@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16 (T383238) | 
  [tools] | 
            
  | 20:57 | 
  <andrew@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 (T383625) | 
  [tools] | 
            
  | 20:53 | 
  <andrew@cloudcumin1001> | 
  START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16 (T383238) | 
  [tools] | 
            
  | 20:53 | 
  <andrew@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-58 (T383238) | 
  [tools] | 
            
  | 20:52 | 
  <andrew@cloudcumin1001> | 
  START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 (T383625) | 
  [tools] | 
            
  | 20:49 | 
  <dcaro> | 
  restart prometheus to pick up the new ips for vms and such | 
  [tools] | 
            
  | 20:48 | 
  <andrew@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-21 (T383625) | 
  [tools] | 
            
  | 20:47 | 
  <andrew@cloudcumin1001> | 
  START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-58 (T383238) | 
  [tools] | 
            
  | 20:47 | 
  <andrew@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-8 | 
  [tools] | 
            
  | 20:43 | 
  <andrew@cloudcumin1001> | 
  START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-21 (T383625) | 
  [tools] | 
            
  | 20:43 | 
  <andrew@cloudcumin1001> | 
  END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-20 (T383625) | 
  [tools] | 
            
  | 20:42 | 
  <andrew@cloudcumin1001> | 
  START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 (T383625) | 
  [tools] | 
            
  | 20:42 | 
  <andrew@cloudcumin1001> | 
  END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-20 (T383238) | 
  [tools] | 
            
  | 20:42 | 
  <andrew@cloudcumin1001> | 
  START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-20 (T383238) | 
  [tools] | 
            
  | 20:42 | 
  <andrew@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-1 (T383238) | 
  [tools] | 
            
  | 20:41 | 
  <andrew@cloudcumin1001> | 
  START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-8 | 
  [tools] | 
            
  | 20:41 | 
  <andrew@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-72 | 
  [tools] | 
            
  | 20:38 | 
  <andrew@cloudcumin1001> | 
  START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-1 (T383238) | 
  [tools] | 
            
  | 20:37 | 
  <andrew@cloudcumin1001> | 
  START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-72 | 
  [tools] | 
            
  | 20:36 | 
  <lucaswerkmeister> | 
  restore root-owned /tmp/framer.txt on tools-sgebastion-10, tools-bastion-12, tools-bastion-13 (cf. 2025-01-05 log entry) following bastion reboots | 
  [tools] |