| 2025-05-13
      
      § | 
    
  | 15:47 | <aborrero@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers | [tools] | 
            
  | 15:33 | <aborrero@cloudcumin1001> | START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers | [tools] | 
            
  | 07:43 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-36 | [tools] | 
            
  | 07:37 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 | [tools] | 
            
  | 07:37 | <taavi@cloudcumin1001> | END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-36 | [tools] | 
            
  | 07:34 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-36 | [tools] | 
            
  
    | 2025-05-12
      
      § | 
    
  | 19:48 | <raymond-ndibe@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api | [tools] | 
            
  | 19:35 | <raymond-ndibe@cloudcumin1001> | START - Cookbook wmcs.toolforge.component.deploy for component jobs-api | [tools] | 
            
  | 16:18 | <dcaro@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli | [tools] | 
            
  | 16:06 | <dcaro@cloudcumin1001> | START - Cookbook wmcs.toolforge.component.deploy for component builds-cli | [tools] | 
            
  | 13:46 | <dcaro@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api | [tools] | 
            
  | 13:34 | <dcaro@cloudcumin1001> | START - Cookbook wmcs.toolforge.component.deploy for component builds-api | [tools] | 
            
  | 13:23 | <dcaro@cloudcumin1001> | END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api | [tools] | 
            
  | 13:22 | <dcaro@cloudcumin1001> | START - Cookbook wmcs.toolforge.component.deploy for component builds-api | [tools] | 
            
  | 13:04 | <arturo> | add container image to docker registry docker-registry.tools.wmflabs.org/tofu-provisioning:20250512  (T393686) | [tools] | 
            
  | 11:51 | <dcaro@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component registry-admission | [tools] | 
            
  | 11:39 | <dcaro@cloudcumin1001> | START - Cookbook wmcs.toolforge.component.deploy for component registry-admission | [tools] | 
            
  | 11:26 | <dcaro@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component ingress-admission | [tools] | 
            
  | 11:14 | <dcaro@cloudcumin1001> | START - Cookbook wmcs.toolforge.component.deploy for component ingress-admission | [tools] | 
            
  | 10:44 | <dcaro@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api | [tools] | 
            
  | 10:32 | <dcaro@cloudcumin1001> | START - Cookbook wmcs.toolforge.component.deploy for component builds-api | [tools] | 
            
  | 10:00 | <dcaro@cloudcumin1001> | END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api | [tools] | 
            
  | 09:57 | <dcaro@cloudcumin1001> | START - Cookbook wmcs.toolforge.component.deploy for component builds-api | [tools] | 
            
  | 09:29 | <dcaro@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api | [tools] | 
            
  | 09:17 | <dcaro@cloudcumin1001> | START - Cookbook wmcs.toolforge.component.deploy for component envvars-api | [tools] | 
            
  | 08:59 | <dcaro@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component volume-admission | [tools] | 
            
  | 08:47 | <dcaro@cloudcumin1001> | START - Cookbook wmcs.toolforge.component.deploy for component volume-admission | [tools] | 
            
  | 02:36 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19 | [tools] | 
            
  | 02:32 | <andrew@cloudcumin1001> | START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19 | [tools] | 
            
  
    | 2025-05-10
      
      § | 
    
  | 17:35 | <lucaswerkmeister> | root@tools-bastion-13:~# systemctl restart sssd-sudo{,.socket} # looks like the reset-failed didn’t work properly, systemd didn’t even try to start the service again afaict (T393732) | [tools] | 
            
  | 17:33 | <lucaswerkmeister> | root@tools-bastion-13:~# systemctl reset-failed sssd-{pam,sudo}.service && systemctl restart sssd-pam{,-priv}.socket # try to reset the rate limits this way (T393732) | [tools] | 
            
  | 16:22 | <lucaswerkmeister> | systemctl restart sssd-{pam{,-priv},sudo}.socket # service-start-limit-hit, T393732? | [tools] | 
            
  | 14:10 | <lucaswerkmeister> | root@tools-bastion-13:~# systemctl restart sssd-sudo.socket # service-start-limit-hit, T393732? | [tools] | 
            
  | 11:53 | <lucaswerkmeister> | T393732 note: restart of sssd-pam.service actually failed, “may be requested by dependency only”; overall it still seems to have worked though (so next time restarting the sockets is probably sufficient) | [tools] | 
            
  | 11:52 | <lucaswerkmeister> | root@tools-bastion-13:~# systemctl restart sssd-pam{,{,-priv}.socket} # all three failed with start-limit-hit / Start request repeated too quickly; T393732? | [tools] | 
            
  
    | 2025-05-08
      
      § | 
    
  | 17:41 | <dcaro@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-emailer | [tools] | 
            
  | 17:40 | <raymond-ndibe@cloudcumin1001> | END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission | [tools] | 
            
  | 17:40 | <raymond-ndibe@cloudcumin1001> | START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission | [tools] | 
            
  | 17:39 | <raymond-ndibe@cloudcumin1001> | END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-admission | [tools] | 
            
  | 17:39 | <raymond-ndibe@cloudcumin1001> | START - Cookbook wmcs.toolforge.component.deploy for component envvars-admission | [tools] | 
            
  | 17:29 | <dcaro@cloudcumin1001> | START - Cookbook wmcs.toolforge.component.deploy for component jobs-emailer | [tools] |