| 2023-12-01
      
      § | 
    
  | 16:19 | <wm-bot2> | fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T351171) | [admin] | 
            
  | 15:48 | <andrewbogott> | reimaging cloudcontrol1005 due to widespread misbehavior | [admin] | 
            
  | 14:24 | <wm-bot2> | fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T351171) | [admin] | 
            
  | 14:20 | <wm-bot2> | fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T351171) | [admin] | 
            
  | 14:19 | <fnegri@cloudcumin1001> | END (ERROR) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=97) (T351171) | [admin] | 
            
  | 14:18 | <fnegri@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T351171) | [admin] | 
            
  | 13:55 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) | [admin] | 
            
  | 13:54 | <taavi@cloudcumin1001> | START - Cookbook wmcs.openstack.restart_openstack | [admin] | 
            
  | 11:57 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) | [admin] | 
            
  | 11:56 | <taavi@cloudcumin1001> | START - Cookbook wmcs.openstack.restart_openstack | [admin] | 
            
  | 11:55 | <taavi> | restart neutron-rpc-server.service on eqiad1 cloudcontrols | [admin] | 
            
  
    | 2023-11-30
      
      § | 
    
  | 20:44 | <andrewbogott> | generating to application credentials for the tests that run on tf-infra-test | [admin] | 
            
  | 19:54 | <andrewbogott> | reimaged cloudrabbit100[23] after https://gerrit.wikimedia.org/r/c/operations/puppet/+/979127.  I didn't reimage 1001 because that will require rebuilding the whole cluster but I did remove the related packages. | [admin] | 
            
  | 18:10 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet' | [admin] | 
            
  | 18:10 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' | [admin] | 
            
  | 18:05 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1039.eqiad.wmnet' | [admin] | 
            
  | 18:05 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1038.eqiad.wmnet' | [admin] | 
            
  | 18:04 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet' | [admin] | 
            
  | 18:04 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet' | [admin] | 
            
  | 18:04 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' | [admin] | 
            
  | 17:59 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1037.eqiad.wmnet' | [admin] | 
            
  | 17:59 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1036.eqiad.wmnet' | [admin] | 
            
  | 17:59 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1035.eqiad.wmnet' | [admin] | 
            
  | 17:53 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' | [admin] | 
            
  | 17:53 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' | [admin] | 
            
  | 17:53 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1032.eqiad.wmnet' | [admin] | 
            
  | 17:52 | <fnegri@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1003.eqiad.wmnet' (T348843) | [admin] | 
            
  | 17:48 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1034.eqiad.wmnet' | [admin] | 
            
  | 17:48 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1033.eqiad.wmnet' | [admin] | 
            
  | 17:48 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1032.eqiad.wmnet' | [admin] | 
            
  | 17:47 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' | [admin] | 
            
  | 17:47 | <fnegri@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1003.eqiad.wmnet' (T348843) | [admin] | 
            
  | 17:47 | <fnegri@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1002.eqiad.wmnet' (T348843) | [admin] | 
            
  | 17:42 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1031.eqiad.wmnet' | [admin] | 
            
  | 17:42 | <fnegri@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1002.eqiad.wmnet' (T348843) | [admin] | 
            
  | 17:42 | <fnegri@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1001.eqiad.wmnet' (T348843) | [admin] | 
            
  | 17:37 | <fnegri@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1001.eqiad.wmnet' (T348843) | [admin] | 
            
  | 17:36 | <fnegri@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1003.eqiad.wmnet' (T348843) | [admin] | 
            
  | 17:30 | <fnegri@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1003.eqiad.wmnet' (T348843) | [admin] | 
            
  | 17:30 | <fnegri@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1002.eqiad.wmnet' (T348843) | [admin] | 
            
  | 17:25 | <fnegri@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1002.eqiad.wmnet' (T348843) | [admin] | 
            
  | 17:25 | <fnegri@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1001.eqiad.wmnet' (T348843) | [admin] | 
            
  | 17:19 | <fnegri@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1001.eqiad.wmnet' (T348843) | [admin] | 
            
  | 17:19 | <fnegri@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet' (T348843) | [admin] | 
            
  | 17:14 | <fnegri@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1045.eqiad.wmnet' (T348843) | [admin] | 
            
  | 17:14 | <fnegri@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1044.eqiad.wmnet' (T348843) | [admin] | 
            
  | 17:09 | <fnegri@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1044.eqiad.wmnet' (T348843) | [admin] | 
            
  | 17:09 | <fnegri@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet' (T348843) | [admin] | 
            
  | 17:04 | <fnegri@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1043.eqiad.wmnet' (T348843) | [admin] | 
            
  | 17:04 | <fnegri@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet' (T348843) | [admin] |