| 
      
        2023-12-02
      
      §
     | 
  
    
  | 00:18 | 
  <taavi@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster | 
  [admin] | 
            
  | 00:18 | 
  <taavi@cloudcumin1001> | 
  Added a new k8s worker tools-k8s-worker-93.tools.eqiad1.wikimedia.cloud to the cluster | 
  [admin] | 
            
  | 00:15 | 
  <taavi@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster | 
  [admin] | 
            
  | 00:15 | 
  <taavi@cloudcumin1001> | 
  Added a new k8s worker tools-k8s-worker-92.tools.eqiad1.wikimedia.cloud to the cluster | 
  [admin] | 
            
  | 00:06 | 
  <taavi@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster | 
  [admin] | 
            
  | 00:06 | 
  <taavi@cloudcumin1001> | 
  Added a new k8s worker tools-k8s-worker-91.tools.eqiad1.wikimedia.cloud to the cluster | 
  [admin] | 
            
  | 00:05 | 
  <taavi@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster | 
  [admin] | 
            
  | 00:05 | 
  <taavi@cloudcumin1001> | 
  Added a new k8s worker tools-k8s-worker-90.tools.eqiad1.wikimedia.cloud to the cluster | 
  [admin] | 
            
  | 00:01 | 
  <taavi@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster | 
  [admin] | 
            
  | 00:01 | 
  <taavi@cloudcumin1001> | 
  Added a new k8s worker tools-k8s-worker-89.tools.eqiad1.wikimedia.cloud to the cluster | 
  [admin] | 
            
  
    | 
      
        2023-12-01
      
      §
     | 
  
    
  | 17:43 | 
  <andrew@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) | 
  [admin] | 
            
  | 17:37 | 
  <andrew@cloudcumin1001> | 
  START - Cookbook wmcs.openstack.restart_openstack | 
  [admin] | 
            
  | 17:01 | 
  <andrew@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) | 
  [admin] | 
            
  | 16:57 | 
  <andrew@cloudcumin1001> | 
  START - Cookbook wmcs.openstack.restart_openstack | 
  [admin] | 
            
  | 16:48 | 
  <andrew@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) | 
  [admin] | 
            
  | 16:46 | 
  <andrew@cloudcumin1001> | 
  START - Cookbook wmcs.openstack.restart_openstack | 
  [admin] | 
            
  | 16:19 | 
  <wm-bot2> | 
  fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T351171) | 
  [admin] | 
            
  | 16:19 | 
  <wm-bot2> | 
  fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T351171) | 
  [admin] | 
            
  | 15:48 | 
  <andrewbogott> | 
  reimaging cloudcontrol1005 due to widespread misbehavior | 
  [admin] | 
            
  | 14:24 | 
  <wm-bot2> | 
  fran@wmf3169 END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T351171) | 
  [admin] | 
            
  | 14:20 | 
  <wm-bot2> | 
  fran@wmf3169 START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T351171) | 
  [admin] | 
            
  | 14:19 | 
  <fnegri@cloudcumin1001> | 
  END (ERROR) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=97) (T351171) | 
  [admin] | 
            
  | 14:18 | 
  <fnegri@cloudcumin1001> | 
  START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T351171) | 
  [admin] | 
            
  | 13:55 | 
  <taavi@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) | 
  [admin] | 
            
  | 13:54 | 
  <taavi@cloudcumin1001> | 
  START - Cookbook wmcs.openstack.restart_openstack | 
  [admin] | 
            
  | 11:57 | 
  <taavi@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) | 
  [admin] | 
            
  | 11:56 | 
  <taavi@cloudcumin1001> | 
  START - Cookbook wmcs.openstack.restart_openstack | 
  [admin] | 
            
  | 11:55 | 
  <taavi> | 
  restart neutron-rpc-server.service on eqiad1 cloudcontrols | 
  [admin] | 
            
  
    | 
      
        2023-11-30
      
      §
     | 
  
    
  | 20:44 | 
  <andrewbogott> | 
  generating to application credentials for the tests that run on tf-infra-test | 
  [admin] | 
            
  | 19:54 | 
  <andrewbogott> | 
  reimaged cloudrabbit100[23] after https://gerrit.wikimedia.org/r/c/operations/puppet/+/979127.  I didn't reimage 1001 because that will require rebuilding the whole cluster but I did remove the related packages. | 
  [admin] | 
            
  | 18:10 | 
  <andrew@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet' | 
  [admin] | 
            
  | 18:10 | 
  <andrew@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' | 
  [admin] | 
            
  | 18:05 | 
  <andrew@cloudcumin1001> | 
  START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1039.eqiad.wmnet' | 
  [admin] | 
            
  | 18:05 | 
  <andrew@cloudcumin1001> | 
  START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1038.eqiad.wmnet' | 
  [admin] | 
            
  | 18:04 | 
  <andrew@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet' | 
  [admin] | 
            
  | 18:04 | 
  <andrew@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet' | 
  [admin] | 
            
  | 18:04 | 
  <andrew@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' | 
  [admin] | 
            
  | 17:59 | 
  <andrew@cloudcumin1001> | 
  START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1037.eqiad.wmnet' | 
  [admin] | 
            
  | 17:59 | 
  <andrew@cloudcumin1001> | 
  START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1036.eqiad.wmnet' | 
  [admin] | 
            
  | 17:59 | 
  <andrew@cloudcumin1001> | 
  START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1035.eqiad.wmnet' | 
  [admin] | 
            
  | 17:53 | 
  <andrew@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' | 
  [admin] | 
            
  | 17:53 | 
  <andrew@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' | 
  [admin] | 
            
  | 17:53 | 
  <andrew@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1032.eqiad.wmnet' | 
  [admin] | 
            
  | 17:52 | 
  <fnegri@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1003.eqiad.wmnet' (T348843) | 
  [admin] | 
            
  | 17:48 | 
  <andrew@cloudcumin1001> | 
  START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1034.eqiad.wmnet' | 
  [admin] | 
            
  | 17:48 | 
  <andrew@cloudcumin1001> | 
  START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1033.eqiad.wmnet' | 
  [admin] | 
            
  | 17:48 | 
  <andrew@cloudcumin1001> | 
  START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1032.eqiad.wmnet' | 
  [admin] | 
            
  | 17:47 | 
  <andrew@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' | 
  [admin] | 
            
  | 17:47 | 
  <fnegri@cloudcumin1001> | 
  START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1003.eqiad.wmnet' (T348843) | 
  [admin] | 
            
  | 17:47 | 
  <fnegri@cloudcumin1001> | 
  END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1002.eqiad.wmnet' (T348843) | 
  [admin] |