| 2024-04-25
      
      ยง | 
    
  | 19:10 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1052.eqiad.wmnet' (T356287) | [admin] | 
            
  | 19:05 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1052.eqiad.wmnet' (T356287) | [admin] | 
            
  | 19:05 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1048.eqiad.wmnet' (T356287) | [admin] | 
            
  | 19:03 | <dancy@deploy1002> | Started scap: Testing | [production] | 
            
  | 19:00 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1048.eqiad.wmnet' (T356287) | [admin] | 
            
  | 19:00 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1003.eqiad.wmnet' (T356287) | [admin] | 
            
  | 18:55 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1003.eqiad.wmnet' (T356287) | [admin] | 
            
  | 18:55 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1002.eqiad.wmnet' (T356287) | [admin] | 
            
  | 18:49 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1002.eqiad.wmnet' (T356287) | [admin] | 
            
  | 18:49 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt-wdqs1001.eqiad.wmnet' (T356287) | [admin] | 
            
  | 18:44 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt-wdqs1001.eqiad.wmnet' (T356287) | [admin] | 
            
  | 18:44 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet' (T356287) | [admin] | 
            
  | 18:39 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1047.eqiad.wmnet' (T356287) | [admin] | 
            
  | 18:39 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1038.eqiad.wmnet' (T356287) | [admin] | 
            
  | 18:35 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1038.eqiad.wmnet' (T356287) | [admin] | 
            
  | 18:35 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1042.eqiad.wmnet' (T356287) | [admin] | 
            
  | 18:30 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1042.eqiad.wmnet' (T356287) | [admin] | 
            
  | 18:30 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1044.eqiad.wmnet' (T356287) | [admin] | 
            
  | 18:25 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1044.eqiad.wmnet' (T356287) | [admin] | 
            
  | 18:25 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1041.eqiad.wmnet' (T356287) | [admin] | 
            
  | 18:23 | <brennen@deploy1002> | rebuilt and synchronized wikiversions files: group2 wikis to 1.43.0-wmf.2  refs T361396 | [production] | 
            
  | 18:22 | <logmsgbot> | nshahquinn-wmf@deploy1002 Finished deploy [airflow-dags/analytics_product@0e9fd9a]: (no justification provided) (duration: 00m 07s) | [production] | 
            
  | 18:22 | <logmsgbot> | nshahquinn-wmf@deploy1002 Started deploy [airflow-dags/analytics_product@0e9fd9a]: (no justification provided) | [production] | 
            
  | 18:20 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1041.eqiad.wmnet' (T356287) | [admin] | 
            
  | 18:20 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1046.eqiad.wmnet' (T356287) | [admin] | 
            
  | 18:15 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1046.eqiad.wmnet' (T356287) | [admin] | 
            
  | 18:15 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1043.eqiad.wmnet' (T356287) | [admin] | 
            
  | 18:10 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1043.eqiad.wmnet' (T356287) | [admin] | 
            
  | 18:10 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1045.eqiad.wmnet' (T356287) | [admin] | 
            
  | 18:08 | <brennen> | train 1.43.0-wmf.2 (T361396) status: no current blockers, rolling to group2 | [production] | 
            
  | 18:05 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1045.eqiad.wmnet' (T356287) | [admin] | 
            
  | 18:05 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1040.eqiad.wmnet' (T356287) | [admin] | 
            
  | 18:01 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1040.eqiad.wmnet' (T356287) | [admin] | 
            
  | 18:00 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1036.eqiad.wmnet' (T356287) | [admin] | 
            
  | 17:58 | <ladsgroup@cumin1002> | dbctl commit (dc=all): 'Depooling db2162 (T352010)', diff saved to https://phabricator.wikimedia.org/P61223 and previous config saved to /var/cache/conftool/dbconfig/20240425-175802-ladsgroup.json | [production] | 
            
  | 17:58 | <ladsgroup@cumin1002> | END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2162.codfw.wmnet with reason: Maintenance | [production] | 
            
  | 17:57 | <ladsgroup@cumin1002> | START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2162.codfw.wmnet with reason: Maintenance | [production] | 
            
  | 17:57 | <ladsgroup@cumin1002> | dbctl commit (dc=all): 'Repooling after maintenance db2161 (T352010)', diff saved to https://phabricator.wikimedia.org/P61222 and previous config saved to /var/cache/conftool/dbconfig/20240425-175739-ladsgroup.json | [production] | 
            
  | 17:56 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1036.eqiad.wmnet' (T356287) | [admin] | 
            
  | 17:56 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1034.eqiad.wmnet' (T356287) | [admin] | 
            
  | 17:51 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1034.eqiad.wmnet' (T356287) | [admin] | 
            
  | 17:51 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1039.eqiad.wmnet' (T356287) | [admin] | 
            
  | 17:46 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1039.eqiad.wmnet' (T356287) | [admin] | 
            
  | 17:46 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1037.eqiad.wmnet' (T356287) | [admin] | 
            
  | 17:42 | <ladsgroup@cumin1002> | dbctl commit (dc=all): 'Repooling after maintenance db2161', diff saved to https://phabricator.wikimedia.org/P61221 and previous config saved to /var/cache/conftool/dbconfig/20240425-174233-ladsgroup.json | [production] | 
            
  | 17:41 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1037.eqiad.wmnet' (T356287) | [admin] | 
            
  | 17:41 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1035.eqiad.wmnet' (T356287) | [admin] | 
            
  | 17:38 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) | [admin] | 
            
  | 17:36 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1035.eqiad.wmnet' (T356287) | [admin] | 
            
  | 17:35 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' (T356287) | [admin] |