| 2024-04-25
      
      ยง | 
    
  | 23:12 | <ladsgroup@cumin1002> | dbctl commit (dc=all): 'Repooling after maintenance db2147 (T352010)', diff saved to https://phabricator.wikimedia.org/P61230 and previous config saved to /var/cache/conftool/dbconfig/20240425-231201-ladsgroup.json | [production] | 
            
  | 22:56 | <ladsgroup@cumin1002> | dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P61229 and previous config saved to /var/cache/conftool/dbconfig/20240425-225654-ladsgroup.json | [production] | 
            
  | 22:41 | <ladsgroup@cumin1002> | dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P61228 and previous config saved to /var/cache/conftool/dbconfig/20240425-224146-ladsgroup.json | [production] | 
            
  | 22:26 | <ladsgroup@cumin1002> | dbctl commit (dc=all): 'Repooling after maintenance db2147 (T352010)', diff saved to https://phabricator.wikimedia.org/P61227 and previous config saved to /var/cache/conftool/dbconfig/20240425-222638-ladsgroup.json | [production] | 
            
  | 22:23 | <brett> | Extend prometheus1005 and prometheus1006 logical volume by an extra 60G due to disk filling up | [production] | 
            
  | 21:32 | <wmbot~bd808@tools-bastion-12> | Started `bridgebot` job (T363028) | [tools.bridgebot] | 
            
  | 21:31 | <wmbot~bd808@tools-bastion-12> | built new image from f4022bd9 (T363028) | [tools.bridgebot] | 
            
  | 21:05 | <wmbot~bd808@tools-bastion-12> | Switched from legacy system to buildservice and jobs configuration (T363028) | [tools.bridgebot] | 
            
  | 21:05 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1032.eqiad.wmnet' (T356287) | [admin] | 
            
  | 21:00 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1032.eqiad.wmnet' (T356287) | [admin] | 
            
  | 20:59 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' (T356287) | [admin] | 
            
  | 20:54 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1031.eqiad.wmnet' (T356287) | [admin] | 
            
  | 20:54 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1063.eqiad.wmnet' (T356287) | [admin] | 
            
  | 20:50 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1063.eqiad.wmnet' (T356287) | [admin] | 
            
  | 20:50 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1066.eqiad.wmnet' (T356287) | [admin] | 
            
  | 20:45 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1066.eqiad.wmnet' (T356287) | [admin] | 
            
  | 20:45 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1064.eqiad.wmnet' (T356287) | [admin] | 
            
  | 20:40 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1064.eqiad.wmnet' (T356287) | [admin] | 
            
  | 20:40 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1067.eqiad.wmnet' (T356287) | [admin] | 
            
  | 20:35 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1067.eqiad.wmnet' (T356287) | [admin] | 
            
  | 20:35 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1065.eqiad.wmnet' (T356287) | [admin] | 
            
  | 20:30 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1065.eqiad.wmnet' (T356287) | [admin] | 
            
  | 20:30 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1062.eqiad.wmnet' (T356287) | [admin] | 
            
  | 20:25 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1062.eqiad.wmnet' (T356287) | [admin] | 
            
  | 20:25 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1003.eqiad.wmnet' (T356287) | [admin] | 
            
  | 20:20 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1003.eqiad.wmnet' (T356287) | [admin] | 
            
  | 20:20 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1002.eqiad.wmnet' (T356287) | [admin] | 
            
  | 20:15 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1002.eqiad.wmnet' (T356287) | [admin] | 
            
  | 20:15 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirtlocal1001.eqiad.wmnet' (T356287) | [admin] | 
            
  | 20:10 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirtlocal1001.eqiad.wmnet' (T356287) | [admin] | 
            
  | 20:10 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1054.eqiad.wmnet' (T356287) | [admin] | 
            
  | 20:05 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1054.eqiad.wmnet' (T356287) | [admin] | 
            
  | 20:05 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1055.eqiad.wmnet' (T356287) | [admin] | 
            
  | 20:00 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1055.eqiad.wmnet' (T356287) | [admin] | 
            
  | 20:00 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1060.eqiad.wmnet' (T356287) | [admin] | 
            
  | 19:55 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1060.eqiad.wmnet' (T356287) | [admin] | 
            
  | 19:55 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1058.eqiad.wmnet' (T356287) | [admin] | 
            
  | 19:51 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1058.eqiad.wmnet' (T356287) | [admin] | 
            
  | 19:51 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1059.eqiad.wmnet' (T356287) | [admin] | 
            
  | 19:46 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1059.eqiad.wmnet' (T356287) | [admin] | 
            
  | 19:46 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1061.eqiad.wmnet' (T356287) | [admin] | 
            
  | 19:41 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1061.eqiad.wmnet' (T356287) | [admin] | 
            
  | 19:41 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1057.eqiad.wmnet' (T356287) | [admin] | 
            
  | 19:36 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1057.eqiad.wmnet' (T356287) | [admin] | 
            
  | 19:36 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1056.eqiad.wmnet' (T356287) | [admin] | 
            
  | 19:33 | <ebernhardson> | T363516 started manual rebuild of enwiki titlesuggest indices in eqiad | [production] | 
            
  | 19:30 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1056.eqiad.wmnet' (T356287) | [admin] | 
            
  | 19:30 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1051.eqiad.wmnet' (T356287) | [admin] | 
            
  | 19:26 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt1051.eqiad.wmnet' (T356287) | [admin] | 
            
  | 19:25 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt1050.eqiad.wmnet' (T356287) | [admin] |