|
2024-04-17
§
|
| 04:38 |
<marostegui@cumin1002> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1222.eqiad.wmnet with reason: Maintenance |
[production] |
| 03:39 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db1206 (T352010)', diff saved to https://phabricator.wikimedia.org/P60699 and previous config saved to /var/cache/conftool/dbconfig/20240417-033948-ladsgroup.json |
[production] |
| 03:39 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1206.eqiad.wmnet with reason: Maintenance |
[production] |
| 03:39 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1206.eqiad.wmnet with reason: Maintenance |
[production] |
| 03:39 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1196 (T352010)', diff saved to https://phabricator.wikimedia.org/P60698 and previous config saved to /var/cache/conftool/dbconfig/20240417-033926-ladsgroup.json |
[production] |
| 03:24 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P60697 and previous config saved to /var/cache/conftool/dbconfig/20240417-032418-ladsgroup.json |
[production] |
| 03:09 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P60696 and previous config saved to /var/cache/conftool/dbconfig/20240417-030911-ladsgroup.json |
[production] |
| 03:01 |
<JJMC89> |
copypatrol-backend-prod-01 deploy d896631..af4b5f5 to address CVE-2024-1135 |
[copypatrol] |
| 02:54 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1196 (T352010)', diff saved to https://phabricator.wikimedia.org/P60695 and previous config saved to /var/cache/conftool/dbconfig/20240417-025403-ladsgroup.json |
[production] |
| 02:48 |
<ryankemper> |
T361525 Trying to powercycle `elastic2088` thru mgmt port (host not responding to ssh) |
[production] |
| 02:43 |
<dani@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/miscweb: apply |
[production] |
| 02:43 |
<dani@deploy1002> |
helmfile [codfw] START helmfile.d/services/miscweb: apply |
[production] |
| 02:43 |
<dani@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/miscweb: apply |
[production] |
| 02:43 |
<dani@deploy1002> |
helmfile [eqiad] START helmfile.d/services/miscweb: apply |
[production] |
| 02:43 |
<dani@deploy1002> |
helmfile [staging] DONE helmfile.d/services/miscweb: apply |
[production] |
| 02:42 |
<dani@deploy1002> |
helmfile [staging] START helmfile.d/services/miscweb: apply |
[production] |
| 01:58 |
<andrew@cloudcumin1001> |
END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) |
[admin] |
| 01:54 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.openstack.restart_openstack |
[admin] |
| 01:48 |
<andrew@cloudcumin1001> |
END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2006-dev.codfw.wmnet' (T356287) |
[admin] |
| 01:43 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2006-dev.codfw.wmnet' (T356287) |
[admin] |
| 01:40 |
<andrew@cloudcumin1001> |
END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2005-dev.codfw.wmnet' (T356287) |
[admin] |
| 01:35 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2005-dev.codfw.wmnet' (T356287) |
[admin] |
| 01:05 |
<andrew@cloudcumin1001> |
END (PASS) - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack (exit_code=0) on host 'cloudvirt2004-dev.codfw.wmnet' (T356287) |
[admin] |
| 01:00 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.openstack.cloudvirt.live_upgrade_openstack on host 'cloudvirt2004-dev.codfw.wmnet' (T356287) |
[admin] |
|
2024-04-16
§
|
| 23:45 |
<wmbot~anticomposite@tools-sgebastion-10> |
./stewardbots/StewardBot/manage.sh restart # RC reader not reading RC |
[tools.stewardbots] |
| 23:25 |
<hmonroy@deploy1002> |
Finished scap: Backport for [[gerrit:1019893|[mediawikiwiki] enable CodeMirror V6 (T357795)]] (duration: 17m 29s) |
[production] |
| 23:12 |
<hmonroy@deploy1002> |
musikanimal and hmonroy: Continuing with sync |
[production] |
| 23:11 |
<hmonroy@deploy1002> |
musikanimal and hmonroy: Backport for [[gerrit:1019893|[mediawikiwiki] enable CodeMirror V6 (T357795)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 23:08 |
<hmonroy@deploy1002> |
Started scap: Backport for [[gerrit:1019893|[mediawikiwiki] enable CodeMirror V6 (T357795)]] |
[production] |
| 23:06 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcontrol2009-dev.codfw.wmnet with OS bookworm |
[production] |
| 23:06 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" |
[production] |
| 23:03 |
<pt1979@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002" |
[production] |
| 22:46 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcontrol2009-dev.codfw.wmnet with reason: host reimage |
[production] |
| 22:43 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcontrol2009-dev.codfw.wmnet with reason: host reimage |
[production] |
| 22:32 |
<JJMC89> |
`update diffs set status=0, status_user_text=NULL where status_user_text='WizardGamer775';` # mass unreview |
[copypatrol] |
| 22:25 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.reimage for host cloudcontrol2009-dev.codfw.wmnet with OS bookworm |
[production] |
| 21:54 |
<pt1979@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcontrol2009-dev.codfw.wmnet with OS bookworm |
[production] |
| 21:52 |
<andrew@cloudcumin1001> |
END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudnet2006-dev.codfw.wmnet' (T356287) |
[admin] |
| 21:48 |
<pfischer@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 21:47 |
<pfischer@deploy1002> |
helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 21:47 |
<pfischer@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 21:47 |
<pfischer@deploy1002> |
helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 21:46 |
<pfischer@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 21:46 |
<pfischer@deploy1002> |
helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 21:46 |
<pfischer@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 21:45 |
<pfischer@deploy1002> |
helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 21:45 |
<pfischer@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 21:45 |
<pfischer@deploy1002> |
helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply |
[production] |
| 21:44 |
<andrew@cloudcumin1001> |
START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudnet2006-dev.codfw.wmnet' (T356287) |
[admin] |
| 21:44 |
<pfischer@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply |
[production] |