9751-9800 of 10000 results (69ms)
2024-05-28 ยง
20:34 <cjming@deploy1002> Finished scap: Backport for [[gerrit:1036723|cirrus: Move remaining public writes to SUP (T363475)]] (duration: 12m 11s) [production]
20:31 <eileen> config revision changed from 6c4cd6c2 to 5b0b4d22 revert schedule [production]
20:25 <cjming@deploy1002> cjming and ebernhardson: Continuing with sync [production]
20:25 <cjming@deploy1002> cjming and ebernhardson: Backport for [[gerrit:1036723|cirrus: Move remaining public writes to SUP (T363475)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
20:22 <cjming@deploy1002> Started scap: Backport for [[gerrit:1036723|cirrus: Move remaining public writes to SUP (T363475)]] [production]
20:21 <eileen> revert to Smarty 2 revision changed from de15d068 to cdc89b59 [production]
20:20 <cjming@deploy1002> Finished scap: Backport for [[gerrit:1034480|deploy(Popups): Make use of conditional user defaults (T364347)]] (duration: 15m 52s) [production]
20:19 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1174 (T364299)', diff saved to https://phabricator.wikimedia.org/P63478 and previous config saved to /var/cache/conftool/dbconfig/20240528-201945-marostegui.json [production]
20:11 <cjming@deploy1002> mabualruz and cjming: Continuing with sync [production]
20:08 <sfaci@deploy1002> helmfile [staging] DONE helmfile.d/services/device-analytics: apply [production]
20:07 <sfaci@deploy1002> helmfile [staging] START helmfile.d/services/device-analytics: apply [production]
20:07 <cjming@deploy1002> mabualruz and cjming: Backport for [[gerrit:1034480|deploy(Popups): Make use of conditional user defaults (T364347)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
20:05 <jclark@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host kafka-main1010.eqiad.wmnet with OS bullseye [production]
20:04 <cjming@deploy1002> Started scap: Backport for [[gerrit:1034480|deploy(Popups): Make use of conditional user defaults (T364347)]] [production]
19:59 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [admin]
19:53 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.restart_openstack [admin]
19:52 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [admin]
19:45 <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Unbanning all hosts in search_eqiad [production]
19:45 <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Unbanning all hosts in search_eqiad [production]
19:43 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.restart_openstack [admin]
19:38 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) [admin]
19:37 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.restart_openstack [admin]
19:36 <jclark@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kafka-main1009.eqiad.wmnet with OS bullseye [production]
19:30 <herron> disable swap on grafana1002 [production]
19:28 <logmsgbot> @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
19:28 <logmsgbot> @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
19:28 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) [admin]
19:27 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.restart_openstack [admin]
19:26 <logmsgbot> @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
19:26 <logmsgbot> @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
19:24 <logmsgbot> @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
19:24 <logmsgbot> @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
19:24 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [admin]
19:24 <herron> ganeti1027:~$ sudo gnt-instance reboot grafana1002 [production]
19:24 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.restart_openstack [admin]
19:22 <logmsgbot> @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
19:22 <logmsgbot> @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
19:22 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) [admin]
19:21 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.restart_openstack [admin]
19:21 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [admin]
19:20 <logmsgbot> @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
19:20 <logmsgbot> @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
19:19 <jclark@cumin1002> START - Cookbook sre.hosts.reimage for host kafka-main1010.eqiad.wmnet with OS bullseye [production]
19:19 <logmsgbot> @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
19:18 <logmsgbot> @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
19:17 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.restart_openstack [admin]
19:16 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.restart_openstack (exit_code=99) [admin]
19:14 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.restart_openstack [admin]
19:01 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host elastic1056.eqiad.wmnet [production]
19:00 <marostegui@cumin1002> dbctl commit (dc=all): 'db1211 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P63476 and previous config saved to /var/cache/conftool/dbconfig/20240528-190021-root.json [production]