5851-5900 of 10000 results (119ms)
2024-03-06 ยง
19:55 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
19:55 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
19:05 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
19:04 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
19:00 <bking@cumin2002> START - Cookbook sre.hosts.reimage for host wdqs1025.eqiad.wmnet with OS bullseye [production]
18:59 <bking@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['wdqs1025'] [production]
18:59 <bking@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wdqs1025'] [production]
18:59 <bking@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wdqs1025.eqiad.wmnet with OS bullseye [production]
18:46 <jnuche@deploy2002> Finished deploy [releng/jenkins-deploy@af71f6e] (releasing): (no justification provided) (duration: 00m 41s) [production]
18:45 <jnuche@deploy2002> Started deploy [releng/jenkins-deploy@af71f6e] (releasing): (no justification provided) [production]
18:21 <urbanecm@deploy2002> Finished scap: Backport for [[gerrit:1009326|JS REST: make POST default to empty object (T359216)]] (duration: 14m 19s) [production]
18:12 <urbanecm@deploy2002> urbanecm: Continuing with sync [production]
18:11 <urbanecm@deploy2002> urbanecm: Backport for [[gerrit:1009326|JS REST: make POST default to empty object (T359216)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
18:07 <urbanecm@deploy2002> Started scap: Backport for [[gerrit:1009326|JS REST: make POST default to empty object (T359216)]] [production]
17:53 <bking@cumin2002> START - Cookbook sre.hosts.reimage for host wdqs1025.eqiad.wmnet with OS bullseye [production]
17:39 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2196 (re)pooling @ 100%: Clone source repooling', diff saved to https://phabricator.wikimedia.org/P58598 and previous config saved to /var/cache/conftool/dbconfig/20240306-173954-arnaudb.json [production]
17:25 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
17:25 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2196 (re)pooling @ 75%: Clone source repooling', diff saved to https://phabricator.wikimedia.org/P58597 and previous config saved to /var/cache/conftool/dbconfig/20240306-172449-arnaudb.json [production]
17:24 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
17:18 <denisse> failing over from alert2001 to alert1001 [production]
17:17 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
17:17 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
17:15 <denisse@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host alert1001.wikimedia.org with OS bookworm [production]
17:10 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
17:10 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
17:09 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2196 (re)pooling @ 50%: Clone source repooling', diff saved to https://phabricator.wikimedia.org/P58596 and previous config saved to /var/cache/conftool/dbconfig/20240306-170944-arnaudb.json [production]
17:06 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
17:06 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
17:02 <claime> restart rsyslog on mw2436 [production]
17:01 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db2151 (T352010)', diff saved to https://phabricator.wikimedia.org/P58595 and previous config saved to /var/cache/conftool/dbconfig/20240306-170125-ladsgroup.json [production]
17:01 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2151.codfw.wmnet with reason: Maintenance [production]
17:01 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2151.codfw.wmnet with reason: Maintenance [production]
17:01 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2124 (T352010)', diff saved to https://phabricator.wikimedia.org/P58594 and previous config saved to /var/cache/conftool/dbconfig/20240306-170106-ladsgroup.json [production]
16:56 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
16:56 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
16:54 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2196 (re)pooling @ 25%: Clone source repooling', diff saved to https://phabricator.wikimedia.org/P58593 and previous config saved to /var/cache/conftool/dbconfig/20240306-165439-arnaudb.json [production]
16:52 <cgoubert@cumin2002> conftool action : set/weight=10:pooled=yes; selector: name=(mw1441.eqiad.wmnet|mw1442.eqiad.wmnet|mw1451.eqiad.wmnet|mw1452.eqiad.wmnet|mw1454.eqiad.wmnet|mw1455.eqiad.wmnet),cluster=kubernetes,service=kubesvc [production]
16:52 <claime> Pooling and uncordoning mw1441.eqiad.wmnet,mw1442.eqiad.wmnet,mw1451.eqiad.wmnet,mw1452.eqiad.wmnet,mw1454.eqiad.wmnet,mw1455.eqiad.wmnet - T351074 [production]
16:52 <denisse@cumin2002> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on alert1001.wikimedia.org with reason: host reimage [production]
16:49 <arnaudb@cumin1002> END (PASS) - Cookbook sre.mysql.clone (exit_code=0) Will create a clone of db2196.codfw.wmnet onto db2131.codfw.wmnet [production]
16:49 <volans> uploaded spicerack_8.4.1 to apt.wikimedia.org bullseye-wikimedia [production]
16:48 <denisse@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on alert1001.wikimedia.org with reason: host reimage [production]
16:46 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P58592 and previous config saved to /var/cache/conftool/dbconfig/20240306-164559-ladsgroup.json [production]
16:44 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
16:44 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
16:38 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
16:38 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
16:36 <claime> Running homer 'cr*eqiad*' commit 'T351074' [production]
16:36 <denisse@cumin2002> START - Cookbook sre.hosts.reimage for host alert1001.wikimedia.org with OS bookworm [production]
16:35 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]