5901-5950 of 10000 results (96ms)
2024-03-06 §
16:34 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
16:30 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P58591 and previous config saved to /var/cache/conftool/dbconfig/20240306-163053-ladsgroup.json [production]
16:26 <denisse> Disable meta-monitoring for alert1001 - T333615 [production]
16:15 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2124 (T352010)', diff saved to https://phabricator.wikimedia.org/P58590 and previous config saved to /var/cache/conftool/dbconfig/20240306-161546-ladsgroup.json [production]
16:05 <logmsgbot> @deploy2002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
16:05 <logmsgbot> @deploy2002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
16:00 <jiji@cumin1002> START - Cookbook sre.switchdc.mediawiki.09-run-puppet-on-db-masters [production]
16:00 <cgoubert@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1451.eqiad.wmnet with OS bullseye [production]
15:59 <jiji@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.09-restore-ttl (exit_code=0) [production]
15:59 <jiji@cumin1002> START - Cookbook sre.switchdc.mediawiki.09-restore-ttl [production]
15:57 <cgoubert@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1455.eqiad.wmnet with OS bullseye [production]
15:57 <jiji@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.08-start-maintenance (exit_code=0) [production]
15:55 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2006.mgmt.codfw.wmnet with reboot policy FORCED [production]
15:55 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host dbprov2005.mgmt.codfw.wmnet with reboot policy FORCED [production]
15:55 <cgoubert@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1442.eqiad.wmnet with OS bullseye [production]
15:54 <jiji@cumin1002> START - Cookbook sre.switchdc.mediawiki.08-start-maintenance [production]
15:51 <cgoubert@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1452.eqiad.wmnet with OS bullseye [production]
15:50 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host dbprov2005.mgmt.codfw.wmnet with reboot policy FORCED [production]
15:50 <cgoubert@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1454.eqiad.wmnet with OS bullseye [production]
15:50 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host dbprov2006.mgmt.codfw.wmnet with reboot policy FORCED [production]
15:49 <jiji@cumin1002> END (FAIL) - Cookbook sre.switchdc.mediawiki.08-restart-mw-jobrunner (exit_code=99) [production]
15:48 <root@deploy2002> helmfile [eqiad] [canary] FAIL helmfile.d/services/mw-jobrunner : sync [production]
15:48 <root@deploy2002> helmfile [eqiad] [main] FAIL helmfile.d/services/mw-jobrunner : sync [production]
15:48 <root@deploy2002> helmfile [eqiad] [canary] START helmfile.d/services/mw-jobrunner : sync [production]
15:48 <root@deploy2002> helmfile [eqiad] [main] START helmfile.d/services/mw-jobrunner : sync [production]
15:48 <jiji@cumin1002> START - Cookbook sre.switchdc.mediawiki.08-restart-mw-jobrunner [production]
15:48 <jiji@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) [production]
15:48 <jiji@cumin1002> [DRY-RUN] MediaWiki read-only period ends at: 2024-03-06 15:48:02.718097 [production]
15:47 <cgoubert@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1441.eqiad.wmnet with OS bullseye [production]
15:47 <jiji@cumin1002> START - Cookbook sre.switchdc.mediawiki.07-set-readwrite [production]
15:46 <jiji@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0) [production]
15:46 <jiji@cumin1002> START - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite [production]
15:45 <jiji@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0) [production]
15:45 <jiji@cumin1002> START - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki [production]
15:44 <eevans@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on restbase1039.eqiad.wmnet with reason: Bootstrapping — T354560 [production]
15:44 <eevans@cumin1002> START - Cookbook sre.hosts.downtime for 30 days, 0:00:00 on restbase1039.eqiad.wmnet with reason: Bootstrapping — T354560 [production]
15:44 <jiji@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0) [production]
15:43 <jiji@cumin1002> [DRY-RUN] MediaWiki read-only period starts at: 2024-03-06 15:43:44.970687 [production]
15:43 <jiji@cumin1002> START - Cookbook sre.switchdc.mediawiki.02-set-readonly [production]
15:42 <cgoubert@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1451.eqiad.wmnet with reason: host reimage [production]
15:39 <cgoubert@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1455.eqiad.wmnet with reason: host reimage [production]
15:36 <cgoubert@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1442.eqiad.wmnet with reason: host reimage [production]
15:34 <jiji@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0) [production]
15:34 <jiji@cumin1002> START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance [production]
15:34 <cgoubert@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1452.eqiad.wmnet with reason: host reimage [production]
15:31 <cgoubert@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1454.eqiad.wmnet with reason: host reimage [production]
15:31 <bking@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wdqs1025.eqiad.wmnet with OS bullseye [production]
15:29 <cgoubert@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1441.eqiad.wmnet with reason: host reimage [production]
15:28 <jiji@cumin1002> END (PASS) - Cookbook sre.switchdc.mediawiki.00-downtime-db-readonly-checks (exit_code=0) [production]
15:28 <cgoubert@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1455.eqiad.wmnet with reason: host reimage [production]