4551-4600 of 10000 results (95ms)
2024-04-03 ยง
15:33 <jiji@cumin1002> END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) status all services in all: None - None [production]
15:33 <jiji@cumin1002> START - Cookbook sre.discovery.datacenter status all services in all: None - None [production]
15:31 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2114', diff saved to https://phabricator.wikimedia.org/P59353 and previous config saved to /var/cache/conftool/dbconfig/20240403-153121-arnaudb.json [production]
15:22 <pfischer@deploy1002> helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
15:22 <Dreamy_Jazz> Starting MediaModeration scanning script again - It crashed due to the outage [production]
15:22 <pfischer@deploy1002> helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
15:16 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db2114 (T360332)', diff saved to https://phabricator.wikimedia.org/P59352 and previous config saved to /var/cache/conftool/dbconfig/20240403-151614-arnaudb.json [production]
15:13 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db2114 (T360332)', diff saved to https://phabricator.wikimedia.org/P59351 and previous config saved to /var/cache/conftool/dbconfig/20240403-151349-arnaudb.json [production]
15:13 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2114.codfw.wmnet with reason: Maintenance [production]
15:13 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 12:00:00 on db2114.codfw.wmnet with reason: Maintenance [production]
15:13 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2097.codfw.wmnet with reason: Maintenance [production]
15:12 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 12:00:00 on db2097.codfw.wmnet with reason: Maintenance [production]
15:12 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance [production]
15:12 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance [production]
15:12 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1231 (T360332)', diff saved to https://phabricator.wikimedia.org/P59350 and previous config saved to /var/cache/conftool/dbconfig/20240403-151233-arnaudb.json [production]
15:04 <jynus@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2098.codfw.wmnet with reason: restart of mysqld [production]
15:03 <jynus@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db2098.codfw.wmnet with reason: restart of mysqld [production]
15:02 <dreamyjazz@deploy1002> Finished scap: (no justification provided) (duration: 18m 54s) [production]
15:01 <hnowlan@deploy1002> helmfile [eqiad] DONE helmfile.d/services/wikifeeds: sync [production]
15:01 <aborrero@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1040.eqiad.wmnet with OS bookworm [production]
15:01 <hnowlan@deploy1002> helmfile [eqiad] START helmfile.d/services/wikifeeds: sync [production]
14:57 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P59349 and previous config saved to /var/cache/conftool/dbconfig/20240403-145725-arnaudb.json [production]
14:44 <dreamyjazz@deploy1002> Started scap: (no justification provided) [production]
14:42 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P59347 and previous config saved to /var/cache/conftool/dbconfig/20240403-144217-arnaudb.json [production]
14:34 <aborrero@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1040.eqiad.wmnet with reason: host reimage [production]
14:31 <aborrero@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1040.eqiad.wmnet with reason: host reimage [production]
14:27 <hnowlan@deploy1002> helmfile [eqiad] DONE helmfile.d/admin 'apply'. [production]
14:27 <arnaudb@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1231 (T360332)', diff saved to https://phabricator.wikimedia.org/P59346 and previous config saved to /var/cache/conftool/dbconfig/20240403-142709-arnaudb.json [production]
14:27 <hnowlan@deploy1002> helmfile [eqiad] START helmfile.d/admin 'apply'. [production]
14:26 <hnowlan@deploy1002> helmfile [codfw] DONE helmfile.d/admin 'apply'. [production]
14:26 <hnowlan@deploy1002> helmfile [codfw] START helmfile.d/admin 'apply'. [production]
14:24 <aborrero@cumin1002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1040 [production]
14:24 <aborrero@cumin1002> START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1040 [production]
14:21 <marostegui@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance [production]
14:21 <marostegui@cumin1002> START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance [production]
14:21 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1226 (T356166)', diff saved to https://phabricator.wikimedia.org/P59345 and previous config saved to /var/cache/conftool/dbconfig/20240403-142142-marostegui.json [production]
14:17 <jmm@cumin2002> END (PASS) - Cookbook sre.maps.roll-restart-reboot (exit_code=0) rolling restart_daemons on A:maps-replica-eqiad [production]
14:16 <aborrero@cumin1002> START - Cookbook sre.hosts.reimage for host cloudvirt1040.eqiad.wmnet with OS bookworm [production]
14:11 <jmm@cumin2002> START - Cookbook sre.maps.roll-restart-reboot rolling restart_daemons on A:maps-replica-eqiad [production]
14:11 <jmm@cumin2002> END (PASS) - Cookbook sre.maps.roll-restart-reboot (exit_code=0) rolling restart_daemons on A:maps-replica-codfw [production]
14:09 <pfischer@deploy1002> helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
14:09 <pfischer@deploy1002> helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
14:08 <ayounsi@cumin1002> END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 2527 [production]
14:07 <ayounsi@cumin1002> START - Cookbook sre.network.peering with action 'configure' for AS: 2527 [production]
14:06 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P59344 and previous config saved to /var/cache/conftool/dbconfig/20240403-140634-marostegui.json [production]
14:06 <jmm@cumin2002> START - Cookbook sre.maps.roll-restart-reboot rolling restart_daemons on A:maps-replica-codfw [production]
13:51 <marostegui@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P59343 and previous config saved to /var/cache/conftool/dbconfig/20240403-135126-marostegui.json [production]
13:41 <arnaudb@cumin1002> dbctl commit (dc=all): 'Depooling db1231 (T360332)', diff saved to https://phabricator.wikimedia.org/P59342 and previous config saved to /var/cache/conftool/dbconfig/20240403-134136-arnaudb.json [production]
13:41 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1231.eqiad.wmnet with reason: Maintenance [production]
13:41 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 12:00:00 on db1231.eqiad.wmnet with reason: Maintenance [production]