451-500 of 10000 results (87ms)
2025-08-29 ยง
16:30 <jhancock@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1002" [production]
16:27 <jhancock@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on maps2011.codfw.wmnet with reason: host reimage [production]
16:26 <jhancock@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host maps2013.codfw.wmnet with OS bookworm [production]
16:26 <jhancock@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1002" [production]
16:24 <jhancock@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin1002" [production]
16:22 <jhancock@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on maps2011.codfw.wmnet with reason: host reimage [production]
16:12 <jhancock@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on maps2014.codfw.wmnet with reason: host reimage [production]
16:08 <jhancock@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on maps2013.codfw.wmnet with reason: host reimage [production]
16:03 <jhancock@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on maps2014.codfw.wmnet with reason: host reimage [production]
16:03 <jhancock@cumin1002> START - Cookbook sre.hosts.reimage for host maps2011.codfw.wmnet with OS bookworm [production]
16:02 <jhancock@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on maps2013.codfw.wmnet with reason: host reimage [production]
15:54 <jhancock@cumin1003> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host maps2011.codfw.wmnet with OS bookworm [production]
15:45 <jhancock@cumin1002> START - Cookbook sre.hosts.reimage for host maps2014.codfw.wmnet with OS bookworm [production]
15:44 <jhancock@cumin1002> START - Cookbook sre.hosts.reimage for host maps2013.codfw.wmnet with OS bookworm [production]
15:43 <jhancock@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host maps2014.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
15:42 <jhancock@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host maps2013.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
15:24 <jhancock@cumin1002> START - Cookbook sre.hosts.provision for host maps2014.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
15:24 <jhancock@cumin1002> START - Cookbook sre.hosts.provision for host maps2013.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
15:21 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host maps2012.codfw.wmnet with OS bookworm [production]
15:21 <jclark@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" [production]
15:21 <jclark@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" [production]
15:03 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on maps2012.codfw.wmnet with reason: host reimage [production]
14:59 <jclark@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on maps2012.codfw.wmnet with reason: host reimage [production]
14:52 <jhancock@cumin1003> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host maps2011.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
14:46 <jhancock@cumin1003> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host maps2012.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
14:40 <jclark@cumin1002> START - Cookbook sre.hosts.reimage for host maps2012.codfw.wmnet with OS bookworm [production]
14:36 <jhancock@cumin1003> START - Cookbook sre.hosts.provision for host maps2011.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
14:30 <jhancock@cumin1003> START - Cookbook sre.hosts.provision for host maps2012.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
14:23 <jhancock@cumin1003> START - Cookbook sre.hosts.reimage for host maps2011.codfw.wmnet with OS bookworm [production]
13:47 <elukey@cumin1003> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1013.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [production]
13:43 <dcausse> restarting blazegraph wdqs on wdqs1022 (stuck) [production]
13:43 <ladsgroup@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2239.codfw.wmnet with reason: Maintenance [production]
13:43 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2227 (T402925)', diff saved to https://phabricator.wikimedia.org/P82113 and previous config saved to /var/cache/conftool/dbconfig/20250829-134308-ladsgroup.json [production]
13:36 <elukey@cumin1003> START - Cookbook sre.hosts.provision for host ml-serve1013.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [production]
13:28 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2227', diff saved to https://phabricator.wikimedia.org/P82112 and previous config saved to /var/cache/conftool/dbconfig/20250829-132800-ladsgroup.json [production]
13:17 <elukey@cumin1003> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-serve1013.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [production]
13:12 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2227', diff saved to https://phabricator.wikimedia.org/P82110 and previous config saved to /var/cache/conftool/dbconfig/20250829-131253-ladsgroup.json [production]
13:12 <elukey@puppetserver1001> conftool action : set/pooled=false; selector: dnsdisc=kartotherian,name=codfw [production]
13:06 <elukey@cumin1003> START - Cookbook sre.hosts.provision for host ml-serve1013.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART [production]
12:57 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2227 (T402925)', diff saved to https://phabricator.wikimedia.org/P82109 and previous config saved to /var/cache/conftool/dbconfig/20250829-125745-ladsgroup.json [production]
12:41 <arnaudb@cumin1003> END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Update [production]
12:35 <elukey@puppetserver1001> conftool action : set/pooled=true; selector: dnsdisc=kartotherian,name=eqiad [production]
12:33 <pfischer@deploy1003> helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
12:33 <pfischer@deploy1003> helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
12:33 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host failoid2003.codfw.wmnet with OS trixie [production]
12:28 <pfischer@deploy1003> helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
12:28 <pfischer@deploy1003> helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
12:22 <pfischer@deploy1003> helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
12:22 <pfischer@deploy1003> helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
12:18 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on failoid2003.codfw.wmnet with reason: host reimage [production]