4401-4450 of 10000 results (128ms)
2024-11-22 ยง
17:41 <herron@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host aux-k8s-worker2002.codfw.wmnet with OS bookworm [production]
17:32 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host es2045.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
17:31 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host es2046.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
17:28 <jhancock@cumin2002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host es2042 [production]
17:28 <jhancock@cumin2002> START - Cookbook sre.network.configure-switch-interfaces for host es2042 [production]
17:25 <herron@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on aux-k8s-worker2002.codfw.wmnet with reason: host reimage [production]
17:23 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host es2044.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
17:23 <cmooney@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cloudsw1-d5-eqiad.mgmt,cloudsw1-e4-eqiad.mgmt with reason: replace optics on faulty WMCS link from D5 to E4 [production]
17:22 <cmooney@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on cloudsw1-d5-eqiad.mgmt,cloudsw1-e4-eqiad.mgmt with reason: replace optics on faulty WMCS link from D5 to E4 [production]
17:22 <herron@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on aux-k8s-worker2002.codfw.wmnet with reason: host reimage [production]
17:20 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host es2046.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
17:20 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host es2045.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
17:11 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host es2044.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
17:09 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host es2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
17:08 <herron@cumin1002> START - Cookbook sre.hosts.reimage for host aux-k8s-worker2002.codfw.wmnet with OS bookworm [production]
17:06 <herron@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM aux-k8s-worker2002.codfw.wmnet - herron@cumin1002" [production]
17:06 <herron@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM aux-k8s-worker2002.codfw.wmnet - herron@cumin1002" [production]
17:05 <herron@cumin1002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) aux-k8s-worker2002.codfw.wmnet on all recursors [production]
17:05 <herron@cumin1002> START - Cookbook sre.dns.wipe-cache aux-k8s-worker2002.codfw.wmnet on all recursors [production]
17:05 <herron@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
17:05 <herron@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM aux-k8s-worker2002.codfw.wmnet - herron@cumin1002" [production]
17:05 <herron@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM aux-k8s-worker2002.codfw.wmnet - herron@cumin1002" [production]
17:00 <herron@cumin1002> START - Cookbook sre.dns.netbox [production]
17:00 <herron@cumin1002> START - Cookbook sre.ganeti.makevm for new host aux-k8s-worker2002.codfw.wmnet [production]
16:57 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host es2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
16:54 <herron@cumin1002> END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd2003.codfw.wmnet to plain [production]
16:53 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host es2042.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
16:53 <herron@cumin1002> START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd2003.codfw.wmnet to plain [production]
16:48 <herron@cumin1002> END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd2004.codfw.wmnet to plain [production]
16:47 <herron@cumin1002> START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd2004.codfw.wmnet to plain [production]
16:43 <herron@cumin1002> END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of aux-k8s-etcd2005.codfw.wmnet to plain [production]
16:43 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es2041.codfw.wmnet with OS bookworm [production]
16:43 <elukey@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1002" [production]
16:43 <elukey@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - elukey@cumin1002" [production]
16:42 <herron@cumin1002> START - Cookbook sre.ganeti.changedisk for changing disk type of aux-k8s-etcd2005.codfw.wmnet to plain [production]
16:40 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host es2042.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
16:27 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es2041.codfw.wmnet with reason: host reimage [production]
16:24 <elukey@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on es2041.codfw.wmnet with reason: host reimage [production]
16:12 <claime> homer 'cr*codfw*' commit 'T380473' [production]
16:11 <cgoubert@cumin1002> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts parse[2002-2020].codfw.wmnet [production]
16:11 <cgoubert@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:10 <cgoubert@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: parse[2002-2020].codfw.wmnet decommissioned, removing all IPs except the asset tag one - cgoubert@cumin1002" [production]
16:10 <cgoubert@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: parse[2002-2020].codfw.wmnet decommissioned, removing all IPs except the asset tag one - cgoubert@cumin1002" [production]
16:09 <elukey@cumin1002> START - Cookbook sre.hosts.reimage for host es2041.codfw.wmnet with OS bookworm [production]
16:08 <bking@deploy2002> Finished deploy [wdqs/wdqs@9927a5a]: 0.3.150 (duration: 03m 00s) [production]
16:07 <cgoubert@cumin1002> START - Cookbook sre.dns.netbox [production]
16:05 <bking@deploy2002> Started deploy [wdqs/wdqs@9927a5a]: 0.3.150 [production]
16:00 <elukey@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host es2041.codfw.wmnet with OS bookworm [production]
15:31 <cgoubert@cumin1002> START - Cookbook sre.hosts.decommission for hosts parse[2002-2020].codfw.wmnet [production]
15:31 <elukey@cumin1002> START - Cookbook sre.hosts.reimage for host es2041.codfw.wmnet with OS bookworm [production]