651-700 of 10000 results (94ms)
2024-08-06 ยง
19:42 <ryankemper@deploy1003> Started deploy [wdqs/wdqs@316bf7f]: deploy to freshly reimaged host [production]
19:38 <ryankemper@deploy1003> Finished deploy [wdqs/wdqs@316bf7f]: deploy to freshly reimaged host (duration: 00m 18s) [production]
19:38 <ryankemper@deploy1003> Started deploy [wdqs/wdqs@316bf7f]: deploy to freshly reimaged host [production]
19:35 <ryankemper@deploy1003> Finished deploy [wdqs/wdqs@316bf7f]: deploy to freshly reimaged host (duration: 00m 09s) [production]
19:35 <ryankemper@deploy1003> Started deploy [wdqs/wdqs@316bf7f]: deploy to freshly reimaged host [production]
19:21 <sukhe> start pybal on lvs4008 [production]
19:19 <sukhe> restart varnishmtail on cp3070 [production]
19:16 <sukhe@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs4008.ulsfo.wmnet [production]
19:13 <sukhe@cumin1002> START - Cookbook sre.hosts.reboot-single for host lvs4008.ulsfo.wmnet [production]
19:04 <jhancock@cumin2002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ml-serve2011 [production]
19:03 <jhancock@cumin2002> START - Cookbook sre.network.configure-switch-interfaces for host ml-serve2011 [production]
19:03 <jhancock@cumin2002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ml-serve2010 [production]
19:03 <jhancock@cumin2002> START - Cookbook sre.network.configure-switch-interfaces for host ml-serve2010 [production]
19:03 <jhancock@cumin2002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ml-serve2009 [production]
19:03 <jhancock@cumin2002> START - Cookbook sre.network.configure-switch-interfaces for host ml-serve2009 [production]
18:57 <sukhe> sudo cumin "lvs4008*" 'disable-puppet "rebooting" && systemctl stop pybal.service' [production]
18:49 <dancy@deploy1003> Finished scap: testing T371904 (duration: 04m 14s) [production]
18:48 <sukhe> re-enable pybal on lvs6001 [production]
18:47 <sukhe@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs6001.drmrs.wmnet [production]
18:45 <dancy@deploy1003> Started scap sync-world: testing T371904 [production]
18:44 <sukhe@cumin1002> START - Cookbook sre.hosts.reboot-single for host lvs6001.drmrs.wmnet [production]
18:44 <dancy@deploy1003> Finished scap: testing T370934 (duration: 31m 05s) [production]
18:41 <brett> start pybal on lvs5005 [production]
18:36 <brett@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs5005.eqsin.wmnet [production]
18:33 <brett@cumin2002> START - Cookbook sre.hosts.reboot-single for host lvs5005.eqsin.wmnet [production]
18:28 <sukhe> sudo cumin "lvs6001*" 'disable-puppet "rebooting" && systemctl stop pybal.service' [production]
18:18 <brett> stop pybal on lvs5005 for server reboot [production]
18:13 <dancy@deploy1003> Started scap sync-world: testing T370934 [production]
17:53 <sukhe@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs5004.eqsin.wmnet [production]
17:50 <sukhe@cumin1002> START - Cookbook sre.hosts.reboot-single for host lvs5004.eqsin.wmnet [production]
17:47 <sukhe> stop pybal on lvs5004 for server reboot [production]
17:40 <mutante> CI - adding a new SSH key to jenkins - in the same file without removing the old key yet - this is expected to have no effect, but if CI breaks will revert - T177826 [production]
17:01 <fnegri@cumin1002> conftool action : set/pooled=yes; selector: name=clouddb1020.eqiad.wmnet,service=s5 [production]
17:01 <fnegri@cumin1002> conftool action : set/pooled=yes; selector: name=clouddb1020.eqiad.wmnet,service=s8 [production]
16:56 <fnegri@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host clouddb1020.eqiad.wmnet with OS bookworm [production]
16:44 <ryankemper@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wdqs1023.eqiad.wmnet with OS bullseye [production]
16:39 <jhancock@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:39 <jhancock@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding payments200 to codfw - jhancock@cumin2002" [production]
16:39 <jhancock@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding payments200 to codfw - jhancock@cumin2002" [production]
16:35 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
16:23 <fnegri@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on clouddb1020.eqiad.wmnet with reason: host reimage [production]
16:21 <fnegri@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on clouddb1020.eqiad.wmnet with reason: host reimage [production]
16:08 <fnegri@cumin1002> START - Cookbook sre.hosts.reimage for host clouddb1020.eqiad.wmnet with OS bookworm [production]
16:08 <sukhe> sudo cumin "A:dnsbox" "run-puppet-agent --enable 'upgrading anycast-hc'": finish anycast-hc upgrade: T370068 [production]
16:08 <sukhe> sudo cumin "A:dnsbox" "run-puppet-agent --enable 'upgrading anycast-hc'": finish anycast-hc upgrade [production]
16:03 <fnegri@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on clouddb1020.eqiad.wmnet with reason: Reimaging clouddb1020 T365424 [production]
16:03 <fnegri@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on clouddb1020.eqiad.wmnet with reason: Reimaging clouddb1020 T365424 [production]
15:46 <jhancock@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:46 <jhancock@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding ml-serve2011 to codfw - jhancock@cumin2002" [production]
15:46 <jhancock@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding ml-serve2011 to codfw - jhancock@cumin2002" [production]