5851-5900 of 10000 results (98ms)
2024-04-30 ยง
19:44 <fabfur@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - fabfur@cumin1002" [production]
19:41 <fabfur@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp7007.magru.wmnet with OS bullseye [production]
19:41 <fabfur@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - fabfur@cumin1002" [production]
19:40 <fabfur@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - fabfur@cumin1002" [production]
19:38 <sukhe@cumin1002> START - Cookbook sre.hosts.reimage for host dns7002.wikimedia.org with OS bookworm [production]
19:32 <sukhe> sudo ipmitool -I lanplus -H "dns7002.mgmt.magru.wmnet" -U root -E chassis power cycle [production]
19:29 <sukhe@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp7003.magru.wmnet with OS bullseye [production]
19:27 <sukhe@cumin1002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dns7002.wikimedia.org with OS bookworm [production]
19:22 <brett@cumin2002> START - Cookbook sre.hosts.reimage for host ncredir1001.eqiad.wmnet with OS bookworm [production]
19:19 <fabfur@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp7008.magru.wmnet with reason: host reimage [production]
19:16 <fabfur@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp7006.magru.wmnet with reason: host reimage [production]
19:15 <fabfur@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on cp7008.magru.wmnet with reason: host reimage [production]
19:15 <aokoth@cumin1002> START - Cookbook sre.hosts.reimage for host lists1004.wikimedia.org with OS bookworm [production]
19:14 <fabfur@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp7007.magru.wmnet with reason: host reimage [production]
19:14 <fabfur@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on cp7006.magru.wmnet with reason: host reimage [production]
19:12 <fabfur@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on cp7007.magru.wmnet with reason: host reimage [production]
19:08 <aokoth@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host lists1004.mgmt.eqiad.wmnet with reboot policy FORCED [production]
19:04 <sukhe@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp7003.magru.wmnet with reason: host reimage [production]
19:01 <sukhe@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on cp7003.magru.wmnet with reason: host reimage [production]
18:59 <bking@deploy1002> helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply [production]
18:57 <sukhe@cumin1002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dns7001 [production]
18:56 <sukhe@cumin1002> START - Cookbook sre.network.configure-switch-interfaces for host dns7001 [production]
18:55 <bking@deploy1002> helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply [production]
18:53 <herron> updated alertmanager IRC alert text format. for details please see https://gerrit.wikimedia.org/r/c/operations/puppet/+/1019840 [production]
18:49 <sukhe@cumin1002> START - Cookbook sre.hosts.reimage for host dns7002.wikimedia.org with OS bookworm [production]
18:48 <fabfur@cumin1002> START - Cookbook sre.hosts.reimage for host cp7008.magru.wmnet with OS bullseye [production]
18:47 <sukhe> sudo cumin -b1 -s10 'C:confd and *.esams.wmnet' 'systemctl restart confd' [production]
18:46 <fabfur@cumin1002> START - Cookbook sre.hosts.reimage for host cp7006.magru.wmnet with OS bullseye [production]
18:46 <fabfur@cumin1002> START - Cookbook sre.hosts.reimage for host cp7007.magru.wmnet with OS bullseye [production]
18:41 <sukhe@cumin1002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) _etcd._tcp.esams.wmnet on all recursors [production]
18:41 <sukhe@cumin1002> START - Cookbook sre.dns.wipe-cache _etcd._tcp.esams.wmnet on all recursors [production]
18:40 <sukhe> running authdns-update for CR 1025800 [production]
18:38 <robh@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cp7014'] [production]
18:38 <robh@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp7014'] [production]
18:32 <sukhe@cumin1002> START - Cookbook sre.hosts.reimage for host cp7003.magru.wmnet with OS bullseye [production]
18:31 <sukhe@cumin1002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['dns7001.magru.wmnet'] [production]
18:31 <sukhe@cumin1002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['dns7001.magru.wmnet'] [production]
18:31 <sukhe@cumin1002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['dns7001.magru.wmnet'] [production]
18:31 <sukhe@cumin1002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['dns7001.magru.wmnet'] [production]
18:30 <aokoth@cumin1002> START - Cookbook sre.hosts.provision for host lists1004.mgmt.eqiad.wmnet with reboot policy FORCED [production]
18:26 <sukhe@cumin1002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp7014.magru.wmnet with OS bullseye [production]
18:21 <xcollazo@deploy1002> Finished deploy [analytics/refinery@4836095] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@4836095f] (duration: 02m 52s) [production]
18:18 <aokoth@cumin1002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host lists1004 [production]
18:18 <xcollazo@deploy1002> Started deploy [analytics/refinery@4836095] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@4836095f] [production]
18:18 <xcollazo@deploy1002> Finished deploy [analytics/refinery@4836095] (thin): Regular analytics weekly train THIN [analytics/refinery@4836095f] (duration: 03m 57s) [production]
18:17 <aokoth@cumin1002> START - Cookbook sre.network.configure-switch-interfaces for host lists1004 [production]
18:16 <sukhe@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
18:16 <sukhe@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: merge pending changes - sukhe@cumin1002" [production]
18:15 <sukhe@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: merge pending changes - sukhe@cumin1002" [production]
18:14 <xcollazo@deploy1002> Started deploy [analytics/refinery@4836095] (thin): Regular analytics weekly train THIN [analytics/refinery@4836095f] [production]