1-50 of 10000 results (84ms)
2024-10-09 ยง
16:32 <bvibber> starting requeueTranscodes on old school mwmaint2002 after the k8s blowup last night [production]
16:23 <sukhe> running authdns-update to fix broken zone files on dns2004 [production]
16:23 <sukhe@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:23 <sukhe@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: picking up zone file 1.0.e.f.0.0.1.a.0.8.c.e.2.0.a.2.ip6.arpa - sukhe@cumin1002" [production]
16:23 <sukhe@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: picking up zone file 1.0.e.f.0.0.1.a.0.8.c.e.2.0.a.2.ip6.arpa - sukhe@cumin1002" [production]
16:21 <sukhe> forcing commit 95858bae44a2ccae5e7fb1fe793cd3bbc7ed9c6b through sre.dns.netbox [production]
16:20 <sukhe@cumin1002> START - Cookbook sre.dns.netbox [production]
16:07 <sukhe@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:05 <sukhe@cumin1002> START - Cookbook sre.dns.netbox [production]
16:03 <jhancock@cumin2002> START - Cookbook sre.hosts.reimage for host mc-misc2002.codfw.wmnet with OS bookworm [production]
16:03 <jhancock@cumin2002> START - Cookbook sre.hosts.reimage for host mc-misc2001.codfw.wmnet with OS bookworm [production]
16:00 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc-misc2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
15:58 <sukhe@cumin1002> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for dns2005.wikimedia.org [production]
15:58 <sukhe@cumin1002> START - Cookbook sre.hosts.remove-downtime for dns2005.wikimedia.org [production]
15:54 <sukhe@cumin1002> END (ERROR) - Cookbook sre.dns.roll-reboot (exit_code=97) rolling reboot on A:dnsbox [production]
15:53 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host mc-misc2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
15:53 <sukhe> running authdns-update [production]
15:52 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mc-misc2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
15:52 <jhathaway@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mx-in2001.wikimedia.org [production]
15:49 <sukhe@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on lvs[5004-5006].eqsin.wmnet with reason: site is depooled, cr2-eqsin is being replaced [production]
15:49 <sukhe@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on lvs[5004-5006].eqsin.wmnet with reason: site is depooled, cr2-eqsin is being replaced [production]
15:48 <jhathaway@cumin1002> START - Cookbook sre.hosts.reboot-single for host mx-in2001.wikimedia.org [production]
15:48 <jhathaway@cumin1002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host mx-in1001.wikimedia.org [production]
15:47 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host mc-misc2002.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
15:45 <sukhe@cumin1002> cookbooks.sre.dns.roll-reboot begin reboot of dns2005.wikimedia.org [production]
15:44 <jhathaway@cumin1002> START - Cookbook sre.hosts.reboot-single for host mx-in1001.wikimedia.org [production]
15:43 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc-misc2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
15:43 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host mc-misc2001.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
15:43 <sukhe@cumin1002> END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling reboot on A:wikidough and A:wikidough [production]
15:30 <sukhe@cumin1002> cookbooks.sre.dns.roll-reboot finished rebooting dns2004.wikimedia.org [production]
15:26 <sukhe@cumin1002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) idp.wikimedia.org on all recursors [production]
15:26 <sukhe@cumin1002> START - Cookbook sre.dns.wipe-cache idp.wikimedia.org on all recursors [production]
15:25 <fabfur> eqsin depooled for T375961 [production]
15:24 <fabfur@cumin1002> END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool site eqsin [reason: eqsin cr replacement, T375961] [production]
15:24 <fabfur@cumin1002> START - Cookbook sre.dns.admin DNS admin: depool site eqsin [reason: eqsin cr replacement, T375961] [production]
15:24 <fabfur@cumin1002> END (FAIL) - Cookbook sre.dns.admin (exit_code=99) DNS admin: depool site eqsin [reason: eqsin cr replacementAA, T375961] [production]
15:24 <fabfur@cumin1002> START - Cookbook sre.dns.admin DNS admin: depool site eqsin [reason: eqsin cr replacementAA, T375961] [production]
15:23 <mutante> stewards* - rebooting machines - T351202 [production]
15:22 <cmooney@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:22 <cmooney@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add IPv6 reverse entry for cloudsw1-b1-codfw interface IPs - cmooney@cumin1002" [production]
15:22 <cmooney@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add IPv6 reverse entry for cloudsw1-b1-codfw interface IPs - cmooney@cumin1002" [production]
15:21 <sukhe@cumin1002> cookbooks.sre.dns.roll-reboot begin reboot of dns2004.wikimedia.org [production]
15:20 <sukhe> running dummy authdns-update [production]
15:19 <cmooney@cumin1002> START - Cookbook sre.dns.netbox [production]
15:17 <mutante> planet.wikimedia.org - rebooting backends [production]
15:09 <mutante> people.wikimedia.org - rebooting backends [production]
15:09 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ldap-maint2001.codfw.wmnet [production]
15:07 <sukhe@cumin1002> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for dns1006.wikimedia.org [production]
15:07 <sukhe@cumin1002> START - Cookbook sre.hosts.remove-downtime for dns1006.wikimedia.org [production]
15:06 <sukhe@cumin1002> cookbooks.sre.dns.roll-reboot finished rebooting dns1006.wikimedia.org [production]