1-50 of 10000 results (23ms)
2026-08-12 ยง
08:07 <btullis@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on an-master1003.eqiad.wmnet with reason: host reimage [production]
08:04 <btullis@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-web1001.eqiad.wmnet with reason: host reimage [production]
07:58 <btullis@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on an-web1001.eqiad.wmnet with reason: host reimage [production]
07:50 <btullis@cumin1003> START - Cookbook sre.hosts.reimage for host an-master1003.eqiad.wmnet with OS bookworm [production]
07:38 <btullis@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-presto1011.eqiad.wmnet with reason: host reimage [production]
07:38 <btullis@cumin1003> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host an-master1003.eqiad.wmnet with OS bookworm [production]
07:35 <jmm@cumin2003> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3007.esams.wmnet [production]
07:35 <jmm@cumin2003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti3007.esams.wmnet [production]
07:33 <btullis@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on an-presto1011.eqiad.wmnet with reason: host reimage [production]
07:27 <jmm@cumin2003> START - Cookbook sre.hosts.reboot-single for host ganeti3007.esams.wmnet [production]
07:25 <jmm@cumin2003> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3007.esams.wmnet [production]
07:25 <jmm@cumin2003> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti3008.esams.wmnet [production]
07:24 <jmm@cumin2003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti3008.esams.wmnet [production]
07:22 <btullis@cumin1003> START - Cookbook sre.hosts.reimage for host an-web1001.eqiad.wmnet with OS bookworm [production]
07:19 <btullis@cumin1003> START - Cookbook sre.hosts.reimage for host an-master1003.eqiad.wmnet with OS bookworm [production]
07:18 <btullis@cumin1003> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts an-master1003.eqiad.wmnet [production]
07:18 <btullis@cumin1003> END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) for host an-master1003.eqiad.wmnet [production]
07:17 <btullis@cumin1003> START - Cookbook sre.hosts.reimage for host an-presto1011.eqiad.wmnet with OS bookworm [production]
07:16 <btullis@cumin1003> START - Cookbook sre.hosts.reimage for host an-presto1009.eqiad.wmnet with OS bookworm [production]
07:16 <jmm@cumin2003> START - Cookbook sre.hosts.reboot-single for host ganeti3008.esams.wmnet [production]
07:15 <btullis@cumin1003> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-presto1009.eqiad.wmnet with OS bookworm [production]
07:14 <btullis@cumin1003> START - Cookbook sre.hosts.reboot-single for host an-master1003.eqiad.wmnet [production]
07:13 <btullis@cumin1003> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts an-master1003.eqiad.wmnet [production]
07:13 <btullis@cumin1003> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts an-master1003.eqiad.wmnet [production]
07:12 <btullis@cumin1003> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts an-master1003.eqiad.wmnet [production]
07:11 <jmm@cumin2003> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti3008.esams.wmnet [production]
07:07 <arnaudb@dns1006> END - running authdns-update [production]
07:07 <jmm@cumin2003> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5007.eqsin.wmnet [production]
07:07 <jmm@cumin2003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5007.eqsin.wmnet [production]
07:05 <arnaudb@dns1006> START - running authdns-update [production]
06:58 <jmm@cumin2003> START - Cookbook sre.hosts.reboot-single for host ganeti5007.eqsin.wmnet [production]
06:54 <jmm@cumin2003> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5007.eqsin.wmnet [production]
06:38 <moritzm> failover ganeti master in eqsin to ganeti5004 [production]
06:36 <jmm@cumin2003> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet [production]
06:36 <jmm@cumin2003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet [production]
06:28 <jmm@cumin2003> START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet [production]
06:23 <jmm@cumin2003> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet [production]
06:20 <jmm@cumin2003> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5005.eqsin.wmnet [production]
06:20 <jmm@cumin2003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5005.eqsin.wmnet [production]
06:12 <jmm@cumin2003> START - Cookbook sre.hosts.reboot-single for host ganeti5005.eqsin.wmnet [production]
06:06 <jmm@cumin2003> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5005.eqsin.wmnet [production]
06:03 <jmm@cumin2003> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5004.eqsin.wmnet [production]
06:03 <jmm@cumin2003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5004.eqsin.wmnet [production]
05:55 <jmm@cumin2003> START - Cookbook sre.hosts.reboot-single for host ganeti5004.eqsin.wmnet [production]
05:53 <jmm@cumin2003> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5004.eqsin.wmnet [production]
04:40 <ryankemper> T434494 reimaged `an-tool1008.eqiad.wmnet` to bookworm; yarn.wikimedia.org is back up [production]
04:16 <ryankemper@cumin2003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-tool1008.eqiad.wmnet with OS bookworm [production]
03:58 <ryankemper@cumin2003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-tool1008.eqiad.wmnet with reason: host reimage [production]
03:53 <ryankemper@cumin2003> START - Cookbook sre.hosts.downtime for 2:00:00 on an-tool1008.eqiad.wmnet with reason: host reimage [production]
03:41 <ryankemper@cumin2003> START - Cookbook sre.hosts.reimage for host an-tool1008.eqiad.wmnet with OS bookworm [production]