1-50 of 10000 results (98ms)
2026-05-19 ยง
11:53 <taavi@cumin1003> START - Cookbook sre.hosts.reboot-single for host cloudidp2001-dev.codfw.wmnet [production]
11:52 <marostegui@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1284.eqiad.wmnet with reason: host reimage [production]
11:50 <jynus@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 18 hosts with reason: restart [production]
11:49 <marostegui@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1277.eqiad.wmnet with reason: host reimage [production]
11:49 <marostegui@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on db1287.eqiad.wmnet with reason: host reimage [production]
11:49 <marostegui@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on db1288.eqiad.wmnet with reason: host reimage [production]
11:48 <marostegui@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on db1286.eqiad.wmnet with reason: host reimage [production]
11:48 <marostegui@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on db1284.eqiad.wmnet with reason: host reimage [production]
11:47 <marostegui@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on db1283.eqiad.wmnet with reason: host reimage [production]
11:47 <marostegui@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on db1282.eqiad.wmnet with reason: host reimage [production]
11:46 <tappof@cumin1003> START - Cookbook sre.hosts.reboot-single for host prometheus2005.codfw.wmnet [production]
11:46 <tappof@cumin1003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus5003.eqsin.wmnet [production]
11:45 <marostegui@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on db1281.eqiad.wmnet with reason: host reimage [production]
11:45 <marostegui@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on db1280.eqiad.wmnet with reason: host reimage [production]
11:44 <marostegui@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on db1279.eqiad.wmnet with reason: host reimage [production]
11:44 <marostegui@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on db1278.eqiad.wmnet with reason: host reimage [production]
11:44 <marostegui@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on db1277.eqiad.wmnet with reason: host reimage [production]
11:42 <mvernon@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-be2003.codfw.wmnet [production]
11:39 <tappof@cumin1003> START - Cookbook sre.hosts.reboot-single for host prometheus5003.eqsin.wmnet [production]
11:39 <tappof@cumin1003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus1008.eqiad.wmnet [production]
11:39 <jynus@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on ms-backup[1003-1004].eqiad.wmnet with reason: restart [production]
11:37 <moritzm> failover Ganeti cluster in eqsin to ganeti5004 [production]
11:37 <moritzm> failover Ganeti cluster in magru to ganeti7001 [production]
11:36 <marostegui@cumin1003> START - Cookbook sre.hosts.reimage for host db1288.eqiad.wmnet with OS trixie [production]
11:35 <marostegui@cumin1003> START - Cookbook sre.hosts.reimage for host db1287.eqiad.wmnet with OS trixie [production]
11:35 <marostegui@cumin1003> START - Cookbook sre.hosts.reimage for host db1286.eqiad.wmnet with OS trixie [production]
11:35 <marostegui@cumin1003> START - Cookbook sre.hosts.reimage for host db1284.eqiad.wmnet with OS trixie [production]
11:34 <mvernon@cumin2002> START - Cookbook sre.hosts.reboot-single for host moss-be2003.codfw.wmnet [production]
11:34 <marostegui@cumin1003> START - Cookbook sre.hosts.reimage for host db1283.eqiad.wmnet with OS trixie [production]
11:34 <marostegui@cumin1003> START - Cookbook sre.hosts.reimage for host db1282.eqiad.wmnet with OS trixie [production]
11:33 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti5006.eqsin.wmnet [production]
11:33 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti7003.magru.wmnet [production]
11:33 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti7003.magru.wmnet [production]
11:33 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti5006.eqsin.wmnet [production]
11:32 <marostegui@cumin1003> START - Cookbook sre.hosts.reimage for host db1281.eqiad.wmnet with OS trixie [production]
11:32 <marostegui@cumin1003> START - Cookbook sre.hosts.reimage for host db1280.eqiad.wmnet with OS trixie [production]
11:31 <marostegui@cumin1003> START - Cookbook sre.hosts.reimage for host db1279.eqiad.wmnet with OS trixie [production]
11:31 <marostegui@cumin1003> START - Cookbook sre.hosts.reimage for host db1278.eqiad.wmnet with OS trixie [production]
11:31 <marostegui@cumin1003> START - Cookbook sre.hosts.reimage for host db1277.eqiad.wmnet with OS trixie [production]
11:29 <tappof@cumin1003> START - Cookbook sre.hosts.reboot-single for host prometheus1008.eqiad.wmnet [production]
11:29 <tappof@cumin1003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus1006.eqiad.wmnet [production]
11:24 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti7003.magru.wmnet [production]
11:24 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti5006.eqsin.wmnet [production]
11:24 <marostegui@cumin1003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1276.eqiad.wmnet with OS trixie [production]
11:21 <marostegui@cumin1003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1269.eqiad.wmnet with OS trixie [production]
11:20 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti7003.magru.wmnet [production]
11:20 <jmm@cumin2002> START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti5006.eqsin.wmnet [production]
11:19 <tappof@cumin1003> START - Cookbook sre.hosts.reboot-single for host prometheus1006.eqiad.wmnet [production]
11:19 <tappof@cumin1003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host prometheus4003.ulsfo.wmnet [production]
11:18 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti7002.magru.wmnet [production]