101-150 of 10000 results (154ms)
2026-08-20 ยง
13:13 <mszwarc@deploy1003> Finished scap sync-world: Backport for [[gerrit:1327511|UIC: Fix page:page instead of page:other in instrumentation]] (duration: 07m 00s) [production]
13:13 <klausman@cumin1003> START - Cookbook sre.ganeti.reboot-vm for VM ml-serve-ctrl2001.codfw.wmnet [production]
13:12 <klausman@cumin1003> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-serve-ctrl2002.codfw.wmnet [production]
13:09 <mszwarc@deploy1003> mszwarc: Continuing with deployment [production]
13:08 <mszwarc@deploy1003> mszwarc: Backport for [[gerrit:1327511|UIC: Fix page:page instead of page:other in instrumentation]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
13:08 <klausman@cumin1003> START - Cookbook sre.ganeti.reboot-vm for VM ml-serve-ctrl2002.codfw.wmnet [production]
13:07 <klausman@cumin1003> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-ctrl2002.codfw.wmnet [production]
13:06 <mszwarc@deploy1003> Started scap sync-world: Backport for [[gerrit:1327511|UIC: Fix page:page instead of page:other in instrumentation]] [production]
13:04 <jmm@dns1004> END - running authdns-update [production]
13:03 <klausman@cumin1003> START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-ctrl2002.codfw.wmnet [production]
13:03 <klausman@cumin1003> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-ctrl2001.codfw.wmnet [production]
13:02 <jmm@dns1004> START - running authdns-update [production]
13:01 <cmooney@dns3003> END - running authdns-update [production]
13:00 <klausman@cumin1003> START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-ctrl2001.codfw.wmnet [production]
12:59 <klausman@cumin1003> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-etcd2001.codfw.wmnet [production]
12:59 <cmooney@dns3003> START - running authdns-update [production]
12:57 <klausman@cumin1003> START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-etcd2001.codfw.wmnet [production]
12:56 <klausman@cumin1003> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-etcd2002.codfw.wmnet [production]
12:55 <cmooney@cumin1003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
12:55 <cmooney@cumin1003> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on drmrs<->eqiad GTT vpls - cmooney@cumin1003" [production]
12:54 <cmooney@cumin1003> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on drmrs<->eqiad GTT vpls - cmooney@cumin1003" [production]
12:54 <klausman@cumin1003> START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-etcd2002.codfw.wmnet [production]
12:54 <klausman@cumin1003> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-staging-etcd2003.codfw.wmnet [production]
12:50 <klausman@cumin1003> START - Cookbook sre.ganeti.reboot-vm for VM ml-staging-etcd2003.codfw.wmnet [production]
12:49 <cmooney@cumin1003> START - Cookbook sre.dns.netbox [production]
12:48 <klausman@cumin1003> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd2001.codfw.wmnet [production]
12:46 <klausman@cumin1003> START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd2001.codfw.wmnet [production]
12:46 <klausman@cumin1003> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd2002.codfw.wmnet [production]
12:43 <klausman@cumin1003> START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd2002.codfw.wmnet [production]
12:43 <klausman@cumin1003> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd2003.codfw.wmnet [production]
12:42 <fceratto@cumin1003> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=1) for new host db1902.eqiad.wmnet [production]
12:42 <fceratto@cumin1003> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=1) for host db1902.eqiad.wmnet with OS trixie [production]
12:41 <klausman@cumin1003> START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd2003.codfw.wmnet [production]
12:40 <klausman@cumin1003> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd1003.eqiad.wmnet [production]
12:38 <klausman@cumin1003> START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd1003.eqiad.wmnet [production]
12:37 <klausman@cumin1003> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd1002.eqiad.wmnet [production]
12:35 <klausman@cumin1003> START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd1002.eqiad.wmnet [production]
12:35 <klausman@cumin1003> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM ml-etcd1001.eqiad.wmnet [production]
12:34 <cmooney@dns3003> END - running authdns-update [production]
12:33 <klausman@cumin1003> START - Cookbook sre.ganeti.reboot-vm for VM ml-etcd1001.eqiad.wmnet [production]
12:32 <btullis@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on stat1009.eqiad.wmnet with reason: host reimage [production]
12:31 <cmooney@dns3003> START - running authdns-update [production]
12:31 <cmooney@cumin1003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
12:31 <cmooney@cumin1003> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on drmrs<->eqiad cct - cmooney@cumin1003" [production]
12:28 <cmooney@cumin1003> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on drmrs<->eqiad cct - cmooney@cumin1003" [production]
12:28 <fceratto@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1902.eqiad.wmnet with reason: host reimage [production]
12:25 <btullis@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on stat1009.eqiad.wmnet with reason: host reimage [production]
12:25 <cmooney@cumin1003> START - Cookbook sre.dns.netbox [production]
12:24 <btullis@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on stat1010.eqiad.wmnet with reason: host reimage [production]
12:22 <fceratto@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on db1902.eqiad.wmnet with reason: host reimage [production]