4351-4400 of 10000 results (96ms)
2023-08-14 §
11:09 <stevemunene@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-airflow1007.eqiad.wmnet with OS buster [production]
10:54 <stevemunene@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-airflow1007.eqiad.wmnet with reason: host reimage [production]
10:51 <stevemunene@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-airflow1007.eqiad.wmnet with reason: host reimage [production]
10:39 <stevemunene@cumin1001> START - Cookbook sre.hosts.reimage for host an-airflow1007.eqiad.wmnet with OS buster [production]
10:39 <stevemunene@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM an-airflow1007.eqiad.wmnet - stevemunene@cumin1001" [production]
10:39 <cmooney@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
10:39 <cmooney@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Rename mr1-esams-new to mr1-esams in dns. - cmooney@cumin1001" [production]
10:38 <stevemunene@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM an-airflow1007.eqiad.wmnet - stevemunene@cumin1001" [production]
10:38 <cmooney@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Rename mr1-esams-new to mr1-esams in dns. - cmooney@cumin1001" [production]
10:38 <stevemunene@cumin1001> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) an-airflow1007.eqiad.wmnet on all recursors [production]
10:38 <stevemunene@cumin1001> START - Cookbook sre.dns.wipe-cache an-airflow1007.eqiad.wmnet on all recursors [production]
10:36 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
10:34 <cmooney@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
10:34 <cmooney@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Rename mr1-esams-new to mr1-esams in dns. - cmooney@cumin1001" [production]
10:33 <cmooney@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Rename mr1-esams-new to mr1-esams in dns. - cmooney@cumin1001" [production]
10:30 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
10:26 <stevemunene@cumin1001> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
10:25 <cmooney@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
10:25 <cmooney@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Rename mr1-esams-new to mr1-esams in dns. - cmooney@cumin1001" [production]
10:24 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1008.eqiad.wmnet [production]
10:24 <cmooney@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Rename mr1-esams-new to mr1-esams in dns. - cmooney@cumin1001" [production]
10:20 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1009.eqiad.wmnet [production]
10:13 <stevemunene@cumin1001> START - Cookbook sre.dns.netbox [production]
10:13 <stevemunene@cumin1001> START - Cookbook sre.ganeti.makevm for new host an-airflow1007.eqiad.wmnet [production]
10:12 <btullis@cumin1001> START - Cookbook sre.hosts.reboot-single for host stat1009.eqiad.wmnet [production]
09:59 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1124.eqiad.wmnet [production]
09:49 <btullis@cumin1001> START - Cookbook sre.hosts.reboot-single for host stat1008.eqiad.wmnet [production]
09:48 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1007.eqiad.wmnet [production]
09:41 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
09:41 <cmooney@cumin1001> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
09:39 <btullis@cumin1001> START - Cookbook sre.hosts.reboot-single for host stat1007.eqiad.wmnet [production]
09:37 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1006.eqiad.wmnet [production]
09:32 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
09:32 <cmooney@cumin1001> END (ERROR) - Cookbook sre.dns.netbox (exit_code=97) [production]
09:32 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
09:28 <btullis@cumin1001> START - Cookbook sre.hosts.reboot-single for host stat1006.eqiad.wmnet [production]
09:27 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1005.eqiad.wmnet [production]
09:26 <btullis@cumin1001> START - Cookbook sre.hosts.reboot-single for host an-worker1124.eqiad.wmnet [production]
09:16 <btullis@cumin1001> START - Cookbook sre.hosts.reboot-single for host stat1005.eqiad.wmnet [production]
09:13 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1004.eqiad.wmnet [production]
09:11 <cmooney@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
09:11 <cmooney@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Rename mr1-esams dns to mr1-eams-old. - cmooney@cumin1001" [production]
09:10 <cmooney@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Rename mr1-esams dns to mr1-eams-old. - cmooney@cumin1001" [production]
09:08 <cmooney@cumin1001> START - Cookbook sre.dns.netbox [production]
09:02 <btullis@cumin1001> START - Cookbook sre.hosts.reboot-single for host stat1004.eqiad.wmnet [production]
2023-08-13 §
16:07 <topranks> powering down cr3-esams [production]
16:05 <topranks> powering down cr2-esams [production]
15:54 <topranks> Disabling esams peering at AMS-IX prior to removing router [production]
15:45 <topranks> Disable transport cct cr2-esams to cr2-eqiad prior to disconnect T329219 [production]
15:26 <topranks> disable transit and peering links on cr2-esams & cr3-esams before decom T329219 [production]