1-50 of 10000 results (4ms)
2026-08-28 ยง
18:33 <eevans@cumin1003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cassandra-dev2003.codfw.wmnet with OS bookworm [production]
18:13 <eevans@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage [production]
18:09 <eevans@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on cassandra-dev2003.codfw.wmnet with reason: host reimage [production]
17:53 <eevans@cumin1003> START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm [production]
17:52 <eevans@cumin1003> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [production]
17:50 <eevans@cumin1003> START - Cookbook sre.hosts.provision for host cassandra-dev2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [production]
17:50 <eevans@cumin1003> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cassandra-dev2003.codfw.wmnet with OS bookworm [production]
17:32 <eevans@cumin1003> START - Cookbook sre.hosts.reimage for host cassandra-dev2003.codfw.wmnet with OS bookworm [production]
16:38 <sukhe@cumin1003> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp5022.eqsin.wmnet with OS trixie [production]
16:03 <sukhe> sudo ipmitool -I lanplus -H "cp5022.mgmt.eqsin.wmnet" -U root -E chassis power cycle: trying to debug why it doesn't come up after reimage [production]
15:43 <elukey> powercycle pki1002 after CPU-related failures (host completely unresponsive) - T434268 [production]
15:40 <elukey@puppetserver1001> conftool action : set/pooled=false; selector: dnsdisc=pki,name=eqiad [production]
15:16 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch [admin]
15:16 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [admin]
15:15 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [admin]
15:13 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [admin]
15:11 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/346 [admin]
15:10 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/346 [admin]
15:10 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [admin]
15:09 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [admin]
15:03 <bking@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
15:03 <bking@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
14:59 <sukhe@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage [production]
14:54 <sukhe@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on cp5022.eqsin.wmnet with reason: host reimage [production]
14:22 <sukhe@cumin1003> END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host cp5022 [production]
14:22 <sukhe@cumin1003> START - Cookbook sre.hosts.move-vlan for host cp5022 [production]
14:22 <sukhe@cumin1003> START - Cookbook sre.hosts.reimage for host cp5022.eqsin.wmnet with OS trixie [production]
14:03 <bking@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
14:01 <bking@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
13:19 <filippo@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1075' (T435923) [admin]
13:13 <filippo@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1075' (T435923) [admin]
13:13 <filippo@cloudcumin1001> END (ERROR) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=97) on host 'cloudvirt1075' (T435921) [admin]
13:12 <filippo@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1075' (T435921) [admin]
13:11 <filippo@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1074' (T435921) [admin]
13:01 <filippo@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1074' (T435921) [admin]
12:57 <filippo@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1073' (T435921) [admin]
12:47 <filippo@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1073' (T435921) [admin]
12:44 <filippo@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1072' (T435921) [admin]
12:31 <filippo@cloudcumin1001> START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1072' (T435921) [admin]
12:31 <jmm@cumin2003> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cumin1004.eqiad.wmnet [production]
12:31 <jmm@cumin2003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cumin1004.eqiad.wmnet with OS trixie [production]
12:14 <jmm@cumin2003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage [production]
12:07 <jmm@cumin2003> START - Cookbook sre.hosts.downtime for 2:00:00 on cumin1004.eqiad.wmnet with reason: host reimage [production]
11:56 <jmm@cumin2003> START - Cookbook sre.hosts.reimage for host cumin1004.eqiad.wmnet with OS trixie [production]
11:43 <moritzm> add new LDAP group cn=airflow-experiment-platform-ops,ou=groups,dc=wikimedia,dc=org T416709 [production]
11:43 <jmm@cumin2003> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" [production]
11:43 <jmm@cumin2003> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cumin1004.eqiad.wmnet - jmm@cumin2003" [production]
11:42 <jmm@cumin2003> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cumin1004.eqiad.wmnet on all recursors [production]
11:41 <jmm@cumin2003> START - Cookbook sre.dns.wipe-cache cumin1004.eqiad.wmnet on all recursors [production]
11:40 <jmm@cumin2003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]