201-250 of 10000 results (86ms)
2024-01-10 §
16:55 <kamila@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1378.eqiad.wmnet with reason: host reimage [production]
16:52 <kamila@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1378.eqiad.wmnet with reason: host reimage [production]
16:37 <kamila@cumin1002> START - Cookbook sre.hosts.reimage for host mw1378.eqiad.wmnet with OS bullseye [production]
16:36 <godog> upgrade prometheus on prometheus2006 - T354399 [production]
16:32 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply [production]
16:32 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply [production]
16:31 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply [production]
16:31 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply [production]
16:30 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply [production]
16:29 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply [production]
16:25 <kamila@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mw[1379-1383].eqiad.wmnet with reason: testing reboot [production]
16:25 <kamila@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on mw[1379-1383].eqiad.wmnet with reason: testing reboot [production]
16:22 <kamila@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1379.eqiad.wmnet with OS bullseye [production]
16:20 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply [production]
16:02 <kamila@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1379.eqiad.wmnet with reason: host reimage [production]
16:00 <kamila@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1383.eqiad.wmnet with OS bullseye [production]
15:59 <kamila@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1381.eqiad.wmnet with OS bullseye [production]
15:57 <kamila@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1382.eqiad.wmnet with OS bullseye [production]
15:57 <kamila@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1379.eqiad.wmnet with reason: host reimage [production]
15:41 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: logging::opensearch::data [production]
15:41 <kamila@cumin1002> START - Cookbook sre.hosts.reimage for host mw1379.eqiad.wmnet with OS bullseye [production]
15:40 <kamila@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1383.eqiad.wmnet with reason: host reimage [production]
15:37 <kamila@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1381.eqiad.wmnet with reason: host reimage [production]
15:37 <kamila@cumin1002> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw1382.eqiad.wmnet with reason: host reimage [production]
15:35 <kamila@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1383.eqiad.wmnet with reason: host reimage [production]
15:35 <kamila@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1382.eqiad.wmnet with reason: host reimage [production]
15:34 <kamila@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on mw1381.eqiad.wmnet with reason: host reimage [production]
15:24 <jmm@cumin2002> START - Cookbook sre.puppet.migrate-role for role: logging::opensearch::data [production]
15:24 <klausman@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-staging2001.codfw.wmnet [production]
15:22 <eevans@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on restbase2013.codfw.wmnet with reason: Decommissioning — T352469 [production]
15:21 <eevans@cumin1002> START - Cookbook sre.hosts.downtime for 30 days, 0:00:00 on restbase2013.codfw.wmnet with reason: Decommissioning — T352469 [production]
15:21 <kamila@cumin1002> START - Cookbook sre.hosts.reimage for host mw1383.eqiad.wmnet with OS bullseye [production]
15:20 <kamila@cumin1002> START - Cookbook sre.hosts.reimage for host mw1382.eqiad.wmnet with OS bullseye [production]
15:20 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: logging::opensearch::collector [production]
15:19 <kamila@cumin1002> START - Cookbook sre.hosts.reimage for host mw1381.eqiad.wmnet with OS bullseye [production]
15:17 <kamila@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1380.eqiad.wmnet with OS bullseye [production]
15:14 <klausman@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-staging2001.codfw.wmnet [production]
15:13 <btullis@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on an-master[1003-1004].eqiad.wmnet with reason: Bringing new nameservers into service [production]
15:13 <klausman@cumin1001> END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) for host ml-staging2001.codfw.wmnet [production]
15:12 <btullis@cumin1002> START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on an-master[1003-1004].eqiad.wmnet with reason: Bringing new nameservers into service [production]
15:07 <taavi@cumin1002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dbproxy[1018-1019].eqiad.wmnet [production]
15:06 <taavi@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:06 <taavi@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbproxy[1018-1019].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - taavi@cumin1002" [production]
15:04 <sukhe@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on lvs2014.codfw.wmnet with reason: T352758 [production]
15:04 <sukhe@cumin2002> START - Cookbook sre.hosts.downtime for 3:00:00 on lvs2014.codfw.wmnet with reason: T352758 [production]
15:03 <taavi@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dbproxy[1018-1019].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - taavi@cumin1002" [production]
15:01 <klausman@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-staging2001.codfw.wmnet [production]
15:01 <sukhe> disable puppet and stop pybal on lvs2014: T352758 [production]
15:00 <taavi@cumin1002> START - Cookbook sre.dns.netbox [production]
14:57 <kamila@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1380.eqiad.wmnet with reason: host reimage [production]