1-50 of 10000 results (64ms)
2022-08-15 ยง
18:33 <ryankemper@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic1081.eqiad.wmnet with reason: host reimage [production]
18:31 <pt1979@cumin2002> END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts ms-be2067.codfw.wmnet [production]
18:29 <ryankemper@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1081.eqiad.wmnet with reason: host reimage [production]
18:24 <pt1979@cumin2002> END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) for host ms-be2067.codfw.wmnet [production]
18:16 <ryankemper@cumin1001> START - Cookbook sre.hosts.reimage for host elastic1081.eqiad.wmnet with OS bullseye [production]
18:07 <herron> thanos compact process was hung, forced thanos-compact restart on thanos-fe2001 [production]
17:48 <ryankemper@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic1052.eqiad.wmnet with OS bullseye [production]
17:32 <ryankemper@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic1052.eqiad.wmnet with reason: host reimage [production]
17:29 <pt1979@cumin2002> START - Cookbook sre.hosts.reboot-single for host ms-be2067.codfw.wmnet [production]
17:28 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ms-be2067.codfw.wmnet [production]
17:28 <ryankemper@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1052.eqiad.wmnet with reason: host reimage [production]
17:28 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ms-be2067.codfw.wmnet [production]
17:28 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ms-be2067.codfw.wmnet [production]
17:24 <ebernhardson@deploy1002> Finished deploy [wikimedia/discovery/analytics@d4137b5]: increase subgraph query SLA and remove same from drop_old_data (duration: 02m 17s) [production]
17:22 <ebernhardson@deploy1002> Started deploy [wikimedia/discovery/analytics@d4137b5]: increase subgraph query SLA and remove same from drop_old_data [production]
17:17 <ryankemper@cumin1001> START - Cookbook sre.hosts.reimage for host elastic1052.eqiad.wmnet with OS bullseye [production]
17:00 <ryankemper@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic1082.eqiad.wmnet with OS bullseye [production]
16:39 <ryankemper@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic1082.eqiad.wmnet with reason: host reimage [production]
16:35 <ryankemper@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1082.eqiad.wmnet with reason: host reimage [production]
16:32 <damilare> payments-wiki upgraded from 0894d75a to 41709763 [production]
16:27 <btullis@puppetmaster1001> conftool action : set/pooled=no; selector: cluster=wikireplicas-b,name=dbproxy1019.eqiad.wmnet [production]
16:25 <btullis@puppetmaster1001> conftool action : set/pooled=yes; selector: cluster=wikireplicas-b,name=dbproxy1018.eqiad.wmnet [production]
16:23 <ryankemper@cumin1001> START - Cookbook sre.hosts.reimage for host elastic1082.eqiad.wmnet with OS bullseye [production]
16:17 <dancy@deploy1002> Installation of scap version "4.13.0" completed for 553 hosts [production]
16:17 <dancy@deploy1002> Installing scap version "4.13.0" for 553 hosts [production]
16:14 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:09 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
16:01 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:56 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
15:43 <pt1979@cumin2002> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts logstash2003.codfw.wmnet [production]
15:35 <pt1979@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts logstash2003.codfw.wmnet [production]
15:32 <mvernon@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ms-be2067.codfw.wmnet with reason: disk fault investigation [production]
15:32 <mvernon@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on ms-be2067.codfw.wmnet with reason: disk fault investigation [production]
15:31 <mvernon@cumin1001> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ms-be2032.codfw.wmnet [production]
15:31 <mvernon@cumin1001> START - Cookbook sre.hosts.remove-downtime for ms-be2032.codfw.wmnet [production]
15:31 <mvernon@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ms-be2032.codfw.wmnet with reason: RAID battery failure [production]
15:31 <mvernon@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on ms-be2032.codfw.wmnet with reason: RAID battery failure [production]
15:31 <mvernon@cumin1001> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for ms-be2032.codfw.wmnet [production]
15:31 <mvernon@cumin1001> START - Cookbook sre.hosts.remove-downtime for ms-be2032.codfw.wmnet [production]
15:01 <ryankemper@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic1068.eqiad.wmnet with OS bullseye [production]
14:39 <ryankemper@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic1068.eqiad.wmnet with reason: host reimage [production]
14:36 <ryankemper@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1068.eqiad.wmnet with reason: host reimage [production]
14:26 <hnowlan@deploy1002> Finished deploy [restbase/deploy@a571f9a]: Add blwiki T310874 (duration: 15m 42s) [production]
14:23 <ryankemper@cumin1001> START - Cookbook sre.hosts.reimage for host elastic1068.eqiad.wmnet with OS bullseye [production]
14:10 <mvernon@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ms-be2032.codfw.wmnet with reason: RAID battery failure [production]
14:10 <hnowlan@deploy1002> Started deploy [restbase/deploy@a571f9a]: Add blwiki T310874 [production]
14:10 <mvernon@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on ms-be2032.codfw.wmnet with reason: RAID battery failure [production]
14:05 <ryankemper@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic1070.eqiad.wmnet with OS bullseye [production]
13:49 <ryankemper@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic1070.eqiad.wmnet with reason: host reimage [production]
13:46 <ryankemper@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on elastic1070.eqiad.wmnet with reason: host reimage [production]