6601-6650 of 10000 results (22ms)
2021-03-08 §
10:15 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1083.eqiad.wmnet with reason: REIMAGE [production]
09:19 <elukey> drain + reimage an-worker108[3,4] to Buster [production]
08:56 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1082.eqiad.wmnet with reason: REIMAGE [production]
08:54 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1081.eqiad.wmnet with reason: REIMAGE [production]
08:53 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1082.eqiad.wmnet with reason: REIMAGE [production]
08:52 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1081.eqiad.wmnet with reason: REIMAGE [production]
08:20 <elukey> drain + reimage an-worker108[1,2] to Buster [production]
07:49 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on analytics1075.eqiad.wmnet with reason: REIMAGE [production]
07:47 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on analytics1075.eqiad.wmnet with reason: REIMAGE [production]
07:46 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on analytics1074.eqiad.wmnet with reason: REIMAGE [production]
07:44 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on analytics1074.eqiad.wmnet with reason: REIMAGE [production]
07:23 <elukey> drain + reimage analytics107[4,5] to Buster [production]
2021-03-07 §
08:01 <elukey> "megacli -LDSetProp -ForcedWB -Immediate -Lall -aAll" on analytics1066 - BBU looks fine, but the raid controller was using WriteThrough [production]
2021-03-05 §
16:11 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1086.eqiad.wmnet with reason: REIMAGE [production]
16:09 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1086.eqiad.wmnet with reason: REIMAGE [production]
15:39 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on analytics1073.eqiad.wmnet with reason: REIMAGE [production]
15:37 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on analytics1073.eqiad.wmnet with reason: REIMAGE [production]
15:07 <elukey> drain + reimage analytics1073 and an-worker1086 to Debian Buster [production]
13:59 <elukey@cumin1001> END (FAIL) - Cookbook sre.hadoop.roll-restart-masters (exit_code=99) [production]
13:38 <elukey@cumin1001> START - Cookbook sre.hadoop.roll-restart-masters [production]
09:28 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1078.eqiad.wmnet with reason: REIMAGE [production]
09:26 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1078.eqiad.wmnet with reason: REIMAGE [production]
09:26 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1079.eqiad.wmnet with reason: REIMAGE [production]
09:24 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1079.eqiad.wmnet with reason: REIMAGE [production]
08:32 <elukey> drain + reimage an-worker107[8,9] to Debian Buster [production]
08:01 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on analytics1071.eqiad.wmnet with reason: REIMAGE [production]
07:59 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on analytics1070.eqiad.wmnet with reason: REIMAGE [production]
07:59 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on analytics1071.eqiad.wmnet with reason: REIMAGE [production]
07:57 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on analytics1070.eqiad.wmnet with reason: REIMAGE [production]
07:33 <elukey> drain + reimage analytics107[0-1] to debian buster [production]
2021-03-04 §
16:56 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on analytics1069.eqiad.wmnet with reason: REIMAGE [production]
16:54 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on analytics1068.eqiad.wmnet with reason: REIMAGE [production]
16:54 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on analytics1069.eqiad.wmnet with reason: REIMAGE [production]
16:52 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on analytics1068.eqiad.wmnet with reason: REIMAGE [production]
15:28 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on analytics1067.eqiad.wmnet with reason: REIMAGE [production]
15:26 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on analytics1066.eqiad.wmnet with reason: REIMAGE [production]
15:26 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on analytics1067.eqiad.wmnet with reason: REIMAGE [production]
15:24 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on analytics1066.eqiad.wmnet with reason: REIMAGE [production]
15:12 <elukey> drain + reimage analytics106[6,7] to Debian Buster [production]
14:40 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on analytics1065.eqiad.wmnet with reason: REIMAGE [production]
14:38 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on analytics1065.eqiad.wmnet with reason: REIMAGE [production]
13:55 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on analytics1064.eqiad.wmnet with reason: REIMAGE [production]
13:53 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on analytics1063.eqiad.wmnet with reason: REIMAGE [production]
13:52 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on analytics1064.eqiad.wmnet with reason: REIMAGE [production]
13:50 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on analytics1063.eqiad.wmnet with reason: REIMAGE [production]
13:32 <elukey> drain + reimage analytics10[63,64] to Debian Buster [production]
13:10 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on analytics1062.eqiad.wmnet with reason: REIMAGE [production]
13:08 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on analytics1061.eqiad.wmnet with reason: REIMAGE [production]
13:07 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on analytics1062.eqiad.wmnet with reason: REIMAGE [production]
13:06 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on analytics1061.eqiad.wmnet with reason: REIMAGE [production]