2021-02-08
ยง
|
23:44 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1387.eqiad.wmnet with reason: REIMAGE |
[production] |
23:29 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1388.eqiad.wmnet |
[production] |
23:17 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1303.eqiad.wmnet |
[production] |
23:16 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2245.codfw.wmnet |
[production] |
23:14 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1397.eqiad.wmnet |
[production] |
22:51 |
<legoktm@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1274.eqiad.wmnet |
[production] |
22:51 |
<legoktm@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1273.eqiad.wmnet |
[production] |
22:51 |
<legoktm@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1272.eqiad.wmnet |
[production] |
22:51 |
<legoktm@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1271.eqiad.wmnet |
[production] |
22:46 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2245.codfw.wmnet |
[production] |
22:46 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1388.eqiad.wmnet |
[production] |
22:45 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1303.eqiad.wmnet |
[production] |
22:45 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1397.eqiad.wmnet |
[production] |
22:29 |
<legoktm@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1274.eqiad.wmnet |
[production] |
22:29 |
<legoktm@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1273.eqiad.wmnet |
[production] |
22:29 |
<legoktm@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1272.eqiad.wmnet |
[production] |
22:29 |
<legoktm@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1271.eqiad.wmnet |
[production] |
22:29 |
<elukey> |
the previous entry was related to the Hadoop backup cluster |
[analytics] |
22:29 |
<elukey> |
hdfs master failover an-worker1118 -> an-worker1124, created dedicated partition for /var/lib/hadoop/name (root partition filled up), restarted namenode on 1118 (now recovering edit logs) |
[analytics] |
21:55 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1303.eqiad.wmnet with reason: REIMAGE |
[production] |
21:53 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1303.eqiad.wmnet with reason: REIMAGE |
[production] |
21:53 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2245.codfw.wmnet with reason: REIMAGE |
[production] |
21:51 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2245.codfw.wmnet with reason: REIMAGE |
[production] |
21:40 |
<brennen> |
enabled administrative approval for new account signups on gitlab-test |
[releng] |
21:40 |
<brennen> |
upgraded gitlab instance on gitlab-test from 13.3.3-ce.0 to 13.8.3-ce.0 |
[releng] |
21:36 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1397.eqiad.wmnet with reason: REIMAGE |
[production] |
21:34 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1397.eqiad.wmnet with reason: REIMAGE |
[production] |
21:30 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1388.eqiad.wmnet with reason: REIMAGE |
[production] |
21:29 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mw1273.eqiad.wmnet with reason: reimaging |
[production] |
21:29 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on mw1273.eqiad.wmnet with reason: reimaging |
[production] |
21:28 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mw1271.eqiad.wmnet with reason: reimaging |
[production] |
21:28 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on mw1271.eqiad.wmnet with reason: reimaging |
[production] |
21:28 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1274.eqiad.wmnet with reason: REIMAGE |
[production] |
21:26 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1388.eqiad.wmnet with reason: REIMAGE |
[production] |
21:25 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1272.eqiad.wmnet with reason: REIMAGE |
[production] |
21:25 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1274.eqiad.wmnet with reason: REIMAGE |
[production] |
21:24 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1304.eqiad.wmnet |
[production] |
21:24 |
<legoktm@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw1271.eqiad.wmnet with reason: REIMAGE |
[production] |
21:24 |
<legoktm@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw1273.eqiad.wmnet with reason: REIMAGE |
[production] |
21:24 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1271.eqiad.wmnet with reason: REIMAGE |
[production] |
21:24 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1273.eqiad.wmnet with reason: REIMAGE |
[production] |
21:23 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1272.eqiad.wmnet with reason: REIMAGE |
[production] |
21:20 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1305.eqiad.wmnet |
[production] |
21:18 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1304.eqiad.wmnet |
[production] |
21:18 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1305.eqiad.wmnet |
[production] |
21:10 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1389.eqiad.wmnet |
[production] |
21:09 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1389.eqiad.wmnet |
[production] |
21:08 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1390.eqiad.wmnet |
[production] |
21:07 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1390.eqiad.wmnet |
[production] |
20:31 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1304.eqiad.wmnet with reason: REIMAGE |
[production] |