|
2021-02-08
ยง
|
| 23:44 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1387.eqiad.wmnet with reason: REIMAGE |
[production] |
| 23:29 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1388.eqiad.wmnet |
[production] |
| 23:17 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1303.eqiad.wmnet |
[production] |
| 23:16 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2245.codfw.wmnet |
[production] |
| 23:14 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1397.eqiad.wmnet |
[production] |
| 22:51 |
<legoktm@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1274.eqiad.wmnet |
[production] |
| 22:51 |
<legoktm@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1273.eqiad.wmnet |
[production] |
| 22:51 |
<legoktm@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1272.eqiad.wmnet |
[production] |
| 22:51 |
<legoktm@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1271.eqiad.wmnet |
[production] |
| 22:46 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw2245.codfw.wmnet |
[production] |
| 22:46 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1388.eqiad.wmnet |
[production] |
| 22:45 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1303.eqiad.wmnet |
[production] |
| 22:45 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1397.eqiad.wmnet |
[production] |
| 22:29 |
<legoktm@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1274.eqiad.wmnet |
[production] |
| 22:29 |
<legoktm@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1273.eqiad.wmnet |
[production] |
| 22:29 |
<legoktm@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1272.eqiad.wmnet |
[production] |
| 22:29 |
<legoktm@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1271.eqiad.wmnet |
[production] |
| 22:29 |
<elukey> |
the previous entry was related to the Hadoop backup cluster |
[analytics] |
| 22:29 |
<elukey> |
hdfs master failover an-worker1118 -> an-worker1124, created dedicated partition for /var/lib/hadoop/name (root partition filled up), restarted namenode on 1118 (now recovering edit logs) |
[analytics] |
| 21:55 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1303.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:53 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1303.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:53 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2245.codfw.wmnet with reason: REIMAGE |
[production] |
| 21:51 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw2245.codfw.wmnet with reason: REIMAGE |
[production] |
| 21:40 |
<brennen> |
enabled administrative approval for new account signups on gitlab-test |
[releng] |
| 21:40 |
<brennen> |
upgraded gitlab instance on gitlab-test from 13.3.3-ce.0 to 13.8.3-ce.0 |
[releng] |
| 21:36 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1397.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:34 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1397.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:30 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1388.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:29 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mw1273.eqiad.wmnet with reason: reimaging |
[production] |
| 21:29 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on mw1273.eqiad.wmnet with reason: reimaging |
[production] |
| 21:28 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mw1271.eqiad.wmnet with reason: reimaging |
[production] |
| 21:28 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on mw1271.eqiad.wmnet with reason: reimaging |
[production] |
| 21:28 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1274.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:26 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1388.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:25 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1272.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:25 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1274.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:24 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1304.eqiad.wmnet |
[production] |
| 21:24 |
<legoktm@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw1271.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:24 |
<legoktm@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on mw1273.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:24 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1271.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:24 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1273.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:23 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1272.eqiad.wmnet with reason: REIMAGE |
[production] |
| 21:20 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1305.eqiad.wmnet |
[production] |
| 21:18 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1304.eqiad.wmnet |
[production] |
| 21:18 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1305.eqiad.wmnet |
[production] |
| 21:10 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1389.eqiad.wmnet |
[production] |
| 21:09 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1389.eqiad.wmnet |
[production] |
| 21:08 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1390.eqiad.wmnet |
[production] |
| 21:07 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1390.eqiad.wmnet |
[production] |
| 20:31 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1304.eqiad.wmnet with reason: REIMAGE |
[production] |