|
2021-02-10
ยง
|
| 19:47 |
<cmjohnson@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 19:46 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1293.eqiad.wmnet with reason: REIMAGE |
[production] |
| 19:25 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1378.eqiad.wmnet with reason: REIMAGE |
[production] |
| 19:23 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1378.eqiad.wmnet with reason: REIMAGE |
[production] |
| 19:20 |
<thcipriani@deploy1001> |
Synchronized wmf-config/ProductionServices.php: [[gerrit:661732|Remove a couple of useless DNS lookups from mediawiki-config]] T231025 (duration: 01m 10s) |
[production] |
| 19:19 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1294.eqiad.wmnet |
[production] |
| 19:17 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1294.eqiad.wmnet |
[production] |
| 19:17 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1379.eqiad.wmnet |
[production] |
| 19:16 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1370.eqiad.wmnet with reason: REIMAGE |
[production] |
| 19:15 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1379.eqiad.wmnet |
[production] |
| 19:14 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1370.eqiad.wmnet with reason: REIMAGE |
[production] |
| 19:12 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1294.eqiad.wmnet |
[production] |
| 19:04 |
<mutante> |
mw1379 - racadm racreset - host did not come back from reboot and DRAC says it can't powercycle it.. while it also ALREADY ON |
[production] |
| 19:00 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1324.eqiad.wmnet with reason: REIMAGE |
[production] |
| 19:00 |
<dzahn@cumin1001> |
conftool action : set/pooled=inactive; selector: name=mw1379.eqiad.wmnet |
[production] |
| 18:58 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1323.eqiad.wmnet with reason: REIMAGE |
[production] |
| 18:58 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1324.eqiad.wmnet with reason: REIMAGE |
[production] |
| 18:57 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1371.eqiad.wmnet |
[production] |
| 18:56 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1322.eqiad.wmnet with reason: REIMAGE |
[production] |
| 18:56 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1323.eqiad.wmnet with reason: REIMAGE |
[production] |
| 18:54 |
<legoktm@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1321.eqiad.wmnet with reason: REIMAGE |
[production] |
| 18:54 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1322.eqiad.wmnet with reason: REIMAGE |
[production] |
| 18:52 |
<legoktm@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1321.eqiad.wmnet with reason: REIMAGE |
[production] |
| 18:36 |
<andrew@deploy1001> |
Finished deploy [horizon/deploy@02cb8a4]: security group dashboard policy updates, now after doing a submodule update! (duration: 03m 31s) |
[production] |
| 18:34 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1379.eqiad.wmnet |
[production] |
| 18:33 |
<andrew@deploy1001> |
Started deploy [horizon/deploy@02cb8a4]: security group dashboard policy updates, now after doing a submodule update! |
[production] |
| 18:32 |
<andrew@deploy1001> |
Finished deploy [horizon/deploy@4f5a5a7]: security group dashboard policy updates (duration: 00m 07s) |
[production] |
| 18:32 |
<andrew@deploy1001> |
Started deploy [horizon/deploy@4f5a5a7]: security group dashboard policy updates |
[production] |
| 18:32 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1371.eqiad.wmnet |
[production] |
| 18:20 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1294.eqiad.wmnet with reason: REIMAGE |
[production] |
| 18:18 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1294.eqiad.wmnet with reason: REIMAGE |
[production] |
| 18:14 |
<jiji@cumin1001> |
END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host thumbor1001.eqiad.wmnet |
[production] |
| 17:51 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw1295.eqiad.wmnet |
[production] |
| 17:43 |
<dzahn@cumin1001> |
conftool action : set/pooled=no; selector: name=mw1295.eqiad.wmnet |
[production] |
| 17:19 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host thumbor1001.eqiad.wmnet |
[production] |
| 17:18 |
<shdubsh> |
restart pybal on low-traffic lvs1015 |
[production] |
| 17:13 |
<shdubsh> |
restart pybal on backup lvs1016 |
[production] |
| 17:13 |
<andrew@deploy1001> |
Finished deploy [horizon/deploy@4f5a5a7]: puppet dashboard policy updates (duration: 03m 53s) |
[production] |
| 17:09 |
<andrew@deploy1001> |
Started deploy [horizon/deploy@4f5a5a7]: puppet dashboard policy updates |
[production] |
| 16:57 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1295.eqiad.wmnet with reason: REIMAGE |
[production] |
| 16:53 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1295.eqiad.wmnet with reason: REIMAGE |
[production] |
| 16:44 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1379.eqiad.wmnet with reason: REIMAGE |
[production] |
| 16:42 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1379.eqiad.wmnet with reason: REIMAGE |
[production] |
| 16:42 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1371.eqiad.wmnet with reason: REIMAGE |
[production] |
| 16:40 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mw1371.eqiad.wmnet with reason: REIMAGE |
[production] |
| 16:20 |
<moritzm> |
installing unzip security updates |
[production] |
| 16:12 |
<moritzm> |
installing atftp security updates |
[production] |
| 16:01 |
<hnowlan@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 20:00:00 on maps1005.eqiad.wmnet with reason: Resyncing database, still |
[production] |
| 16:01 |
<hnowlan@cumin1001> |
START - Cookbook sre.hosts.downtime for 20:00:00 on maps1005.eqiad.wmnet with reason: Resyncing database, still |
[production] |
| 15:26 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Do not produce canary events for rdf-streaming-updater streams - T269619 (duration: 01m 13s) |
[production] |