|
2019-10-24
ยง
|
| 20:24 |
<bblack@cumin1001> |
conftool action : set/pooled=no; selector: name=cp3033.esams.wmnet |
[production] |
| 20:23 |
<bblack@cumin1001> |
conftool action : set/pooled=yes; selector: name=cp3053.esams.wmnet |
[production] |
| 20:22 |
<bblack@cumin1001> |
conftool action : set/pooled=yes; selector: name=cp3054.esams.wmnet |
[production] |
| 20:04 |
<bblack> |
reboot cp3054 again for good measure |
[production] |
| 19:57 |
<bblack> |
cp3054 - trying racadm serveraction hardreset |
[production] |
| 19:32 |
<bblack> |
reboot dns3001 |
[production] |
| 19:31 |
<urandom> |
restbase cassandra rolling restart, codfw / rack 'b' -- T200803 |
[production] |
| 19:10 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 19:07 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 19:05 |
<urandom> |
restbase cassandra rolling restart, rack 'd' -- T200803 |
[production] |
| 19:05 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 19:05 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 19:05 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 19:03 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 19:01 |
<bblack@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 19:01 |
<bblack@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 19:01 |
<bblack@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 19:00 |
<bblack@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 19:00 |
<bblack@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 18:59 |
<bblack@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 18:57 |
<bblack@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 18:56 |
<bblack@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 18:56 |
<bblack@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 18:56 |
<bblack@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 18:55 |
<bblack@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 18:55 |
<bblack@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 18:55 |
<bblack@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 18:55 |
<bblack@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 18:55 |
<bblack@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 18:55 |
<bblack@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 18:55 |
<bblack@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 18:55 |
<bblack@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 18:55 |
<bblack@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 18:55 |
<Urbanecm> |
Morning SWAT done |
[production] |
| 18:55 |
<bblack@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 18:46 |
<urandom> |
restbase cassandra rolling restart, rack 'b' -- T200803 |
[production] |
| 18:44 |
<bblack@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 18:42 |
<bblack@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 18:42 |
<bblack@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 18:42 |
<bblack@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 18:31 |
<bblack> |
cr3-esams: add dns3001 to anycast4 neighbors |
[production] |
| 18:30 |
<bblack> |
cr2-esams: add dns3001 to anycast4 neighbors |
[production] |
| 18:29 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: 263fd0f: Enable Wikibase client access on commonswiki (T223792) (duration: 00m 52s) |
[production] |
| 18:25 |
<urandom> |
restbase cassandra rolling restart, rack 'a' -- T200803 |
[production] |
| 18:22 |
<robh> |
completing ps1-b6-eqiad setup, pdu will reboot twice, power output unaffected T227540 |
[production] |
| 18:20 |
<robh> |
ps1-a6-eqiad setup complete, icinga errors should clear up T227142 |
[production] |
| 18:15 |
<urbanecm@deploy1001> |
Synchronized wmf-config/: SWAT: 84c48df: rename service definition (T222851) (duration: 00m 53s) |
[production] |
| 18:06 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: b20d6de: Reference Previews: full beta deployment (T235083) (duration: 00m 52s) |
[production] |
| 18:03 |
<robh> |
setting ip info for ps1-a6-eqiad, it is rebooting. T227142 |
[production] |
| 17:41 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |