|
2019-05-29
ยง
|
| 15:55 |
<jynus> |
upgrade and restart db2087 |
[production] |
| 15:11 |
<moritzm> |
draining ganeti2008 for eventual reboot to pick up MDS-enabled kernel |
[production] |
| 15:06 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 15:06 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 15:06 |
<anomie@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Setting actor migration to write-new/read-new on group 1 (T188327) (duration: 00m 57s) |
[production] |
| 14:54 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:54 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:54 |
<moritzm> |
draining ganeti2007 for eventual reboot to pick up MDS-enabled kernel |
[production] |
| 14:51 |
<XioNoX> |
`request chassis fpc online slot 0` on cr1-codfw - T224511 |
[production] |
| 14:48 |
<XioNoX> |
`request chassis fpc offline slot 0` on cr1-codfw - T224511 |
[production] |
| 14:47 |
<XioNoX> |
disable et- interfaces on cr1-codfw - T224511 |
[production] |
| 14:45 |
<andrewbogott> |
reimaging cloudcontrol1003 T221770 |
[production] |
| 14:34 |
<moritzm> |
draining ganeti2006 for eventual reboot to pick up MDS-enabled kernel |
[production] |
| 14:34 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:34 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:32 |
<andrewbogott> |
powering off cloudcontrol1003 as one last check to see what explodes before I reimage it |
[production] |
| 14:30 |
<_joe_> |
installing the new service checker on restbase in eqiad |
[production] |
| 14:29 |
<_joe_> |
installing new service checker version on restbase in codfw |
[production] |
| 14:20 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:20 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:09 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:09 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:01 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:01 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:58 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-reboot |
[production] |
| 13:58 |
<gehel@cumin1001> |
END (ERROR) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=97) |
[production] |
| 13:54 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:54 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:48 |
<urandom> |
decommissioning restbase1015-c -- T223976 |
[production] |
| 13:35 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:35 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:19 |
<zfilipin@deploy1001> |
Synchronized php: group1 wikis to 1.34.0-wmf.7 (duration: 00m 58s) |
[production] |
| 13:18 |
<zfilipin@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.34.0-wmf.7 |
[production] |
| 13:16 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:16 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:12 |
<Urbanecm> |
mwscript emptyUserGroup.php --wiki=fawiki 'uploader' finished (T221441) |
[production] |
| 13:06 |
<andrewbogott> |
stopping openstack services on cloudcontrol1003 in anticipation of a re-image |
[production] |
| 13:03 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-reboot |
[production] |
| 13:02 |
<gehel@cumin1001> |
END (ERROR) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=97) |
[production] |
| 13:02 |
<gehel@cumin1001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 13:02 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 13:01 |
<gehel@cumin1001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 13:01 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 13:00 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-reboot |
[production] |
| 12:42 |
<Zppix> |
[12:27:02] jbond@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 12:41 |
<Zppix> |
[12:27:02] jbond@cumin1001 START - Cookbook sre.hosts.downtime |
[production] |
| 12:40 |
<Zppix> |
[12:23:06] <jijiki> Rolling restart pdfrender on scb* |
[production] |
| 12:39 |
<Zppix> |
[[12:20:49] jbond@cumin1001 START - Cookbook sre.hosts.downtime |
[production] |
| 12:39 |
<Zppix> |
[12:20:49] jbond@cumin1001 START - Cookbook sre.hosts.downtime |
[production] |
| 12:38 |
<Zppix> |
[12:11:55] jbond@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |