|
2019-05-29
ยง
|
| 14:45 |
<andrewbogott> |
reimaging cloudcontrol1003 T221770 |
[production] |
| 14:34 |
<moritzm> |
draining ganeti2006 for eventual reboot to pick up MDS-enabled kernel |
[production] |
| 14:34 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:34 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:32 |
<andrewbogott> |
powering off cloudcontrol1003 as one last check to see what explodes before I reimage it |
[production] |
| 14:30 |
<_joe_> |
installing the new service checker on restbase in eqiad |
[production] |
| 14:29 |
<_joe_> |
installing new service checker version on restbase in codfw |
[production] |
| 14:20 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:20 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:09 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:09 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:01 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:01 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:58 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-reboot |
[production] |
| 13:58 |
<gehel@cumin1001> |
END (ERROR) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=97) |
[production] |
| 13:54 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:54 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:48 |
<urandom> |
decommissioning restbase1015-c -- T223976 |
[production] |
| 13:35 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:35 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:19 |
<zfilipin@deploy1001> |
Synchronized php: group1 wikis to 1.34.0-wmf.7 (duration: 00m 58s) |
[production] |
| 13:18 |
<zfilipin@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.34.0-wmf.7 |
[production] |
| 13:16 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:16 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:12 |
<Urbanecm> |
mwscript emptyUserGroup.php --wiki=fawiki 'uploader' finished (T221441) |
[production] |
| 13:06 |
<andrewbogott> |
stopping openstack services on cloudcontrol1003 in anticipation of a re-image |
[production] |
| 13:03 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-reboot |
[production] |
| 13:02 |
<gehel@cumin1001> |
END (ERROR) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=97) |
[production] |
| 13:02 |
<gehel@cumin1001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 13:02 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 13:01 |
<gehel@cumin1001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 13:01 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 13:00 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-reboot |
[production] |
| 12:42 |
<Zppix> |
[12:27:02] jbond@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 12:41 |
<Zppix> |
[12:27:02] jbond@cumin1001 START - Cookbook sre.hosts.downtime |
[production] |
| 12:40 |
<Zppix> |
[12:23:06] <jijiki> Rolling restart pdfrender on scb* |
[production] |
| 12:39 |
<Zppix> |
[[12:20:49] jbond@cumin1001 START - Cookbook sre.hosts.downtime |
[production] |
| 12:39 |
<Zppix> |
[12:20:49] jbond@cumin1001 START - Cookbook sre.hosts.downtime |
[production] |
| 12:38 |
<Zppix> |
[12:11:55] jbond@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 12:38 |
<Zppix> |
[12:11:54] jbond@cumin1001 START - Cookbook sre.hosts.downtime |
[production] |
| 12:37 |
<Zppix> |
[12:01:54] jbond@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0 |
[production] |
| 12:36 |
<Zppix> |
[12:01:54] jbond@cumin1001 START - Cookbook sre.hosts.downtime |
[production] |
| 12:35 |
<Zppix> |
[12:00:21] <logmsgbot> !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Remove db2037 from config as it will be decommissioned T221533 (duration: 00m 56s) |
[production] |
| 12:35 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 12:35 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:34 |
<Zppix> |
[11:59:19] marostegui@deploy1001 Synchronized wmf-config/db-codfw.php: Remove db2037 from config as it will be decommissioned T221533 |
[production] |
| 12:33 |
<Zppix> |
[11:58:16] <arturo> T221770 icinga downtime cloudcontrol1003.wikimedia.org for upcoming rebuild as stretch |
[production] |
| 12:32 |
<Zppix> |
[11:57:57] aborrero@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 12:32 |
<Zppix> |
[11:57:55] aborrero@cumin1001 START - Cookbook sre.hosts.downtime |
[production] |
| 12:31 |
<Zppix> |
[11:55:54] <Urbanecm> EU SWAT finished, maintenance script emptyUserGroup.php still running in separate tmux session |
[production] |