4151-4200 of 10000 results (68ms)
2019-05-29 ยง
14:54 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
14:54 <moritzm> draining ganeti2007 for eventual reboot to pick up MDS-enabled kernel [production]
14:51 <XioNoX> `request chassis fpc online slot 0` on cr1-codfw - T224511 [production]
14:48 <XioNoX> `request chassis fpc offline slot 0` on cr1-codfw - T224511 [production]
14:47 <XioNoX> disable et- interfaces on cr1-codfw - T224511 [production]
14:45 <andrewbogott> reimaging cloudcontrol1003 T221770 [production]
14:34 <moritzm> draining ganeti2006 for eventual reboot to pick up MDS-enabled kernel [production]
14:34 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:34 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
14:32 <andrewbogott> powering off cloudcontrol1003 as one last check to see what explodes before I reimage it [production]
14:30 <_joe_> installing the new service checker on restbase in eqiad [production]
14:29 <_joe_> installing new service checker version on restbase in codfw [production]
14:20 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:20 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:09 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:09 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:01 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:01 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:58 <gehel@cumin1001> START - Cookbook sre.elasticsearch.rolling-reboot [production]
13:58 <gehel@cumin1001> END (ERROR) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=97) [production]
13:54 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:54 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:48 <urandom> decommissioning restbase1015-c -- T223976 [production]
13:35 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:35 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
13:19 <zfilipin@deploy1001> Synchronized php: group1 wikis to 1.34.0-wmf.7 (duration: 00m 58s) [production]
13:18 <zfilipin@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.34.0-wmf.7 [production]
13:16 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:16 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:12 <Urbanecm> mwscript emptyUserGroup.php --wiki=fawiki 'uploader' finished (T221441) [production]
13:06 <andrewbogott> stopping openstack services on cloudcontrol1003 in anticipation of a re-image [production]
13:03 <gehel@cumin1001> START - Cookbook sre.elasticsearch.rolling-reboot [production]
13:02 <gehel@cumin1001> END (ERROR) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=97) [production]
13:02 <gehel@cumin1001> END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) [production]
13:02 <gehel@cumin1001> START - Cookbook sre.elasticsearch.force-shard-allocation [production]
13:01 <gehel@cumin1001> END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) [production]
13:01 <gehel@cumin1001> START - Cookbook sre.elasticsearch.force-shard-allocation [production]
13:00 <gehel@cumin1001> START - Cookbook sre.elasticsearch.rolling-reboot [production]
12:42 <Zppix> [12:27:02] jbond@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
12:41 <Zppix> [12:27:02] jbond@cumin1001 START - Cookbook sre.hosts.downtime [production]
12:40 <Zppix> [12:23:06] <jijiki> Rolling restart pdfrender on scb* [production]
12:39 <Zppix> [[12:20:49] jbond@cumin1001 START - Cookbook sre.hosts.downtime [production]
12:39 <Zppix> [12:20:49] jbond@cumin1001 START - Cookbook sre.hosts.downtime [production]
12:38 <Zppix> [12:11:55] jbond@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
12:38 <Zppix> [12:11:54] jbond@cumin1001 START - Cookbook sre.hosts.downtime [production]
12:37 <Zppix> [12:01:54] jbond@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0 [production]
12:36 <Zppix> [12:01:54] jbond@cumin1001 START - Cookbook sre.hosts.downtime [production]
12:35 <Zppix> [12:00:21] <logmsgbot> !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Remove db2037 from config as it will be decommissioned T221533 (duration: 00m 56s) [production]
12:35 <jbond@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
12:35 <jbond@cumin1001> START - Cookbook sre.hosts.downtime [production]