|
2019-07-16
ยง
|
| 18:58 |
<gehel@cumin1001> |
END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) |
[production] |
| 18:58 |
<gehel@cumin1001> |
START - Cookbook sre.wdqs.data-transfer |
[production] |
| 18:54 |
<gehel> |
data copy from wdqs2004 to wdqs2001 - T228122 |
[production] |
| 18:46 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: retry - Produce revision-create stream to eventgate-main - T211248 (duration: 00m 54s) |
[production] |
| 18:22 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Produce revision-create stream to eventgate-main - T211248 (duration: 00m 54s) |
[production] |
| 18:08 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Update ExtensionDistributor config to point to REL1_33 as the released version (duration: 00m 54s) |
[production] |
| 18:05 |
<fsero> |
republishing base images for nodejs-slim due to registry T228196 |
[production] |
| 18:02 |
<andrewbogott> |
rebooting cloudcontrol2003-dev, cloudweb2001-dev, cloudcontrol1004 for T225713 |
[production] |
| 17:39 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Produce centralnotice.campaign-* streams to eventgate-main - T211248 (duration: 00m 55s) |
[production] |
| 17:23 |
<bsitzmann@deploy1001> |
Finished deploy [mobileapps/deploy@cb6e7bc]: Update mobileapps to 334a4c4 (T227907) (duration: 04m 51s) |
[production] |
| 17:19 |
<bsitzmann@deploy1001> |
Started deploy [mobileapps/deploy@cb6e7bc]: Update mobileapps to 334a4c4 (T227907) |
[production] |
| 16:55 |
<mutante> |
netmon1003: shutdown -h now | ganeti1001: gnt-instance shutdown netmon1003.wikmedia.org - removed from icinga T198939 T220355 |
[production] |
| 16:36 |
<jiji@deploy1001> |
Finished deploy [cpjobqueue/deploy@5d8128e]: Migrating videoscaling jobs to PHP7 - T219150 (duration: 00m 50s) |
[production] |
| 16:35 |
<jiji@deploy1001> |
Started deploy [cpjobqueue/deploy@5d8128e]: Migrating videoscaling jobs to PHP7 - T219150 |
[production] |
| 16:28 |
<dcausse> |
reindexing wikidata (elastic@eqiad) T227136 |
[production] |
| 15:57 |
<tarrow@> |
helmfile [STAGING] Ran 'apply' command on namespace 'termbox' for release 'staging' . |
[production] |
| 15:37 |
<elukey> |
reboot analytics1072 as attempt to force the raid controller to set a drive failed - T226467 |
[production] |
| 15:12 |
<elukey> |
start mariadb on db1107 and re-enable mysql consumers on eventlog1002 and replication on db1108 |
[production] |
| 14:53 |
<elukey> |
stop mariadb on db1107 to allow maintenance |
[production] |
| 14:53 |
<elukey> |
stop eventlogging mysql consumers on eventlog1002 and eventlogging_sync on db1108 to allow db1107 maintenance |
[production] |
| 14:52 |
<jbond42> |
will restart redis on oresdb at 16:00 UTC - T228045 |
[production] |
| 14:51 |
<jbond42> |
enable puppet accross the fleat |
[production] |
| 14:50 |
<jbond@cumin1001> |
conftool action : set/pooled=yes; selector: name=dns1001.wikimedia.org |
[production] |
| 14:43 |
<jbond@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:43 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 14:43 |
<jbond@cumin1001> |
conftool action : set/pooled=no; selector: name=dns1001.wikimedia.org |
[production] |
| 14:40 |
<jbond42> |
disable puppet accross the fleat to make a change to the hiera |
[production] |
| 14:29 |
<jijiki> |
Enable puppet and rolling restart thumbor* in codfw - T224572 |
[production] |
| 14:16 |
<jijiki> |
Depool thumbor2001 and pool back - T224572 |
[production] |
| 14:13 |
<jijiki> |
Disabling puppet on thumbor*codfw.wmnet - T224572 |
[production] |
| 14:08 |
<liw> |
group0 to 1.34.0-wmf.14 |
[production] |
| 14:06 |
<liw@deploy1001> |
rebuilt and synchronized wikiversions files: group0 to php-1.34.0-wmf.14 |
[production] |
| 13:41 |
<liw@deploy1001> |
Finished scap: testwiki to php-1.34.0-wmf.14 and rebuild l10n cache (duration: 26m 45s) |
[production] |
| 13:24 |
<vgutierrez> |
restarting pybal on lvs2001 and lvs1013 |
[production] |
| 13:20 |
<vgutierrez> |
restarting pybal on lvs2004 and lvs1016 |
[production] |
| 13:14 |
<liw@deploy1001> |
Started scap: testwiki to php-1.34.0-wmf.14 and rebuild l10n cache |
[production] |
| 12:59 |
<liw@deploy1001> |
Pruned MediaWiki: 1.34.0-wmf.8 (duration: 01m 46s) |
[production] |
| 12:57 |
<liw@deploy1001> |
Pruned MediaWiki: 1.34.0-wmf.7 (duration: 02m 01s) |
[production] |
| 12:54 |
<liw@deploy1001> |
Pruned MediaWiki: 1.34.0-wmf.6 (duration: 02m 04s) |
[production] |
| 12:52 |
<liw@deploy1001> |
Pruned MediaWiki: 1.34.0-wmf.4 (duration: 02m 11s) |
[production] |
| 12:49 |
<liw@deploy1001> |
Pruned MediaWiki: 1.34.0-wmf.5 (duration: 07m 42s) |
[production] |
| 12:42 |
<dcausse> |
deleting stale wikidata indices (elastic@eqiad) T227136 |
[production] |
| 12:11 |
<jijiki> |
Depool mw1293 and pool back |
[production] |
| 11:57 |
<moritzm> |
synched docker-ce, docker-ce-cli, containerd.io to thirdparty/ci for stretch-wikimedia (T226236) |
[production] |
| 11:12 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 11:12 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 11:12 |
<moritzm> |
rebooting remaining swift frontends in eqiad to pick up a kernel with SACK fixed (T228086) |
[production] |
| 10:29 |
<moritzm> |
rebooting ms-fe1005 to pick up kernel with SACK fixed (T228086) |
[production] |
| 10:28 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 10:28 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |