|
2019-12-02
ยง
|
| 18:32 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'logging-external' . |
[production] |
| 18:30 |
<joal@deploy1001> |
Started deploy [analytics/refinery@980298b]: Analytics deploy - Fixes for today deploy |
[production] |
| 18:22 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 18:21 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 18:19 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 18:18 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 18:16 |
<onimisionipe@deploy1001> |
Finished deploy [wdqs/wdqs@97d17f6]: New blazegraph and WDQS build plus GUI changes (duration: 15m 42s) |
[production] |
| 18:10 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 18:08 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 18:06 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 18:05 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 18:00 |
<onimisionipe@deploy1001> |
Started deploy [wdqs/wdqs@97d17f6]: New blazegraph and WDQS build plus GUI changes |
[production] |
| 17:56 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@ff7862f]: Switch sr and zh wikipediae back to Parsoid/JS - T229015 (duration: 14m 06s) |
[production] |
| 17:51 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 17:49 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 17:42 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@ff7862f]: Switch sr and zh wikipediae back to Parsoid/JS - T229015 |
[production] |
| 17:29 |
<ppchelko@deploy1001> |
Finished deploy [cpjobqueue/deploy@deafe56]: Followup on cirrusSearchElasticWrite partitioning T230495 (duration: 01m 14s) |
[production] |
| 17:28 |
<ppchelko@deploy1001> |
Started deploy [cpjobqueue/deploy@deafe56]: Followup on cirrusSearchElasticWrite partitioning T230495 |
[production] |
| 17:21 |
<ssastry@deploy1001> |
Finished deploy [parsoid/deploy@743efb0]: Updating Parsoid to ca588b25 + fix broken langconv library / deploy (duration: 07m 48s) |
[production] |
| 17:14 |
<ssastry@deploy1001> |
Started deploy [parsoid/deploy@743efb0]: Updating Parsoid to ca588b25 + fix broken langconv library / deploy |
[production] |
| 17:09 |
<ejegg> |
disabled fundraising job omnimail_groupmember_load |
[production] |
| 16:45 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:43 |
<ejegg> |
updated fundraising internal dashboard from 8fc2726736 to 3a93d2aba4 |
[production] |
| 16:43 |
<effie> |
restart all API cluster in eqiad |
[production] |
| 16:43 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:42 |
<hashar> |
Restarted CI Jenkins |
[production] |
| 16:41 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@3516382]: Switch ru, sr and zh wikipediae to Parsoid/PHP - T229015 (duration: 13m 53s) |
[production] |
| 16:41 |
<ema> |
cp3050: ats-be restart with proxy.config.http.server_session_sharing.pool=global T238494 |
[production] |
| 16:32 |
<ema> |
cp3053: repooling after firmware update T239041 |
[production] |
| 16:27 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@3516382]: Switch ru, sr and zh wikipediae to Parsoid/PHP - T229015 |
[production] |
| 16:19 |
<effie> |
reimage mw1295.eqiad.wmnet mw1294.eqiad.wmnet mw1293.eqiad.wmnet |
[production] |
| 16:11 |
<robh> |
cp3053 depooling and rebooting for firmware update T239041 |
[production] |
| 16:10 |
<robh> |
cp3035 depooling and rebooting for firmware update T239041 |
[production] |
| 15:38 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) |
[production] |
| 15:38 |
<mobrovac@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Parsoid VRS: Switch groups 0 and 1 to Parsoid/PHP - T229015 (duration: 00m 59s) |
[production] |
| 15:35 |
<elukey@cumin1001> |
START - Cookbook sre.aqs.roll-restart |
[production] |
| 15:30 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@d6d5a6e]: Parsoid Proxy: Do not use the fall-back for linting transforms - T239607 (duration: 14m 51s) |
[production] |
| 15:26 |
<effie> |
Rolling restart mw1345-1348 |
[production] |
| 15:15 |
<mobrovac@deploy1001> |
Started deploy [restbase/deploy@d6d5a6e]: Parsoid Proxy: Do not use the fall-back for linting transforms - T239607 |
[production] |
| 14:46 |
<ema> |
cp-ats: set server_session_sharing.match=2 everywhere (puppet re-enable and run) T238494 |
[production] |
| 14:31 |
<ema> |
cp-ats: merge server_session_sharing.match=2 (https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/553490/) with puppet disabled, test on cp3050 T238494 |
[production] |
| 14:18 |
<godog> |
set grafana theme back to light, was dark for some reason |
[production] |
| 14:10 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 14:08 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1078 for schema change', diff saved to https://phabricator.wikimedia.org/P9794 and previous config saved to /var/cache/conftool/dbconfig/20191202-135643-marostegui.json |
[production] |
| 13:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1075 after schema change', diff saved to https://phabricator.wikimedia.org/P9793 and previous config saved to /var/cache/conftool/dbconfig/20191202-135543-marostegui.json |
[production] |
| 13:47 |
<ema> |
power-cycle cp3053 T239041 |
[production] |
| 13:44 |
<hashar> |
Restarted CI Jenkins |
[production] |
| 13:30 |
<hashar> |
Restarted CI Jenkins |
[production] |
| 13:14 |
<mobrovac@deploy1001> |
Finished deploy [restbase/deploy@eedba38]: Parsoid Proxy: Fixes - T229015 (duration: 14m 49s) |
[production] |