|
2020-01-08
ยง
|
| 19:56 |
<joal> |
kill wikidata-articleplaceholder_metrics-coord as it is buggy |
[analytics] |
| 19:56 |
<joal> |
Kill-restart edit-hourly-coord and edit-hourly-druid-coord |
[analytics] |
| 19:48 |
<joal> |
Kill-restart wikidata-articleplaceholder_metrics-coord |
[analytics] |
| 19:44 |
<joal> |
Kill-restart mediawiki-history-load-coord, mediawiki-history-denormalize-coord, mediawiki-history-check_denormalize-coord, mediawiki-history-metrics-coord, mediawiki-history-reduced-coord, mediawiki-history-dumps-coord |
[analytics] |
| 19:42 |
<joal> |
Kill-restart mediawiki-history-load-coord, |
[analytics] |
| 19:31 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'logging-external' . |
[production] |
| 19:29 |
<joal> |
Kill-restart webrequest-druid-daily-coord and webrequest-druid-hourly-coord after deploy |
[analytics] |
| 19:25 |
<James_F> |
Tagged Quibble 0.0.40 @ ba365ab36fd84a87fcfb10e0910255e8b44432f1 T192167 T220586 T236222 T236680 |
[releng] |
| 19:16 |
<joal> |
Deploy refinery on HDFS |
[analytics] |
| 19:16 |
<mutante> |
LDAP - added 'sihe' to 'wmde' and 'nda' (T242080) |
[production] |
| 19:15 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'logging-external' . |
[production] |
| 19:13 |
<joal@deploy1001> |
Finished deploy [analytics/refinery@c205576] (thin): Regular analytics weekly deploy train [thin] (duration: 00m 07s) |
[production] |
| 19:13 |
<joal@deploy1001> |
Started deploy [analytics/refinery@c205576] (thin): Regular analytics weekly deploy train [thin] |
[production] |
| 19:13 |
<joal@deploy1001> |
Finished deploy [analytics/refinery@c205576]: Regular analytics weekly deploy train (duration: 08m 36s) |
[production] |
| 19:04 |
<joal@deploy1001> |
Started deploy [analytics/refinery@c205576]: Regular analytics weekly deploy train |
[production] |
| 19:04 |
<joal> |
Deploy refinery using scap |
[analytics] |
| 18:46 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'logging-external' . |
[production] |
| 18:46 |
<marostegui> |
Remove partitions from dewiki.revision on db1096:3315 T239453 |
[production] |
| 18:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1096:3315', diff saved to https://phabricator.wikimedia.org/P10090 and previous config saved to /var/cache/conftool/dbconfig/20200108-184510-marostegui.json |
[production] |
| 18:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1097:3315', diff saved to https://phabricator.wikimedia.org/P10089 and previous config saved to /var/cache/conftool/dbconfig/20200108-184350-marostegui.json |
[production] |
| 18:39 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'logging-external' . |
[production] |
| 18:36 |
<ppchelko@deploy1001> |
Finished deploy [restbase/deploy@ebb1849]: Clean up Parsoid-PHP transition code & config T241756 (duration: 14m 27s) |
[production] |
| 18:33 |
<volans> |
restarted wikibugs |
[production] |
| 18:30 |
<joal> |
Releasing refinery-0.0.110 to archiva using Jenkins |
[analytics] |
| 18:22 |
<ppchelko@deploy1001> |
Started deploy [restbase/deploy@ebb1849]: Clean up Parsoid-PHP transition code & config T241756 |
[production] |
| 18:21 |
<ppchelko@deploy1001> |
Finished deploy [restbase/deploy@ebb1849] (dev-cluster): Clean up Parsoid-PHP transition code & config T241756 (duration: 02m 41s) |
[production] |
| 18:18 |
<ppchelko@deploy1001> |
Started deploy [restbase/deploy@ebb1849] (dev-cluster): Clean up Parsoid-PHP transition code & config T241756 |
[production] |
| 18:11 |
<joal> |
AQS deployed with new druid datasource (2019-12) |
[analytics] |
| 18:07 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) |
[production] |
| 18:04 |
<elukey@cumin1001> |
START - Cookbook sre.aqs.roll-restart |
[production] |
| 18:03 |
<elukey@cumin1001> |
END (FAIL) - Cookbook sre.aqs.roll-restart (exit_code=99) |
[production] |
| 18:03 |
<elukey@cumin1001> |
START - Cookbook sre.aqs.roll-restart |
[production] |
| 17:52 |
<joal> |
Rerun webrequest-load-wf-text-2020-1-8-15 with updated thresholds after frontend issue |
[analytics] |
| 16:26 |
<halfak> |
deploiying ores 039251f |
[releng] |
| 16:26 |
<halfak> |
deploiying ores 039251f |
[deployment-prep] |
| 16:25 |
<_joe_> |
running puppet on deploy1001 to remove my hot-patch to scap.cfg |
[production] |
| 16:20 |
<ema> |
rolling ats-be restart on !text@eqiad, !text@esams to apply https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/562849/ |
[production] |
| 16:00 |
<bblack> |
re-pooling esams text traffic in DNS |
[production] |
| 15:45 |
<ema> |
cumin -s10 -b1 'A:cp-text_eqiad' 'run-puppet-agent -q ; ats-backend-restart' |
[production] |
| 15:40 |
<vgutierrez> |
restarting ats-tls on esams text nodes |
[production] |
| 15:37 |
<ema> |
cumin -s10 -b1 'A:cp-text_esams' 'run-puppet-agent -q ; ats-backend-restart' |
[production] |
| 15:37 |
<bblack> |
authdns-update to depool esams |
[production] |
| 15:26 |
<otto@deploy1001> |
Synchronized wmf-config/ProductionServices.php: REVERT Make EventBus use TLS for eventgate-analytics - T242224 (duration: 00m 34s) |
[production] |
| 15:23 |
<otto@deploy1001> |
sync-file aborted: REVERT Make EventBus use TLS for eventgate-analytics - T242224 (duration: 03m 56s) |
[production] |
| 15:19 |
<otto@deploy1001> |
sync-file aborted: REVERT Make EventBus use TLS for eventgate-analytics - T242224 (duration: 06m 33s) |
[production] |
| 15:12 |
<otto@deploy1001> |
Scap failed!: 4/11 canaries failed their endpoint checks(http://en.wikipedia.org) |
[production] |
| 15:11 |
<otto@deploy1001> |
sync-file aborted: Make EventBus use TLS for eventgate-analytics - T242224 (duration: 00m 00s) |
[production] |
| 15:10 |
<otto@deploy1001> |
Synchronized wmf-config/ProductionServices.php: Make EventBus use TLS for eventgate-analytics - T242224 (duration: 06m 10s) |
[production] |
| 15:02 |
<XioNoX> |
Routinator 0.6.4 looking good on rpki2001, upgrading rpki1001 - T242197 |
[production] |
| 15:00 |
<ottomata> |
deploying change to use new TLS port for eventgate-analytics - T242224 |
[production] |