|
2020-01-08
ยง
|
| 21:30 |
<mutante> |
phab1003 - running decom cookbook - shutdown host, removed from puppetmaster, debmonitor etc (T238957) |
[production] |
| 21:30 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 21:29 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 21:28 |
<jhuneidi@deploy1001> |
rebuilt and synchronized wikiversions files: Revert "commonswiki to 1.35.0-wmf.11" |
[production] |
| 21:21 |
<halfak@deploy1001> |
Finished deploy [ores/deploy@039251f]: T242035 (duration: 16m 32s) |
[production] |
| 21:07 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-main' for release 'main' . |
[production] |
| 21:04 |
<halfak@deploy1001> |
Started deploy [ores/deploy@039251f]: T242035 |
[production] |
| 21:03 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'logging-external' . |
[production] |
| 21:00 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-analytics' for release 'analytics' . |
[production] |
| 20:53 |
<XenoRyet> |
updated civicrm from 51b6fca9b2 to 2468d85f95 |
[production] |
| 20:51 |
<jhuneidi@deploy1001> |
Synchronized php: group1 wikis to 1.35.0-wmf.14 refs T233862 (duration: 01m 04s) |
[production] |
| 20:50 |
<jhuneidi@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.35.0-wmf.14 refs T233862 |
[production] |
| 20:40 |
<mutante> |
contint1001 - restarting zuul service |
[production] |
| 20:00 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-analytics' for release 'analytics' . |
[production] |
| 19:31 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'logging-external' . |
[production] |
| 19:16 |
<mutante> |
LDAP - added 'sihe' to 'wmde' and 'nda' (T242080) |
[production] |
| 19:15 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'logging-external' . |
[production] |
| 19:13 |
<joal@deploy1001> |
Finished deploy [analytics/refinery@c205576] (thin): Regular analytics weekly deploy train [thin] (duration: 00m 07s) |
[production] |
| 19:13 |
<joal@deploy1001> |
Started deploy [analytics/refinery@c205576] (thin): Regular analytics weekly deploy train [thin] |
[production] |
| 19:13 |
<joal@deploy1001> |
Finished deploy [analytics/refinery@c205576]: Regular analytics weekly deploy train (duration: 08m 36s) |
[production] |
| 19:04 |
<joal@deploy1001> |
Started deploy [analytics/refinery@c205576]: Regular analytics weekly deploy train |
[production] |
| 18:46 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'logging-external' . |
[production] |
| 18:46 |
<marostegui> |
Remove partitions from dewiki.revision on db1096:3315 T239453 |
[production] |
| 18:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1096:3315', diff saved to https://phabricator.wikimedia.org/P10090 and previous config saved to /var/cache/conftool/dbconfig/20200108-184510-marostegui.json |
[production] |
| 18:43 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1097:3315', diff saved to https://phabricator.wikimedia.org/P10089 and previous config saved to /var/cache/conftool/dbconfig/20200108-184350-marostegui.json |
[production] |
| 18:39 |
<otto@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-logging-external' for release 'logging-external' . |
[production] |
| 18:36 |
<ppchelko@deploy1001> |
Finished deploy [restbase/deploy@ebb1849]: Clean up Parsoid-PHP transition code & config T241756 (duration: 14m 27s) |
[production] |
| 18:33 |
<volans> |
restarted wikibugs |
[production] |
| 18:22 |
<ppchelko@deploy1001> |
Started deploy [restbase/deploy@ebb1849]: Clean up Parsoid-PHP transition code & config T241756 |
[production] |
| 18:21 |
<ppchelko@deploy1001> |
Finished deploy [restbase/deploy@ebb1849] (dev-cluster): Clean up Parsoid-PHP transition code & config T241756 (duration: 02m 41s) |
[production] |
| 18:18 |
<ppchelko@deploy1001> |
Started deploy [restbase/deploy@ebb1849] (dev-cluster): Clean up Parsoid-PHP transition code & config T241756 |
[production] |
| 18:07 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) |
[production] |
| 18:04 |
<elukey@cumin1001> |
START - Cookbook sre.aqs.roll-restart |
[production] |
| 18:03 |
<elukey@cumin1001> |
END (FAIL) - Cookbook sre.aqs.roll-restart (exit_code=99) |
[production] |
| 18:03 |
<elukey@cumin1001> |
START - Cookbook sre.aqs.roll-restart |
[production] |
| 16:25 |
<_joe_> |
running puppet on deploy1001 to remove my hot-patch to scap.cfg |
[production] |
| 16:20 |
<ema> |
rolling ats-be restart on !text@eqiad, !text@esams to apply https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/562849/ |
[production] |
| 16:00 |
<bblack> |
re-pooling esams text traffic in DNS |
[production] |
| 15:45 |
<ema> |
cumin -s10 -b1 'A:cp-text_eqiad' 'run-puppet-agent -q ; ats-backend-restart' |
[production] |
| 15:40 |
<vgutierrez> |
restarting ats-tls on esams text nodes |
[production] |
| 15:37 |
<ema> |
cumin -s10 -b1 'A:cp-text_esams' 'run-puppet-agent -q ; ats-backend-restart' |
[production] |
| 15:37 |
<bblack> |
authdns-update to depool esams |
[production] |
| 15:26 |
<otto@deploy1001> |
Synchronized wmf-config/ProductionServices.php: REVERT Make EventBus use TLS for eventgate-analytics - T242224 (duration: 00m 34s) |
[production] |
| 15:23 |
<otto@deploy1001> |
sync-file aborted: REVERT Make EventBus use TLS for eventgate-analytics - T242224 (duration: 03m 56s) |
[production] |
| 15:19 |
<otto@deploy1001> |
sync-file aborted: REVERT Make EventBus use TLS for eventgate-analytics - T242224 (duration: 06m 33s) |
[production] |
| 15:12 |
<otto@deploy1001> |
Scap failed!: 4/11 canaries failed their endpoint checks(http://en.wikipedia.org) |
[production] |
| 15:11 |
<otto@deploy1001> |
sync-file aborted: Make EventBus use TLS for eventgate-analytics - T242224 (duration: 00m 00s) |
[production] |
| 15:10 |
<otto@deploy1001> |
Synchronized wmf-config/ProductionServices.php: Make EventBus use TLS for eventgate-analytics - T242224 (duration: 06m 10s) |
[production] |
| 15:02 |
<XioNoX> |
Routinator 0.6.4 looking good on rpki2001, upgrading rpki1001 - T242197 |
[production] |
| 15:00 |
<ottomata> |
deploying change to use new TLS port for eventgate-analytics - T242224 |
[production] |