|
2019-08-14
§
|
| 23:13 |
<ebernhardson> |
leave cloudelastic writes paused, and dropping from backlog queue, to allow primary clusters to catch up |
[production] |
| 22:41 |
<eileen> |
civicrm revision changed from 569e52e23d to 3caf54a0d2, config revision is 1c76e94ac3 |
[production] |
| 22:38 |
<ebernhardson> |
freeze writes to cloudelastic for real this time |
[production] |
| 22:03 |
<ejegg> |
updated fundraising python tools from 827ce3750e to 5c080bac63 |
[production] |
| 22:01 |
<robh> |
starting scs-ulsfo replacement. There will be icinga errors and they are intentionally being allowed so we know when things dont recover properly T230077 |
[production] |
| 21:37 |
<XioNoX> |
advertise core v6 range (2620:0:860::/46) from eqord - T167841 |
[production] |
| 21:30 |
<@> |
helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' . |
[production] |
| 21:26 |
<ebernhardson> |
thaw writes to cloudelastic |
[production] |
| 21:24 |
<ejegg> |
updated payments-wiki from 9533f70fab to 44eae2d65f |
[production] |
| 21:18 |
<@> |
helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' . |
[production] |
| 21:13 |
<ebernhardson> |
apply freeze to cloudelastic writes, to determine if backlog processing can catchup while deferring cloudelastic writes |
[production] |
| 20:49 |
<@> |
helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' . |
[production] |
| 20:46 |
<@> |
helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' . |
[production] |
| 20:44 |
<@> |
helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' . |
[production] |
| 20:44 |
<@> |
helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' . |
[production] |
| 18:01 |
<gehel@cumin1001> |
END (PASS) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=0) |
[production] |
| 17:29 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-reboot |
[production] |
| 16:32 |
<gehel@cumin1001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 16:32 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 16:31 |
<gehel@cumin1001> |
END (ERROR) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=97) |
[production] |
| 15:50 |
<ema> |
cp5002: ats-backend-restart to disable compress plugin while I'm not around |
[production] |
| 15:45 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-reboot |
[production] |
| 15:41 |
<gehel> |
powercycling elastic101[789] |
[production] |
| 15:30 |
<gehel@cumin1001> |
END (FAIL) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=99) |
[production] |
| 14:55 |
<vgutierrez> |
upgrade nginx to 1.13.9-1wm2 in cp3032 |
[production] |
| 14:17 |
<fsero> |
upgrading envoy package to 1.11.1 |
[production] |
| 14:09 |
<vgutierrez> |
rolling back nginx upgrade in cp3032 |
[production] |
| 14:01 |
<reedy@deploy1001> |
Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 03m 04s) |
[production] |
| 13:58 |
<reedy@deploy1001> |
Synchronized static/images/project-logos/: T210752 (duration: 00m 47s) |
[production] |
| 13:56 |
<reedy@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T210752 (duration: 00m 47s) |
[production] |
| 13:55 |
<reedy@deploy1001> |
rebuilt and synchronized wikiversions files: T212881 |
[production] |
| 13:53 |
<reedy@deploy1001> |
Synchronized dblists/: T212881 (duration: 00m 48s) |
[production] |
| 12:48 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-reboot |
[production] |
| 12:47 |
<James_F> |
<sadtrombone> Wiki creation is still not working correctly, unfortunately. |
[production] |
| 12:24 |
<James_F|Away> |
We're going to try making a new wiki. T212881 |
[production] |
| 12:20 |
<vgutierrez> |
rolling upgrade of nginx to 1.13.9-1+wmf2 in the cache cluster |
[production] |
| 12:17 |
<gehel@cumin1001> |
END (ERROR) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=97) |
[production] |
| 11:20 |
<vgutierrez> |
repooling cp5002 |
[production] |
| 11:19 |
<tarrow> |
termbox smoketests finished |
[production] |
| 11:06 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-reboot |
[production] |
| 10:46 |
<ema> |
depool cp5002 after crash. See /var/log/trafficserver/crash-2019-08-14-104502.log |
[production] |
| 10:28 |
<tarrow> |
Starting smoketest of termbox service on eqiad: T229907 |
[production] |
| 09:40 |
<gehel@cumin1001> |
END (ERROR) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=97) |
[production] |
| 09:20 |
<ema> |
cp5002: ats-backend-restart to enable compress plugin |
[production] |
| 08:52 |
<vgutierrez> |
upgrading nginx to 1.13.9-1+wmf2 in cp1075, cp2001, cp3030 and cp4027 (text) and cp1076, cp2002, cp3034, cp4021 (upload) |
[production] |
| 08:25 |
<vgutierrez> |
upgrading nginx to 1.13.9-1+wmf2 in cp5001 (upload) and cp5007 (text) |
[production] |
| 08:17 |
<vgutierrez> |
uploaded nginx-1.13.9-1+wmf2 to apt.wikimedia.org (stretch) |
[production] |
| 08:16 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-reboot |
[production] |
| 08:12 |
<gehel@cumin1001> |
END (FAIL) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=99) |
[production] |