|
2019-08-14
§
|
| 21:13 |
<ebernhardson> |
apply freeze to cloudelastic writes, to determine if backlog processing can catchup while deferring cloudelastic writes |
[production] |
| 20:49 |
<@> |
helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' . |
[production] |
| 20:46 |
<@> |
helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' . |
[production] |
| 20:44 |
<@> |
helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' . |
[production] |
| 20:44 |
<@> |
helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' . |
[production] |
| 18:01 |
<gehel@cumin1001> |
END (PASS) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=0) |
[production] |
| 17:29 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-reboot |
[production] |
| 16:32 |
<gehel@cumin1001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 16:32 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 16:31 |
<gehel@cumin1001> |
END (ERROR) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=97) |
[production] |
| 15:50 |
<ema> |
cp5002: ats-backend-restart to disable compress plugin while I'm not around |
[production] |
| 15:45 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-reboot |
[production] |
| 15:41 |
<gehel> |
powercycling elastic101[789] |
[production] |
| 15:30 |
<gehel@cumin1001> |
END (FAIL) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=99) |
[production] |
| 14:55 |
<vgutierrez> |
upgrade nginx to 1.13.9-1wm2 in cp3032 |
[production] |
| 14:17 |
<fsero> |
upgrading envoy package to 1.11.1 |
[production] |
| 14:09 |
<vgutierrez> |
rolling back nginx upgrade in cp3032 |
[production] |
| 14:01 |
<reedy@deploy1001> |
Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 03m 04s) |
[production] |
| 13:58 |
<reedy@deploy1001> |
Synchronized static/images/project-logos/: T210752 (duration: 00m 47s) |
[production] |
| 13:56 |
<reedy@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T210752 (duration: 00m 47s) |
[production] |
| 13:55 |
<reedy@deploy1001> |
rebuilt and synchronized wikiversions files: T212881 |
[production] |
| 13:53 |
<reedy@deploy1001> |
Synchronized dblists/: T212881 (duration: 00m 48s) |
[production] |
| 12:48 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-reboot |
[production] |
| 12:47 |
<James_F> |
<sadtrombone> Wiki creation is still not working correctly, unfortunately. |
[production] |
| 12:24 |
<James_F|Away> |
We're going to try making a new wiki. T212881 |
[production] |
| 12:20 |
<vgutierrez> |
rolling upgrade of nginx to 1.13.9-1+wmf2 in the cache cluster |
[production] |
| 12:17 |
<gehel@cumin1001> |
END (ERROR) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=97) |
[production] |
| 11:20 |
<vgutierrez> |
repooling cp5002 |
[production] |
| 11:19 |
<tarrow> |
termbox smoketests finished |
[production] |
| 11:06 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-reboot |
[production] |
| 10:46 |
<ema> |
depool cp5002 after crash. See /var/log/trafficserver/crash-2019-08-14-104502.log |
[production] |
| 10:28 |
<tarrow> |
Starting smoketest of termbox service on eqiad: T229907 |
[production] |
| 09:40 |
<gehel@cumin1001> |
END (ERROR) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=97) |
[production] |
| 09:20 |
<ema> |
cp5002: ats-backend-restart to enable compress plugin |
[production] |
| 08:52 |
<vgutierrez> |
upgrading nginx to 1.13.9-1+wmf2 in cp1075, cp2001, cp3030 and cp4027 (text) and cp1076, cp2002, cp3034, cp4021 (upload) |
[production] |
| 08:25 |
<vgutierrez> |
upgrading nginx to 1.13.9-1+wmf2 in cp5001 (upload) and cp5007 (text) |
[production] |
| 08:17 |
<vgutierrez> |
uploaded nginx-1.13.9-1+wmf2 to apt.wikimedia.org (stretch) |
[production] |
| 08:16 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-reboot |
[production] |
| 08:12 |
<gehel@cumin1001> |
END (FAIL) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=99) |
[production] |
| 08:10 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-reboot |
[production] |
| 07:09 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Remove db2063 from config T230459 (duration: 00m 47s) |
[production] |
| 07:08 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Remove db2063 from config T230459 (duration: 00m 48s) |
[production] |
|
2019-08-13
§
|
| 20:43 |
<ejegg> |
rolled back payments-wiki from 9ed8be0532 to 9533f70fab |
[production] |
| 20:34 |
<ejegg> |
updated payments-wiki from 9533f70fab to 9ed8be0532 |
[production] |
| 20:32 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/InitialiseSettings-labs.php: Fix MachineVision provider config (duration: 00m 47s) |
[production] |
| 19:48 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=0) |
[production] |
| 19:23 |
<ppchelko@deploy1001> |
Finished deploy [cpjobqueue/deploy@3882ddb]: Increase cirrusSearchLinksUpdatePrioritized concurrency 150 -> 200 T220625 (duration: 00m 58s) |
[production] |
| 19:22 |
<ppchelko@deploy1001> |
Started deploy [cpjobqueue/deploy@3882ddb]: Increase cirrusSearchLinksUpdatePrioritized concurrency 150 -> 200 T220625 |
[production] |
| 19:19 |
<ppchelko@deploy1001> |
deploy aborted: Revert on canary (duration: 00m 18s) |
[production] |
| 19:18 |
<ppchelko@deploy1001> |
Started deploy [cpjobqueue/deploy@f1a562e]: Revert on canary |
[production] |