351-400 of 10000 results (54ms)
2019-08-15 §
08:52 <gehel@cumin1001> START - Cookbook sre.elasticsearch.force-shard-allocation [production]
07:35 <ema> cp5002: ats-backend-restart to enable compress plugin [production]
06:38 <ema> wdqs1009: restart wdqs-updater.service [production]
00:15 <robh> scs-ulsfo offline due to networking issues, rob returning tomorrow with fix T230077 [production]
00:03 <twentyafterfour> starting phabricator upgrade to 2019-08-14/1 refs T215697 [production]
2019-08-14 §
23:13 <ebernhardson> leave cloudelastic writes paused, and dropping from backlog queue, to allow primary clusters to catch up [production]
22:41 <eileen> civicrm revision changed from 569e52e23d to 3caf54a0d2, config revision is 1c76e94ac3 [production]
22:38 <ebernhardson> freeze writes to cloudelastic for real this time [production]
22:03 <ejegg> updated fundraising python tools from 827ce3750e to 5c080bac63 [production]
22:01 <robh> starting scs-ulsfo replacement. There will be icinga errors and they are intentionally being allowed so we know when things dont recover properly T230077 [production]
21:37 <XioNoX> advertise core v6 range (2620:0:860::/46) from eqord - T167841 [production]
21:30 <@> helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' . [production]
21:26 <ebernhardson> thaw writes to cloudelastic [production]
21:24 <ejegg> updated payments-wiki from 9533f70fab to 44eae2d65f [production]
21:18 <@> helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' . [production]
21:13 <ebernhardson> apply freeze to cloudelastic writes, to determine if backlog processing can catchup while deferring cloudelastic writes [production]
20:49 <@> helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' . [production]
20:46 <@> helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' . [production]
20:44 <@> helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' . [production]
20:44 <@> helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' . [production]
18:01 <gehel@cumin1001> END (PASS) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=0) [production]
17:29 <gehel@cumin1001> START - Cookbook sre.elasticsearch.rolling-reboot [production]
16:32 <gehel@cumin1001> END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) [production]
16:32 <gehel@cumin1001> START - Cookbook sre.elasticsearch.force-shard-allocation [production]
16:31 <gehel@cumin1001> END (ERROR) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=97) [production]
15:50 <ema> cp5002: ats-backend-restart to disable compress plugin while I'm not around [production]
15:45 <gehel@cumin1001> START - Cookbook sre.elasticsearch.rolling-reboot [production]
15:41 <gehel> powercycling elastic101[789] [production]
15:30 <gehel@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=99) [production]
14:55 <vgutierrez> upgrade nginx to 1.13.9-1wm2 in cp3032 [production]
14:17 <fsero> upgrading envoy package to 1.11.1 [production]
14:09 <vgutierrez> rolling back nginx upgrade in cp3032 [production]
14:01 <reedy@deploy1001> Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 03m 04s) [production]
13:58 <reedy@deploy1001> Synchronized static/images/project-logos/: T210752 (duration: 00m 47s) [production]
13:56 <reedy@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T210752 (duration: 00m 47s) [production]
13:55 <reedy@deploy1001> rebuilt and synchronized wikiversions files: T212881 [production]
13:53 <reedy@deploy1001> Synchronized dblists/: T212881 (duration: 00m 48s) [production]
12:48 <gehel@cumin1001> START - Cookbook sre.elasticsearch.rolling-reboot [production]
12:47 <James_F> <sadtrombone> Wiki creation is still not working correctly, unfortunately. [production]
12:24 <James_F|Away> We're going to try making a new wiki. T212881 [production]
12:20 <vgutierrez> rolling upgrade of nginx to 1.13.9-1+wmf2 in the cache cluster [production]
12:17 <gehel@cumin1001> END (ERROR) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=97) [production]
11:20 <vgutierrez> repooling cp5002 [production]
11:19 <tarrow> termbox smoketests finished [production]
11:06 <gehel@cumin1001> START - Cookbook sre.elasticsearch.rolling-reboot [production]
10:46 <ema> depool cp5002 after crash. See /var/log/trafficserver/crash-2019-08-14-104502.log [production]
10:28 <tarrow> Starting smoketest of termbox service on eqiad: T229907 [production]
09:40 <gehel@cumin1001> END (ERROR) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=97) [production]
09:20 <ema> cp5002: ats-backend-restart to enable compress plugin [production]
08:52 <vgutierrez> upgrading nginx to 1.13.9-1+wmf2 in cp1075, cp2001, cp3030 and cp4027 (text) and cp1076, cp2002, cp3034, cp4021 (upload) [production]