|
2019-08-15
§
|
| 11:26 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove napwikisource from wgProofreadPageNamespaceIds (T230541) (duration: 00m 47s) |
[production] |
| 11:12 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: 0d8c516: Fix addition of Hubblesite.org and Spacetelescope.org to commons wgCopyUploadsDomains (T230083) (duration: 00m 48s) |
[production] |
| 10:50 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T230533: Add more import sources for napwikisource (duration: 00m 52s) |
[production] |
| 08:54 |
<gehel@cumin1001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 08:54 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 08:52 |
<gehel@cumin1001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 08:52 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 07:35 |
<ema> |
cp5002: ats-backend-restart to enable compress plugin |
[production] |
| 06:38 |
<ema> |
wdqs1009: restart wdqs-updater.service |
[production] |
| 00:15 |
<robh> |
scs-ulsfo offline due to networking issues, rob returning tomorrow with fix T230077 |
[production] |
| 00:03 |
<twentyafterfour> |
starting phabricator upgrade to 2019-08-14/1 refs T215697 |
[production] |
|
2019-08-14
§
|
| 23:13 |
<ebernhardson> |
leave cloudelastic writes paused, and dropping from backlog queue, to allow primary clusters to catch up |
[production] |
| 22:41 |
<eileen> |
civicrm revision changed from 569e52e23d to 3caf54a0d2, config revision is 1c76e94ac3 |
[production] |
| 22:38 |
<ebernhardson> |
freeze writes to cloudelastic for real this time |
[production] |
| 22:03 |
<ejegg> |
updated fundraising python tools from 827ce3750e to 5c080bac63 |
[production] |
| 22:01 |
<robh> |
starting scs-ulsfo replacement. There will be icinga errors and they are intentionally being allowed so we know when things dont recover properly T230077 |
[production] |
| 21:37 |
<XioNoX> |
advertise core v6 range (2620:0:860::/46) from eqord - T167841 |
[production] |
| 21:30 |
<@> |
helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' . |
[production] |
| 21:26 |
<ebernhardson> |
thaw writes to cloudelastic |
[production] |
| 21:24 |
<ejegg> |
updated payments-wiki from 9533f70fab to 44eae2d65f |
[production] |
| 21:18 |
<@> |
helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' . |
[production] |
| 21:13 |
<ebernhardson> |
apply freeze to cloudelastic writes, to determine if backlog processing can catchup while deferring cloudelastic writes |
[production] |
| 20:49 |
<@> |
helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' . |
[production] |
| 20:46 |
<@> |
helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' . |
[production] |
| 20:44 |
<@> |
helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' . |
[production] |
| 20:44 |
<@> |
helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' . |
[production] |
| 18:01 |
<gehel@cumin1001> |
END (PASS) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=0) |
[production] |
| 17:29 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-reboot |
[production] |
| 16:32 |
<gehel@cumin1001> |
END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) |
[production] |
| 16:32 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.force-shard-allocation |
[production] |
| 16:31 |
<gehel@cumin1001> |
END (ERROR) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=97) |
[production] |
| 15:50 |
<ema> |
cp5002: ats-backend-restart to disable compress plugin while I'm not around |
[production] |
| 15:45 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-reboot |
[production] |
| 15:41 |
<gehel> |
powercycling elastic101[789] |
[production] |
| 15:30 |
<gehel@cumin1001> |
END (FAIL) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=99) |
[production] |
| 14:55 |
<vgutierrez> |
upgrade nginx to 1.13.9-1wm2 in cp3032 |
[production] |
| 14:17 |
<fsero> |
upgrading envoy package to 1.11.1 |
[production] |
| 14:09 |
<vgutierrez> |
rolling back nginx upgrade in cp3032 |
[production] |
| 14:01 |
<reedy@deploy1001> |
Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 03m 04s) |
[production] |
| 13:58 |
<reedy@deploy1001> |
Synchronized static/images/project-logos/: T210752 (duration: 00m 47s) |
[production] |
| 13:56 |
<reedy@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: T210752 (duration: 00m 47s) |
[production] |
| 13:55 |
<reedy@deploy1001> |
rebuilt and synchronized wikiversions files: T212881 |
[production] |
| 13:53 |
<reedy@deploy1001> |
Synchronized dblists/: T212881 (duration: 00m 48s) |
[production] |
| 12:48 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-reboot |
[production] |
| 12:47 |
<James_F> |
<sadtrombone> Wiki creation is still not working correctly, unfortunately. |
[production] |
| 12:24 |
<James_F|Away> |
We're going to try making a new wiki. T212881 |
[production] |
| 12:20 |
<vgutierrez> |
rolling upgrade of nginx to 1.13.9-1+wmf2 in the cache cluster |
[production] |
| 12:17 |
<gehel@cumin1001> |
END (ERROR) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=97) |
[production] |
| 11:20 |
<vgutierrez> |
repooling cp5002 |
[production] |
| 11:19 |
<tarrow> |
termbox smoketests finished |
[production] |