|
2019-08-14
§
|
| 11:06 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-reboot |
[production] |
| 10:46 |
<ema> |
depool cp5002 after crash. See /var/log/trafficserver/crash-2019-08-14-104502.log |
[production] |
| 10:28 |
<tarrow> |
Starting smoketest of termbox service on eqiad: T229907 |
[production] |
| 09:40 |
<gehel@cumin1001> |
END (ERROR) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=97) |
[production] |
| 09:20 |
<ema> |
cp5002: ats-backend-restart to enable compress plugin |
[production] |
| 08:52 |
<vgutierrez> |
upgrading nginx to 1.13.9-1+wmf2 in cp1075, cp2001, cp3030 and cp4027 (text) and cp1076, cp2002, cp3034, cp4021 (upload) |
[production] |
| 08:25 |
<vgutierrez> |
upgrading nginx to 1.13.9-1+wmf2 in cp5001 (upload) and cp5007 (text) |
[production] |
| 08:17 |
<vgutierrez> |
uploaded nginx-1.13.9-1+wmf2 to apt.wikimedia.org (stretch) |
[production] |
| 08:16 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-reboot |
[production] |
| 08:12 |
<gehel@cumin1001> |
END (FAIL) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=99) |
[production] |
| 08:10 |
<gehel@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-reboot |
[production] |
| 07:09 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Remove db2063 from config T230459 (duration: 00m 47s) |
[production] |
| 07:08 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Remove db2063 from config T230459 (duration: 00m 48s) |
[production] |
|
2019-08-13
§
|
| 20:43 |
<ejegg> |
rolled back payments-wiki from 9ed8be0532 to 9533f70fab |
[production] |
| 20:34 |
<ejegg> |
updated payments-wiki from 9533f70fab to 9ed8be0532 |
[production] |
| 20:32 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/InitialiseSettings-labs.php: Fix MachineVision provider config (duration: 00m 47s) |
[production] |
| 19:48 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=0) |
[production] |
| 19:23 |
<ppchelko@deploy1001> |
Finished deploy [cpjobqueue/deploy@3882ddb]: Increase cirrusSearchLinksUpdatePrioritized concurrency 150 -> 200 T220625 (duration: 00m 58s) |
[production] |
| 19:22 |
<ppchelko@deploy1001> |
Started deploy [cpjobqueue/deploy@3882ddb]: Increase cirrusSearchLinksUpdatePrioritized concurrency 150 -> 200 T220625 |
[production] |
| 19:19 |
<ppchelko@deploy1001> |
deploy aborted: Revert on canary (duration: 00m 18s) |
[production] |
| 19:18 |
<ppchelko@deploy1001> |
Started deploy [cpjobqueue/deploy@f1a562e]: Revert on canary |
[production] |
| 19:17 |
<ppchelko@deploy1001> |
deploy aborted: Increase cirrusSearchLinksUpdatePrioritized concurrency 150 -> 200 T220625 (duration: 01m 30s) |
[production] |
| 19:15 |
<ppchelko@deploy1001> |
Started deploy [cpjobqueue/deploy@f1a562e]: Increase cirrusSearchLinksUpdatePrioritized concurrency 150 -> 200 T220625 |
[production] |
| 19:03 |
<@> |
helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' . |
[production] |
| 18:50 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.rolling-reboot |
[production] |
| 18:41 |
<ebernhardson> |
set cpufreq scaling_governor to performance on cloudelastic100[1-4] to test any changes to indexing performance |
[production] |
| 18:38 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Enable MachineVision on Beta (4/4) (duration: 00m 48s) |
[production] |
| 18:34 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/InitialiseSettings-labs.php: Enable MachineVision on Beta (3/4) (duration: 00m 47s) |
[production] |
| 18:33 |
<crusnov@deploy1001> |
Finished deploy [netbox/deploy@367ca84]: Update Netbox to v2.6.1-wmf3 affects: T223292 (fix perms) (duration: 00m 09s) |
[production] |
| 18:33 |
<crusnov@deploy1001> |
Started deploy [netbox/deploy@367ca84]: Update Netbox to v2.6.1-wmf3 affects: T223292 (fix perms) |
[production] |
| 18:33 |
<crusnov@deploy1001> |
Finished deploy [netbox/deploy@367ca84]: Update Netbox to v2.6.1-wmf3 affects: T223292 (duration: 00m 43s) |
[production] |
| 18:32 |
<crusnov@deploy1001> |
Started deploy [netbox/deploy@367ca84]: Update Netbox to v2.6.1-wmf3 affects: T223292 |
[production] |
| 18:32 |
<crusnov@deploy1001> |
Finished deploy [netbox/deploy@367ca84]: Update Netbox to v2.6.1-wmf3 affects: T223292 (duration: 00m 36s) |
[production] |
| 18:31 |
<crusnov@deploy1001> |
Started deploy [netbox/deploy@367ca84]: Update Netbox to v2.6.1-wmf3 affects: T223292 |
[production] |
| 18:30 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Enable MachineVision on Beta (2/4) (duration: 00m 48s) |
[production] |
| 18:27 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/extension-list: Enable MachineVision on Beta (1/4) (duration: 00m 48s) |
[production] |
| 17:44 |
<XioNoX> |
set target netflow port to 2000 in eqiad |
[production] |
| 17:11 |
<XioNoX> |
repool eqsin |
[production] |
| 17:06 |
<XioNoX> |
rollback: disable all peering and transit on cr2-eqsin |
[production] |
| 16:57 |
<XioNoX> |
reboot cr2-eqsin |
[production] |
| 16:46 |
<XioNoX> |
disable all peering and transit on cr2-eqsin |
[production] |
| 16:25 |
<filippo@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:25 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:25 |
<filippo@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 16:25 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:07 |
<ppchelko@deploy1001> |
Finished deploy [restbase/deploy@8fca708]: Expose transform/wikitext/to/mobile-html endpoint T211026, take 2 (duration: 10m 12s) |
[production] |
| 15:56 |
<ppchelko@deploy1001> |
Started deploy [restbase/deploy@8fca708]: Expose transform/wikitext/to/mobile-html endpoint T211026, take 2 |
[production] |
| 15:56 |
<ppchelko@deploy1001> |
Finished deploy [restbase/deploy@8fca708]: Expose transform/wikitext/to/mobile-html endpoint T211026 (duration: 07m 35s) |
[production] |
| 15:49 |
<ppchelko@deploy1001> |
Started deploy [restbase/deploy@8fca708]: Expose transform/wikitext/to/mobile-html endpoint T211026 |
[production] |
| 15:46 |
<XioNoX> |
fail vrrp master to cr1-eqsin |
[production] |