|
2019-08-13
§
|
| 20:43 |
<ejegg> |
rolled back payments-wiki from 9ed8be0532 to 9533f70fab |
[production] |
| 20:34 |
<ejegg> |
updated payments-wiki from 9533f70fab to 9ed8be0532 |
[production] |
| 20:32 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/InitialiseSettings-labs.php: Fix MachineVision provider config (duration: 00m 47s) |
[production] |
| 19:48 |
<gehel@cumin2001> |
END (PASS) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=0) |
[production] |
| 19:23 |
<ppchelko@deploy1001> |
Finished deploy [cpjobqueue/deploy@3882ddb]: Increase cirrusSearchLinksUpdatePrioritized concurrency 150 -> 200 T220625 (duration: 00m 58s) |
[production] |
| 19:22 |
<ppchelko@deploy1001> |
Started deploy [cpjobqueue/deploy@3882ddb]: Increase cirrusSearchLinksUpdatePrioritized concurrency 150 -> 200 T220625 |
[production] |
| 19:19 |
<ppchelko@deploy1001> |
deploy aborted: Revert on canary (duration: 00m 18s) |
[production] |
| 19:18 |
<ppchelko@deploy1001> |
Started deploy [cpjobqueue/deploy@f1a562e]: Revert on canary |
[production] |
| 19:17 |
<ppchelko@deploy1001> |
deploy aborted: Increase cirrusSearchLinksUpdatePrioritized concurrency 150 -> 200 T220625 (duration: 01m 30s) |
[production] |
| 19:15 |
<ppchelko@deploy1001> |
Started deploy [cpjobqueue/deploy@f1a562e]: Increase cirrusSearchLinksUpdatePrioritized concurrency 150 -> 200 T220625 |
[production] |
| 19:03 |
<@> |
helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' . |
[production] |
| 18:50 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.rolling-reboot |
[production] |
| 18:41 |
<ebernhardson> |
set cpufreq scaling_governor to performance on cloudelastic100[1-4] to test any changes to indexing performance |
[production] |
| 18:38 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Enable MachineVision on Beta (4/4) (duration: 00m 48s) |
[production] |
| 18:34 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/InitialiseSettings-labs.php: Enable MachineVision on Beta (3/4) (duration: 00m 47s) |
[production] |
| 18:33 |
<crusnov@deploy1001> |
Finished deploy [netbox/deploy@367ca84]: Update Netbox to v2.6.1-wmf3 affects: T223292 (fix perms) (duration: 00m 09s) |
[production] |
| 18:33 |
<crusnov@deploy1001> |
Started deploy [netbox/deploy@367ca84]: Update Netbox to v2.6.1-wmf3 affects: T223292 (fix perms) |
[production] |
| 18:33 |
<crusnov@deploy1001> |
Finished deploy [netbox/deploy@367ca84]: Update Netbox to v2.6.1-wmf3 affects: T223292 (duration: 00m 43s) |
[production] |
| 18:32 |
<crusnov@deploy1001> |
Started deploy [netbox/deploy@367ca84]: Update Netbox to v2.6.1-wmf3 affects: T223292 |
[production] |
| 18:32 |
<crusnov@deploy1001> |
Finished deploy [netbox/deploy@367ca84]: Update Netbox to v2.6.1-wmf3 affects: T223292 (duration: 00m 36s) |
[production] |
| 18:31 |
<crusnov@deploy1001> |
Started deploy [netbox/deploy@367ca84]: Update Netbox to v2.6.1-wmf3 affects: T223292 |
[production] |
| 18:30 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Enable MachineVision on Beta (2/4) (duration: 00m 48s) |
[production] |
| 18:27 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/extension-list: Enable MachineVision on Beta (1/4) (duration: 00m 48s) |
[production] |
| 17:44 |
<XioNoX> |
set target netflow port to 2000 in eqiad |
[production] |
| 17:11 |
<XioNoX> |
repool eqsin |
[production] |
| 17:06 |
<XioNoX> |
rollback: disable all peering and transit on cr2-eqsin |
[production] |
| 16:57 |
<XioNoX> |
reboot cr2-eqsin |
[production] |
| 16:46 |
<XioNoX> |
disable all peering and transit on cr2-eqsin |
[production] |
| 16:25 |
<filippo@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:25 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:25 |
<filippo@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 16:25 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:07 |
<ppchelko@deploy1001> |
Finished deploy [restbase/deploy@8fca708]: Expose transform/wikitext/to/mobile-html endpoint T211026, take 2 (duration: 10m 12s) |
[production] |
| 15:56 |
<ppchelko@deploy1001> |
Started deploy [restbase/deploy@8fca708]: Expose transform/wikitext/to/mobile-html endpoint T211026, take 2 |
[production] |
| 15:56 |
<ppchelko@deploy1001> |
Finished deploy [restbase/deploy@8fca708]: Expose transform/wikitext/to/mobile-html endpoint T211026 (duration: 07m 35s) |
[production] |
| 15:49 |
<ppchelko@deploy1001> |
Started deploy [restbase/deploy@8fca708]: Expose transform/wikitext/to/mobile-html endpoint T211026 |
[production] |
| 15:46 |
<XioNoX> |
fail vrrp master to cr1-eqsin |
[production] |
| 15:42 |
<gehel@cumin2001> |
END (FAIL) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=99) |
[production] |
| 15:39 |
<bblack> |
puppet re-enabled on lvs1014, lvs1016, icinga1001 |
[production] |
| 15:35 |
<XioNoX> |
depool eqsin for cr2-eqsin upgrade |
[production] |
| 15:32 |
<bblack> |
disabled pupped on lvs1014, lvs1016, icinga1001 ahead of deploying https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/528885/ - T229621 |
[production] |
| 15:32 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.rolling-reboot |
[production] |
| 15:30 |
<XioNoX> |
rollback ospf + bgp changes on cr2-eqord |
[production] |
| 15:19 |
<XioNoX> |
restart cr2-eqord - T227886 |
[production] |
| 15:12 |
<XioNoX> |
disable all peering and transit on cr2-eqord |
[production] |
| 15:01 |
<XioNoX> |
increase ospf cost of cr2-eqord<->cr2-eqiad link (+1000) |
[production] |
| 14:57 |
<ema> |
cp5002: reboot for kernel upgrade |
[production] |
| 14:42 |
<gehel@cumin2001> |
END (FAIL) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=99) |
[production] |