|
2019-08-13
ยง
|
| 19:15 |
<ppchelko@deploy1001> |
Started deploy [cpjobqueue/deploy@f1a562e]: Increase cirrusSearchLinksUpdatePrioritized concurrency 150 -> 200 T220625 |
[production] |
| 19:03 |
<@> |
helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' . |
[production] |
| 18:50 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.rolling-reboot |
[production] |
| 18:41 |
<ebernhardson> |
set cpufreq scaling_governor to performance on cloudelastic100[1-4] to test any changes to indexing performance |
[production] |
| 18:38 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Enable MachineVision on Beta (4/4) (duration: 00m 48s) |
[production] |
| 18:34 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/InitialiseSettings-labs.php: Enable MachineVision on Beta (3/4) (duration: 00m 47s) |
[production] |
| 18:33 |
<crusnov@deploy1001> |
Finished deploy [netbox/deploy@367ca84]: Update Netbox to v2.6.1-wmf3 affects: T223292 (fix perms) (duration: 00m 09s) |
[production] |
| 18:33 |
<crusnov@deploy1001> |
Started deploy [netbox/deploy@367ca84]: Update Netbox to v2.6.1-wmf3 affects: T223292 (fix perms) |
[production] |
| 18:33 |
<crusnov@deploy1001> |
Finished deploy [netbox/deploy@367ca84]: Update Netbox to v2.6.1-wmf3 affects: T223292 (duration: 00m 43s) |
[production] |
| 18:32 |
<crusnov@deploy1001> |
Started deploy [netbox/deploy@367ca84]: Update Netbox to v2.6.1-wmf3 affects: T223292 |
[production] |
| 18:32 |
<crusnov@deploy1001> |
Finished deploy [netbox/deploy@367ca84]: Update Netbox to v2.6.1-wmf3 affects: T223292 (duration: 00m 36s) |
[production] |
| 18:31 |
<crusnov@deploy1001> |
Started deploy [netbox/deploy@367ca84]: Update Netbox to v2.6.1-wmf3 affects: T223292 |
[production] |
| 18:30 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Enable MachineVision on Beta (2/4) (duration: 00m 48s) |
[production] |
| 18:27 |
<mholloway-shell@deploy1001> |
Synchronized wmf-config/extension-list: Enable MachineVision on Beta (1/4) (duration: 00m 48s) |
[production] |
| 17:44 |
<XioNoX> |
set target netflow port to 2000 in eqiad |
[production] |
| 17:11 |
<XioNoX> |
repool eqsin |
[production] |
| 17:06 |
<XioNoX> |
rollback: disable all peering and transit on cr2-eqsin |
[production] |
| 16:57 |
<XioNoX> |
reboot cr2-eqsin |
[production] |
| 16:46 |
<XioNoX> |
disable all peering and transit on cr2-eqsin |
[production] |
| 16:25 |
<filippo@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 16:25 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:25 |
<filippo@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) |
[production] |
| 16:25 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 16:07 |
<ppchelko@deploy1001> |
Finished deploy [restbase/deploy@8fca708]: Expose transform/wikitext/to/mobile-html endpoint T211026, take 2 (duration: 10m 12s) |
[production] |
| 15:56 |
<ppchelko@deploy1001> |
Started deploy [restbase/deploy@8fca708]: Expose transform/wikitext/to/mobile-html endpoint T211026, take 2 |
[production] |
| 15:56 |
<ppchelko@deploy1001> |
Finished deploy [restbase/deploy@8fca708]: Expose transform/wikitext/to/mobile-html endpoint T211026 (duration: 07m 35s) |
[production] |
| 15:49 |
<ppchelko@deploy1001> |
Started deploy [restbase/deploy@8fca708]: Expose transform/wikitext/to/mobile-html endpoint T211026 |
[production] |
| 15:46 |
<XioNoX> |
fail vrrp master to cr1-eqsin |
[production] |
| 15:42 |
<gehel@cumin2001> |
END (FAIL) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=99) |
[production] |
| 15:39 |
<bblack> |
puppet re-enabled on lvs1014, lvs1016, icinga1001 |
[production] |
| 15:35 |
<XioNoX> |
depool eqsin for cr2-eqsin upgrade |
[production] |
| 15:32 |
<bblack> |
disabled pupped on lvs1014, lvs1016, icinga1001 ahead of deploying https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/528885/ - T229621 |
[production] |
| 15:32 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.rolling-reboot |
[production] |
| 15:30 |
<XioNoX> |
rollback ospf + bgp changes on cr2-eqord |
[production] |
| 15:19 |
<XioNoX> |
restart cr2-eqord - T227886 |
[production] |
| 15:12 |
<XioNoX> |
disable all peering and transit on cr2-eqord |
[production] |
| 15:01 |
<XioNoX> |
increase ospf cost of cr2-eqord<->cr2-eqiad link (+1000) |
[production] |
| 14:57 |
<ema> |
cp5002: reboot for kernel upgrade |
[production] |
| 14:42 |
<gehel@cumin2001> |
END (FAIL) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=99) |
[production] |
| 14:42 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.rolling-reboot |
[production] |
| 14:31 |
<gehel@cumin2001> |
END (FAIL) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=99) |
[production] |
| 14:31 |
<gehel@cumin2001> |
START - Cookbook sre.elasticsearch.rolling-reboot |
[production] |
| 14:29 |
<XioNoX> |
rollback: disable all peering and transit on cr2-eqdfw |
[production] |
| 14:18 |
<XioNoX> |
reboot cr2-eqdfw for software upgrade - T227886 |
[production] |
| 14:14 |
<XioNoX> |
disable all peering and transit on cr2-eqdfw |
[production] |
| 14:04 |
<volans@cumin2001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 14:04 |
<volans@cumin2001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 13:20 |
<jbond42> |
rolling update of postgresql-9.6 |
[production] |
| 13:07 |
<jijiki> |
rolling restart hhvm on api servers in eqiad |
[production] |
| 12:57 |
<jijiki> |
Restart hhvm on mw1235 |
[production] |