|
2019-09-12
ยง
|
| 17:31 |
<XioNoX> |
power off re0.cr2-eqiad - T226424 |
[production] |
| 17:24 |
<XioNoX> |
failover cr2-eqiad master RE from RE0 to RE1 - T226424 |
[production] |
| 17:19 |
<halfak@deploy1001> |
Finished deploy [ores/deploy@7d45b80]: T232660 (duration: 13m 41s) |
[production] |
| 17:05 |
<halfak@deploy1001> |
Started deploy [ores/deploy@7d45b80]: T232660 |
[production] |
| 17:04 |
<XioNoX> |
power off re1.cr2-eqiad - T226424 |
[production] |
| 17:02 |
<moritzm> |
installing unzip security updates on buster |
[production] |
| 17:00 |
<XioNoX> |
+1000 metric to all transport to/from cr2-eqiad - T226424 |
[production] |
| 16:57 |
<moritzm> |
installing libxslt security updates on buster |
[production] |
| 16:49 |
<XioNoX> |
Deactivate IX/transit/private-peer v4/v6 BGP on cr2-eqiad - T226424 |
[production] |
| 16:47 |
<moritzm> |
installing NSS security updates on buster |
[production] |
| 16:42 |
<XioNoX> |
er, switch VRRP master to cr1-eqiad - T226424 |
[production] |
| 16:42 |
<XioNoX> |
switch VRRP master to cr2-eqiad - T226424 |
[production] |
| 16:36 |
<bblack> |
lvs1013: restart pybal to move bgp session to cr1 - T226424 |
[production] |
| 16:36 |
<bblack> |
lvs1014: restart pybal to move bgp session to cr1 - T226424 |
[production] |
| 16:35 |
<bblack> |
lvs1015: restart pybal to move bgp session to cr1 - T226424 |
[production] |
| 16:34 |
<bblack> |
lvs1016: restart pybal to move bgp session to cr1 - T226424 |
[production] |
| 16:19 |
<XioNoX> |
rollback force VRRP backup on cr1-eqiad - T226424 |
[production] |
| 16:16 |
<XioNoX> |
activate CF tunnel on cr1-eqiad - T226424 |
[production] |
| 16:15 |
<XioNoX> |
activate transit4/6 on cr1-eqiad - T226424 |
[production] |
| 16:09 |
<urandom> |
bootstrapping Cassandra, restbase1018-a -- T224553 |
[production] |
| 16:04 |
<XioNoX> |
reboot cr1-eqiad - T226424 |
[production] |
| 16:01 |
<XioNoX> |
force offline/online of FPC3 on cr1-eqiad |
[production] |
| 15:45 |
<XioNoX> |
failover master RE from RE1 to RE0 on cr1-eqiad - T226424 |
[production] |
| 15:39 |
<XioNoX> |
deactivate transit4/6 on cr1-eqiad - T226424 |
[production] |
| 15:31 |
<XioNoX> |
shutdown re0.cr1-eqiad - T226424 |
[production] |
| 15:22 |
<XioNoX> |
failover master RE from RE0 to RE1 on cr1-eqiad - T226424 |
[production] |
| 15:13 |
<XioNoX> |
shutdown re1.cr1-eqiad - T226424 |
[production] |
| 15:05 |
<XioNoX> |
disable primary tunnel to CF in eqiad (for real this time, I did see an uptake of traffic on backup link before the rollback) |
[production] |
| 15:03 |
<XioNoX> |
rolled back disable primary tunnel to CF in eqiad |
[production] |
| 15:02 |
<XioNoX> |
disable primary tunnel to CF in eqiad |
[production] |
| 14:53 |
<bblack> |
restart pybal on lvs1013 to move BGP conn to cr2-eqiad - https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/536209 - T226424 |
[production] |
| 14:50 |
<bblack> |
restart pybal on lvs1016 to move BGP conn to cr2-eqiad - https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/536209 - T226424 |
[production] |
| 14:45 |
<akosiaris@> |
helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 14:41 |
<akosiaris@> |
helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'coredns' . |
[production] |
| 14:39 |
<akosiaris@> |
helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 14:37 |
<akosiaris@> |
helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 14:29 |
<XioNoX> |
ensure cr1-eqiad is vrrp backup for all groups - T226424 |
[production] |
| 13:22 |
<akosiaris@> |
helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'coredns' . |
[production] |
| 13:03 |
<jmm@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:01 |
<jmm@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:57 |
<effie> |
restarting hhvm on mw1233 and repooling |
[production] |
| 12:56 |
<effie> |
depool mw12333 |
[production] |
| 12:38 |
<moritzm> |
reimaging restbase1018 to stretch |
[production] |
| 12:03 |
<Amir1> |
EU SWAT is done |
[production] |
| 12:03 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:536167|Set item terms on write both up to Q20mio (T225055)]] (duration: 01m 31s) |
[production] |
| 11:11 |
<akosiaris@> |
helmfile [EQIAD] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 11:11 |
<akosiaris@> |
helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 11:09 |
<akosiaris@> |
helmfile [CODFW] Ran 'apply' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 11:09 |
<akosiaris@> |
helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 11:00 |
<akosiaris@> |
helmfile [STAGING] Ran 'apply' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |