|
2019-09-12
ยง
|
| 16:42 |
<XioNoX> |
switch VRRP master to cr2-eqiad - T226424 |
[production] |
| 16:36 |
<bblack> |
lvs1013: restart pybal to move bgp session to cr1 - T226424 |
[production] |
| 16:36 |
<bblack> |
lvs1014: restart pybal to move bgp session to cr1 - T226424 |
[production] |
| 16:35 |
<bblack> |
lvs1015: restart pybal to move bgp session to cr1 - T226424 |
[production] |
| 16:34 |
<bblack> |
lvs1016: restart pybal to move bgp session to cr1 - T226424 |
[production] |
| 16:19 |
<XioNoX> |
rollback force VRRP backup on cr1-eqiad - T226424 |
[production] |
| 16:16 |
<XioNoX> |
activate CF tunnel on cr1-eqiad - T226424 |
[production] |
| 16:15 |
<XioNoX> |
activate transit4/6 on cr1-eqiad - T226424 |
[production] |
| 16:09 |
<urandom> |
bootstrapping Cassandra, restbase1018-a -- T224553 |
[production] |
| 16:04 |
<XioNoX> |
reboot cr1-eqiad - T226424 |
[production] |
| 16:01 |
<XioNoX> |
force offline/online of FPC3 on cr1-eqiad |
[production] |
| 15:45 |
<XioNoX> |
failover master RE from RE1 to RE0 on cr1-eqiad - T226424 |
[production] |
| 15:39 |
<XioNoX> |
deactivate transit4/6 on cr1-eqiad - T226424 |
[production] |
| 15:31 |
<XioNoX> |
shutdown re0.cr1-eqiad - T226424 |
[production] |
| 15:22 |
<XioNoX> |
failover master RE from RE0 to RE1 on cr1-eqiad - T226424 |
[production] |
| 15:13 |
<XioNoX> |
shutdown re1.cr1-eqiad - T226424 |
[production] |
| 15:05 |
<XioNoX> |
disable primary tunnel to CF in eqiad (for real this time, I did see an uptake of traffic on backup link before the rollback) |
[production] |
| 15:03 |
<XioNoX> |
rolled back disable primary tunnel to CF in eqiad |
[production] |
| 15:02 |
<XioNoX> |
disable primary tunnel to CF in eqiad |
[production] |
| 14:53 |
<bblack> |
restart pybal on lvs1013 to move BGP conn to cr2-eqiad - https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/536209 - T226424 |
[production] |
| 14:50 |
<bblack> |
restart pybal on lvs1016 to move BGP conn to cr2-eqiad - https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/536209 - T226424 |
[production] |
| 14:45 |
<akosiaris@> |
helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 14:41 |
<akosiaris@> |
helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'coredns' . |
[production] |
| 14:39 |
<akosiaris@> |
helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 14:37 |
<akosiaris@> |
helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 14:29 |
<XioNoX> |
ensure cr1-eqiad is vrrp backup for all groups - T226424 |
[production] |
| 13:22 |
<akosiaris@> |
helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'coredns' . |
[production] |
| 13:03 |
<jmm@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:01 |
<jmm@cumin1001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 12:57 |
<effie> |
restarting hhvm on mw1233 and repooling |
[production] |
| 12:56 |
<effie> |
depool mw12333 |
[production] |
| 12:38 |
<moritzm> |
reimaging restbase1018 to stretch |
[production] |
| 12:03 |
<Amir1> |
EU SWAT is done |
[production] |
| 12:03 |
<ladsgroup@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:536167|Set item terms on write both up to Q20mio (T225055)]] (duration: 01m 31s) |
[production] |
| 11:11 |
<akosiaris@> |
helmfile [EQIAD] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 11:11 |
<akosiaris@> |
helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 11:09 |
<akosiaris@> |
helmfile [CODFW] Ran 'apply' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 11:09 |
<akosiaris@> |
helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 11:00 |
<akosiaris@> |
helmfile [STAGING] Ran 'apply' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 09:42 |
<jynus> |
compressing tables on labsdb1012 T232446 |
[production] |
| 08:22 |
<vgutierrez> |
upgrading to acme-chief 0.21 on acmechief-test instances - T219765 |
[production] |
| 08:17 |
<vgutierrez> |
restarting pybal on lvs1015 and lvs2003 - T176875 |
[production] |
| 08:13 |
<vgutierrez@puppetmaster1001> |
conftool action : set/pooled=yes; selector: cluster=wdqs,service=wdqs-heavy-queries |
[production] |
| 08:11 |
<vgutierrez@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=puppetmaster1001.eqiad.wmnet,service=wdqs-heavy-queries |
[production] |
| 08:07 |
<vgutierrez> |
restarting pybal on lvs2006 - T176875 |
[production] |
| 08:01 |
<vgutierrez> |
restarting pybal on lvs1016 - T176875 |
[production] |
| 07:45 |
<vgutierrez> |
uploaded acme-chief 0.21 to apt.wikimedia.org (buster) - T219765 |
[production] |
| 06:51 |
<vgutierrez> |
restarting ATS-TLS on cp4021 and cp2002 to get the new SSL session cache size - T232298 |
[production] |
| 06:00 |
<marostegui> |
Stop MySQL on db1073 for decommission T231892 |
[production] |
| 05:59 |
<marostegui> |
Remove db1073 from tendril and zarcillo T231892 |
[production] |