|
2019-09-12
ยง
|
| 22:43 |
<ayounsi@cumin2001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) |
[production] |
| 22:43 |
<ayounsi@cumin2001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 22:20 |
<XenoRyet> |
payments-wiki updated from 4ebbdb247d to 1f556670cf |
[production] |
| 22:14 |
<XioNoX> |
remove extra prepend in AMS-IX |
[production] |
| 21:18 |
<hashar@deploy1001> |
Synchronized php-1.34.0-wmf.22/includes/libs/rdbms/lbfactory/LBFactoryMulti.php: Hardcode posix signal and log coredump - T232613 (duration: 01m 04s) |
[production] |
| 21:17 |
<mbsantos@deploy1001> |
Finished deploy [tilerator/deploy@5996843]: Deploy tilerator 1.1.4-wmf.0 (duration: 03m 18s) |
[production] |
| 21:14 |
<mbsantos@deploy1001> |
Started deploy [tilerator/deploy@5996843]: Deploy tilerator 1.1.4-wmf.0 |
[production] |
| 21:13 |
<mbsantos@deploy1001> |
Finished deploy [kartotherian/deploy@c4c9e8b]: Deploy kartotherian 1.1.4-wmf.0 (duration: 03m 52s) |
[production] |
| 21:09 |
<mbsantos@deploy1001> |
Started deploy [kartotherian/deploy@c4c9e8b]: Deploy kartotherian 1.1.4-wmf.0 |
[production] |
| 21:00 |
<urandom> |
decommissioning Cassandra, restbase2009 -- T224553 |
[production] |
| 20:33 |
<krinkle@deploy1001> |
Synchronized wmf-config/: d495d5e24949 (duration: 01m 03s) |
[production] |
| 20:28 |
<krinkle@deploy1001> |
Synchronized multiversion/MWConfigCacheGenerator.php: d495d5e24949 (duration: 01m 04s) |
[production] |
| 20:27 |
<eileen> |
civicrm revision changed from 4075e396d5 to f00c6482bf, config revision is 635f198b92 |
[production] |
| 20:05 |
<krinkle@deploy1001> |
Synchronized wmf-config/InitialiseSettings-labs.php: beta-only (duration: 01m 02s) |
[production] |
| 20:03 |
<krinkle@deploy1001> |
Synchronized wmf-config/CommonSettings-labs.php: beta-only (duration: 01m 04s) |
[production] |
| 20:02 |
<moritzm> |
installing firmware-nonfree update from Buster 10.1 point release |
[production] |
| 19:51 |
<moritzm> |
installing systemd bugfix update from Buster 10.1 point release |
[production] |
| 19:44 |
<moritzm> |
installing 4.19.67 kernel from 10.1 point release on Buster systems |
[production] |
| 19:34 |
<urandom> |
bootstrapping Cassandra, restbase1018-c -- T224553 |
[production] |
| 18:59 |
<hashar@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Enable coredump on some mysterious php7.2 failure - T232613 (duration: 01m 04s) |
[production] |
| 18:32 |
<moritzm> |
installing gdb updates from buster 10.1 point release |
[production] |
| 18:28 |
<bblack> |
lvs1016: restart pybal to revert test |
[production] |
| 18:21 |
<bblack> |
lvs1016: restart pybal to test dual bgp peering |
[production] |
| 18:04 |
<bblack> |
lvs1015: restart pybal to return BGP session to cr2 - T226424 |
[production] |
| 18:03 |
<bblack> |
lvs1014: restart pybal to return BGP session to cr2 - T226424 |
[production] |
| 17:58 |
<XioNoX> |
revert VRRP priority change cr2-eqiad - T226424 |
[production] |
| 17:54 |
<XioNoX> |
revert OSPF priority change on cr2-eqiad - T226424 |
[production] |
| 17:53 |
<XioNoX> |
re-enabled external BGP on cr2-eqiad - T226424 |
[production] |
| 17:46 |
<urandom> |
bootstrapping Cassandra, restbase1018-b -- T224553 |
[production] |
| 17:43 |
<XioNoX> |
reboot cr2-eqiad - T226424 |
[production] |
| 17:40 |
<XioNoX> |
failover cr2-eqiad master RE from RE1 to RE0 - T226424 |
[production] |
| 17:31 |
<jforrester@deploy1001> |
Synchronized php-1.34.0-wmf.22/includes/libs/rdbms/lbfactory/LBFactoryMulti.php: T232613 Add ability to core dump on empty string array key that should exist (wmf.22 only, flagged off) (duration: 01m 03s) |
[production] |
| 17:31 |
<XioNoX> |
power off re0.cr2-eqiad - T226424 |
[production] |
| 17:24 |
<XioNoX> |
failover cr2-eqiad master RE from RE0 to RE1 - T226424 |
[production] |
| 17:19 |
<halfak@deploy1001> |
Finished deploy [ores/deploy@7d45b80]: T232660 (duration: 13m 41s) |
[production] |
| 17:05 |
<halfak@deploy1001> |
Started deploy [ores/deploy@7d45b80]: T232660 |
[production] |
| 17:04 |
<XioNoX> |
power off re1.cr2-eqiad - T226424 |
[production] |
| 17:02 |
<moritzm> |
installing unzip security updates on buster |
[production] |
| 17:00 |
<XioNoX> |
+1000 metric to all transport to/from cr2-eqiad - T226424 |
[production] |
| 16:57 |
<moritzm> |
installing libxslt security updates on buster |
[production] |
| 16:49 |
<XioNoX> |
Deactivate IX/transit/private-peer v4/v6 BGP on cr2-eqiad - T226424 |
[production] |
| 16:47 |
<moritzm> |
installing NSS security updates on buster |
[production] |
| 16:42 |
<XioNoX> |
er, switch VRRP master to cr1-eqiad - T226424 |
[production] |
| 16:42 |
<XioNoX> |
switch VRRP master to cr2-eqiad - T226424 |
[production] |
| 16:36 |
<bblack> |
lvs1013: restart pybal to move bgp session to cr1 - T226424 |
[production] |
| 16:36 |
<bblack> |
lvs1014: restart pybal to move bgp session to cr1 - T226424 |
[production] |
| 16:35 |
<bblack> |
lvs1015: restart pybal to move bgp session to cr1 - T226424 |
[production] |
| 16:34 |
<bblack> |
lvs1016: restart pybal to move bgp session to cr1 - T226424 |
[production] |
| 16:19 |
<XioNoX> |
rollback force VRRP backup on cr1-eqiad - T226424 |
[production] |
| 16:16 |
<XioNoX> |
activate CF tunnel on cr1-eqiad - T226424 |
[production] |