|
2019-09-13
§
|
| 08:52 |
<godog> |
downtime kartotherian pages for 1h |
[production] |
| 08:48 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0) |
[production] |
| 08:48 |
<jmm@cumin2001> |
Updating IPMI password on 1 hosts - jmm@cumin2001 |
[production] |
| 08:47 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.ipmi-password-reset |
[production] |
| 08:47 |
<jmm@cumin2001> |
END (FAIL) - Cookbook sre.hosts.ipmi-password-reset (exit_code=99) |
[production] |
| 08:47 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.ipmi-password-reset |
[production] |
| 08:45 |
<gehel> |
stop tilerator on maps to help reduce load |
[production] |
| 08:37 |
<_joe_> |
rolling restart of karotherian |
[production] |
| 08:33 |
<_joe_> |
restarting kartotherian on maps1003, all workers seem stuck |
[production] |
| 05:58 |
<oblivian@deploy1001> |
Synchronized w/fatal-error.php: Adding core dump function to fatal-error (duration: 01m 04s) |
[production] |
| 05:40 |
<_joe_> |
live-hacking mw1348, setting rlimit_core = unlimited to allow core dumps to be taken |
[production] |
| 05:17 |
<effie> |
Rolling restart php-fpm across the fleet for 536400 |
[production] |
| 04:53 |
<vgutierrez> |
restarting ats-tls on cp4021 and cp2002 to pick up the new SSL session cache timeout - T231849 |
[production] |
| 04:50 |
<eileen> |
process-control config revision is 43a2677bcf - turned off gender import |
[production] |
| 02:23 |
<eileen> |
civicrm revision changed from c5ab5aea9e to 45dbfdb96f, config revision is 1da8391a9a |
[production] |
| 01:09 |
<XioNoX> |
add IPv6 sampling to cr1-eqiad |
[production] |
| 01:07 |
<XioNoX> |
enable netflow sampling on cr2-codfw |
[production] |
|
2019-09-12
§
|
| 23:35 |
<XioNoX> |
enable netflow sampling on cr1-codfw |
[production] |
| 23:21 |
<urandom> |
decommissioning Cassandra, restbase2009-b -- T224553 |
[production] |
| 23:19 |
<jforrester@deploy1001> |
Synchronized wmf-config/CommonSettings.php: T223602 Read config from JSON, not serialised PHP on testwiki (duration: 01m 03s) |
[production] |
| 23:18 |
<jforrester@deploy1001> |
Synchronized multiversion/MWConfigCacheGenerator.php: T223602 Add ability to read config from JSON, not serialised PHP (duration: 01m 04s) |
[production] |
| 23:10 |
<eileen> |
process-control config revision is 1da8391a9a |
[production] |
| 22:53 |
<ayounsi@cumin2001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 22:48 |
<ayounsi@cumin2001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 22:43 |
<ayounsi@cumin2001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) |
[production] |
| 22:43 |
<ayounsi@cumin2001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 22:20 |
<XenoRyet> |
payments-wiki updated from 4ebbdb247d to 1f556670cf |
[production] |
| 22:14 |
<XioNoX> |
remove extra prepend in AMS-IX |
[production] |
| 21:18 |
<hashar@deploy1001> |
Synchronized php-1.34.0-wmf.22/includes/libs/rdbms/lbfactory/LBFactoryMulti.php: Hardcode posix signal and log coredump - T232613 (duration: 01m 04s) |
[production] |
| 21:17 |
<mbsantos@deploy1001> |
Finished deploy [tilerator/deploy@5996843]: Deploy tilerator 1.1.4-wmf.0 (duration: 03m 18s) |
[production] |
| 21:14 |
<mbsantos@deploy1001> |
Started deploy [tilerator/deploy@5996843]: Deploy tilerator 1.1.4-wmf.0 |
[production] |
| 21:13 |
<mbsantos@deploy1001> |
Finished deploy [kartotherian/deploy@c4c9e8b]: Deploy kartotherian 1.1.4-wmf.0 (duration: 03m 52s) |
[production] |
| 21:09 |
<mbsantos@deploy1001> |
Started deploy [kartotherian/deploy@c4c9e8b]: Deploy kartotherian 1.1.4-wmf.0 |
[production] |
| 21:00 |
<urandom> |
decommissioning Cassandra, restbase2009 -- T224553 |
[production] |
| 20:33 |
<krinkle@deploy1001> |
Synchronized wmf-config/: d495d5e24949 (duration: 01m 03s) |
[production] |
| 20:28 |
<krinkle@deploy1001> |
Synchronized multiversion/MWConfigCacheGenerator.php: d495d5e24949 (duration: 01m 04s) |
[production] |
| 20:27 |
<eileen> |
civicrm revision changed from 4075e396d5 to f00c6482bf, config revision is 635f198b92 |
[production] |
| 20:05 |
<krinkle@deploy1001> |
Synchronized wmf-config/InitialiseSettings-labs.php: beta-only (duration: 01m 02s) |
[production] |
| 20:03 |
<krinkle@deploy1001> |
Synchronized wmf-config/CommonSettings-labs.php: beta-only (duration: 01m 04s) |
[production] |
| 20:02 |
<moritzm> |
installing firmware-nonfree update from Buster 10.1 point release |
[production] |
| 19:51 |
<moritzm> |
installing systemd bugfix update from Buster 10.1 point release |
[production] |
| 19:44 |
<moritzm> |
installing 4.19.67 kernel from 10.1 point release on Buster systems |
[production] |
| 19:34 |
<urandom> |
bootstrapping Cassandra, restbase1018-c -- T224553 |
[production] |
| 18:59 |
<hashar@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Enable coredump on some mysterious php7.2 failure - T232613 (duration: 01m 04s) |
[production] |
| 18:32 |
<moritzm> |
installing gdb updates from buster 10.1 point release |
[production] |
| 18:28 |
<bblack> |
lvs1016: restart pybal to revert test |
[production] |
| 18:21 |
<bblack> |
lvs1016: restart pybal to test dual bgp peering |
[production] |
| 18:04 |
<bblack> |
lvs1015: restart pybal to return BGP session to cr2 - T226424 |
[production] |
| 18:03 |
<bblack> |
lvs1014: restart pybal to return BGP session to cr2 - T226424 |
[production] |
| 17:58 |
<XioNoX> |
revert VRRP priority change cr2-eqiad - T226424 |
[production] |