|
2019-08-21
ยง
|
| 18:04 |
<XioNoX> |
increase OSPF cost on cr2-codfw links - T226422 |
[production] |
| 17:56 |
<XioNoX> |
rollback: apply BGP graceful shutdown to cr1-codfw transits - T226422 |
[production] |
| 17:55 |
<XioNoX> |
Rollback: increase OSPF cost on ulsfo-codfw link - T226422 |
[production] |
| 17:53 |
<XioNoX> |
rollback: disable BGP from cr1-codfw to lvs2001/2/3 - T226422 |
[production] |
| 17:43 |
<XioNoX> |
restart both REs on cr1-codfw - T226422 |
[production] |
| 17:33 |
<XioNoX> |
failover master RE to RE0 on cr1-codfw - T226422 |
[production] |
| 17:33 |
<cmjohnson1> |
cloudvirt1015 down for a new motherboard |
[production] |
| 17:25 |
<XioNoX> |
shutdown RE0 on cr1-codfw - T226422 |
[production] |
| 17:17 |
<bstorm_> |
reboot cloudvirt1024 to try and reset raid T230289 |
[production] |
| 17:17 |
<XioNoX> |
failover master RE to RE1 on cr1-codfw - T226422 |
[production] |
| 17:08 |
<XioNoX> |
disable BGP from cr1-codfw to lvs2001/2/3 - T226422 |
[production] |
| 17:02 |
<cmjohnson1> |
rebooting cloudvirt1024 |
[production] |
| 17:00 |
<tarrow> |
continuing the SWAT window to backport train blocker fixes |
[production] |
| 16:56 |
<XioNoX> |
Varnish: redirect eqsin/ulsfo text to eqiad - T226422 |
[production] |
| 16:51 |
<XioNoX> |
increase OSPF cost on ulsfo-codfw link - T226422 |
[production] |
| 16:46 |
<XioNoX> |
apply BGP graceful shutdown to cr1-codfw transits - T226422 |
[production] |
| 16:37 |
<XioNoX> |
depool eqsin and codfw - T226422 |
[production] |
| 16:01 |
<moritzm> |
fixed apt config on krypton, broken getenvoy-jessie.list made apt-get update fail |
[production] |
| 15:16 |
<elukey@deploy1001> |
Finished deploy [analytics/superset/deploy@UNKNOWN]: Rollback to 0.32 (duration: 00m 25s) |
[production] |
| 15:15 |
<elukey@deploy1001> |
Started deploy [analytics/superset/deploy@UNKNOWN]: Rollback to 0.32 |
[production] |
| 15:07 |
<moritzm> |
installing python-cryptography update from Stretch point release |
[production] |
| 15:00 |
<jbond42> |
adding interface::add_ip6_mapped to media wiki servers |
[production] |
| 14:46 |
<elukey@deploy1001> |
Finished deploy [analytics/superset/deploy@868635a]: Upgrading superset to 0.34rc1 (duration: 00m 33s) |
[production] |
| 14:46 |
<elukey@deploy1001> |
Started deploy [analytics/superset/deploy@868635a]: Upgrading superset to 0.34rc1 |
[production] |
| 14:42 |
<moritzm> |
installing java-common update from Stretch point release |
[production] |
| 14:36 |
<moritzm> |
installing dns-root-data update from Stretch point release |
[production] |
| 14:29 |
<godog> |
silence average mw appserver latency alerts for 24h, too noisy |
[production] |
| 14:28 |
<elukey> |
swap turnilo backend in varnish from analytics-tool1002 to an-tool1007 |
[production] |
| 14:27 |
<moritzm> |
installing ca-certificates-java update from Stretch point release |
[production] |
| 14:10 |
<marostegui> |
Upgrade mysql on db2075 |
[production] |
| 13:12 |
<zfilipin@deploy1001> |
Synchronized php: group1 wikis to 1.34.0-wmf.19 (duration: 00m 55s) |
[production] |
| 13:11 |
<zfilipin@deploy1001> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.34.0-wmf.19 |
[production] |
| 11:59 |
<jbond42> |
add ipv6 mapped address to mw codfw servers |
[production] |
| 11:41 |
<Amir1> |
EU SWAT is done |
[production] |
| 11:38 |
<jijiki> |
Restarting ores on ores1004 and ores1005 |
[production] |
| 11:37 |
<elukey> |
restart celery-ores-worker on ores1002 |
[production] |
| 10:57 |
<Urbanecm> |
Run scap pull on mwdebug1002 (T230601) |
[production] |
| 10:52 |
<Urbanecm> |
Move 0a87e3c's code to abusefilter.php on mwdebug1002 (T230601) |
[production] |
| 10:49 |
<Urbanecm> |
Previous log entry was for mwdebug1002 |
[production] |
| 10:49 |
<Urbanecm> |
Wrapped code added to CommonSettings.php in T230601 to wgExtensionFunctions |
[production] |
| 10:45 |
<Urbanecm> |
Run mwscript namespaceDupes.php --wiki=zhwikisource --add-prefix=FIXME --fix (T230548) |
[production] |
| 10:02 |
<moritzm> |
installing puppetdb1002 |
[production] |
| 09:46 |
<tarrow> |
finished enabling termbox on wikidatawiki |
[production] |
| 09:36 |
<tarrow@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:531433|Enable Termbox on wikidatawiki (T230896)]] (duration: 00m 55s) |
[production] |
| 09:29 |
<moritzm> |
rebooting db2102 (reverting to a proper stretch 4.9 kernel, it used a bpo kernel due to some hardware debuging a while back) |
[production] |
| 09:20 |
<@> |
helmfile [EQIAD] Ran 'apply' command on namespace 'termbox' for release 'production' . |
[production] |
| 09:15 |
<@> |
helmfile [CODFW] Ran 'apply' command on namespace 'termbox' for release 'production' . |
[production] |
| 09:09 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 09:09 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 09:09 |
<@> |
helmfile [STAGING] Ran 'apply' command on namespace 'termbox' for release 'staging' . |
[production] |