|
2016-05-20
§
|
| 20:36 |
<chasemp> |
restart rabbitmq on labcontrol1001 |
[production] |
| 17:05 |
<moritzm> |
uploaded librsvg 2.40.2-1+wm2 for trusty-wikimedia to carbon (backported patches from librsvg DSA to our custom trusty build) |
[production] |
| 16:54 |
<bd808> |
Cleaned up /tmp/mw-cache-1.27.0-wmf.2* cache files on tin |
[production] |
| 16:50 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Fri May 20 16:50:25 UTC 2016 (duration 9m 38s) |
[production] |
| 16:40 |
<bd808@tin> |
scap sync-l10n completed (1.28.0-wmf.2) (duration: 10m 06s) |
[production] |
| 16:18 |
<bd808> |
kicking off manual l10nupdate run on tin |
[production] |
| 16:11 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.23) (duration: 20990m 58s) |
[production] |
| 15:52 |
<jynus> |
performing schema change on s5 T130692 |
[production] |
| 15:16 |
<godog> |
roll-restart cassandra-metrics-collector in eqiad for T135385 |
[production] |
| 15:05 |
<gehel> |
starting cluster rejoining for cassandra onmaps-test2001 |
[production] |
| 15:03 |
<godog> |
roll-restart cassandra-metrics-collector in codfw for T135385 |
[production] |
| 15:02 |
<moritzm> |
uploaded librsvg 2.40.5-1+deb8u2+wmf1 for jessie-wikimedia to carbon (rebase of locally patched package on top of latest security update) |
[production] |
| 14:50 |
<gehel> |
shutting down kartotherian on maps-test2001 (accidental data deletion) |
[production] |
| 14:22 |
<gehel> |
cassandra downgraded on maps2*.codfw.wmnet |
[production] |
| 14:04 |
<_joe_> |
removing libicu48 from trusty archives, kept a copy of the packages in my homedir on carbon |
[production] |
| 13:45 |
<mark> |
Enabled cr2-codfw et-0/* interfaces, reenabling OSPF/OSPF3 |
[production] |
| 13:38 |
<mark> |
Bringing cr2-codfw FPC 0 back up |
[production] |
| 13:37 |
<moritzm> |
upgraded java on xenon/praseodymium/cerium and restbase2001 to latest openjdk-8 release (along with restarts of Cassandra) |
[production] |
| 13:37 |
<mark> |
Offlining cr2-codfw FPC 0 |
[production] |
| 13:35 |
<jynus> |
changing dbstore1001 to be a direct slave of db1075 |
[production] |
| 13:31 |
<mark> |
Disabling cr2-codfw et-0/* interfaces |
[production] |
| 13:24 |
<mark> |
Disabling OSPF on all cr2-codfw row subnets to drain FPC0 |
[production] |
| 13:14 |
<mark> |
Lowering VRRP priority to 50 on all VRRP groups on cr2-codfw to drain FPC0 |
[production] |
| 13:07 |
<mark> |
Performing acupuncture on cr2-codfw:ae4.2020 (Lowered VRRP priority from 100 to 50, inet/inet6) |
[production] |
| 13:05 |
<jynus> |
freeing up space on db1038 by defragmenting its tables |
[production] |
| 12:31 |
<moritzm> |
rolling restart of nginx in eqiad to pick up expat update |
[production] |
| 12:22 |
<moritzm> |
rolling restart of nginx in codfw to pick up expat update |
[production] |
| 12:16 |
<elukey> |
restarting cassandra on aqs100[123] for Java upgrades |
[production] |
| 12:02 |
<moritzm> |
rolling restart of nginx in esams to pick up expat update |
[production] |
| 11:49 |
<moritzm> |
rolling restart of nginx in ulsfo to pick up expat update |
[production] |
| 10:40 |
<jynus@tin> |
Synchronized wmf-config/db-eqiad.php: Increase db1029 and db1033 weight back to normal after upgrade (duration: 01m 52s) |
[production] |
| 09:00 |
<jynus> |
altering db1040 commonswiki.categorylinks |
[production] |
| 08:50 |
<elukey> |
upgrading cassandra from 2.1.12 to 2.1.13 on aqs1003.eqiad.mwnet |
[production] |
| 08:45 |
<awight> |
Update payments fraud config |
[production] |
| 08:11 |
<elukey> |
upgrading cassandra from 2.1.12 to 2.1.13 on aqs1002.eqiad.mwnet |
[production] |
| 08:08 |
<mobrovac> |
mathoid deploying 243a530 |
[production] |
| 07:36 |
<jynus> |
testing medata lock detectiom on db1069 |
[production] |
| 06:11 |
<bblack> |
restarted gdnsd on eeden.esams (with new config, esams marked down) |
[production] |
| 05:58 |
<bblack> |
gdnsd stopped on eeden.esams, puppet disabled |
[production] |
| 05:56 |
<volans> |
Killed transaction 3262258 on db1040 (alter table stuck in "Waiting for table metadata lock" blocking the replica) T130692 |
[production] |
| 04:24 |
<moritzm> |
restarted hhvm on mw1014 (got stuck, output of hhvm-dump-debug available) |
[production] |