|
2016-05-23
§
|
| 11:24 |
<godog> |
run puppet and roll-restart cassandra-metrics-collector on restbase codfw/eqiad |
[production] |
| 11:20 |
<godog> |
deploy new version of cassandra-metrics-collector T135385 |
[production] |
| 10:57 |
<moritzm> |
restarting hhvm on app servers in codfw for librsvg update |
[production] |
| 10:32 |
<_joe_> |
running updateCollations.php --force on ptwiki, T58041 |
[production] |
| 10:24 |
<moritzm> |
rolling restart of restbase1* for openjdk-8 update |
[production] |
| 10:00 |
<moritzm> |
reverted net.netfilter.nf_conntrack_tcp_timeout_time_wait on kafka1013 back to 65 (as set by default by puppet) |
[production] |
| 08:56 |
<jynus> |
stopping, backing up and reimage db2010 T135973 |
[production] |
| 08:44 |
<moritzm> |
rolling restart of restbase2* for openjdk-8 update |
[production] |
| 08:20 |
<moritzm> |
install restarting hhvm on canary systems for librsvg update |
[production] |
| 08:01 |
<jynus> |
performing schema change on s7 T130692 |
[production] |
| 07:00 |
<_joe_> |
installed the new hhvm package on mw2017, T86096 |
[production] |
| 06:55 |
<_joe_> |
uploaded a new hhvm package for trusty linked to libicu52, T86096 |
[production] |
| 06:42 |
<elukey> |
Removed Kafka temp. override for webrequest_upload retention.ms after freeing some disk space. |
[production] |
| 06:31 |
<elukey> |
Set kafka retention.ms=172800000 for the topic webrequest_upload to free some disk space on kafka1022 |
[production] |
| 02:33 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Mon May 23 02:33:00 UTC 2016 (duration 8m 38s) |
[production] |
| 02:24 |
<mwdeploy@tin> |
scap sync-l10n completed (1.28.0-wmf.2) (duration: 10m 41s) |
[production] |
|
2016-05-20
§
|
| 20:36 |
<chasemp> |
restart rabbitmq on labcontrol1001 |
[production] |
| 17:05 |
<moritzm> |
uploaded librsvg 2.40.2-1+wm2 for trusty-wikimedia to carbon (backported patches from librsvg DSA to our custom trusty build) |
[production] |
| 16:54 |
<bd808> |
Cleaned up /tmp/mw-cache-1.27.0-wmf.2* cache files on tin |
[production] |
| 16:50 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Fri May 20 16:50:25 UTC 2016 (duration 9m 38s) |
[production] |
| 16:40 |
<bd808@tin> |
scap sync-l10n completed (1.28.0-wmf.2) (duration: 10m 06s) |
[production] |
| 16:18 |
<bd808> |
kicking off manual l10nupdate run on tin |
[production] |
| 16:11 |
<mwdeploy@tin> |
sync-l10n completed (1.27.0-wmf.23) (duration: 20990m 58s) |
[production] |
| 15:52 |
<jynus> |
performing schema change on s5 T130692 |
[production] |
| 15:16 |
<godog> |
roll-restart cassandra-metrics-collector in eqiad for T135385 |
[production] |
| 15:05 |
<gehel> |
starting cluster rejoining for cassandra onmaps-test2001 |
[production] |
| 15:03 |
<godog> |
roll-restart cassandra-metrics-collector in codfw for T135385 |
[production] |
| 15:02 |
<moritzm> |
uploaded librsvg 2.40.5-1+deb8u2+wmf1 for jessie-wikimedia to carbon (rebase of locally patched package on top of latest security update) |
[production] |
| 14:50 |
<gehel> |
shutting down kartotherian on maps-test2001 (accidental data deletion) |
[production] |
| 14:22 |
<gehel> |
cassandra downgraded on maps2*.codfw.wmnet |
[production] |
| 14:04 |
<_joe_> |
removing libicu48 from trusty archives, kept a copy of the packages in my homedir on carbon |
[production] |
| 13:45 |
<mark> |
Enabled cr2-codfw et-0/* interfaces, reenabling OSPF/OSPF3 |
[production] |
| 13:38 |
<mark> |
Bringing cr2-codfw FPC 0 back up |
[production] |
| 13:37 |
<moritzm> |
upgraded java on xenon/praseodymium/cerium and restbase2001 to latest openjdk-8 release (along with restarts of Cassandra) |
[production] |
| 13:37 |
<mark> |
Offlining cr2-codfw FPC 0 |
[production] |
| 13:35 |
<jynus> |
changing dbstore1001 to be a direct slave of db1075 |
[production] |
| 13:31 |
<mark> |
Disabling cr2-codfw et-0/* interfaces |
[production] |
| 13:24 |
<mark> |
Disabling OSPF on all cr2-codfw row subnets to drain FPC0 |
[production] |
| 13:14 |
<mark> |
Lowering VRRP priority to 50 on all VRRP groups on cr2-codfw to drain FPC0 |
[production] |
| 13:07 |
<mark> |
Performing acupuncture on cr2-codfw:ae4.2020 (Lowered VRRP priority from 100 to 50, inet/inet6) |
[production] |
| 13:05 |
<jynus> |
freeing up space on db1038 by defragmenting its tables |
[production] |