|
2016-03-03
ยง
|
| 20:20 |
<urandom> |
rolling restart of restbase production to apply config change : T127387 |
[production] |
| 20:03 |
<urandom> |
rolling restart of restbase staging complete : T127387 |
[production] |
| 19:58 |
<urandom> |
rolling restart of restbase in staging to apply config change : T127387 |
[production] |
| 19:44 |
<gehel> |
elastic1031.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101) |
[production] |
| 19:27 |
<volans> |
Completed migration of data from es200[68] to es201[579], added es201[579] to tendril. T127330 |
[production] |
| 19:08 |
<csteipp@tin> |
Synchronized wmf-config/CommonSettings-labs.php: (no message) (duration: 00m 40s) |
[production] |
| 19:05 |
<chasemp> |
create temp 3T lv on labstore2001 to store test backup deltas |
[production] |
| 18:41 |
<gehel> |
elastic1030.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101) |
[production] |
| 18:14 |
<urandom> |
lowering outbound stream throughput limit on restbase1010-a.eqiad.wmnet to 25mbps : T128107 T95253 |
[production] |
| 17:58 |
<urandom> |
increasing stream throughput for restbase1010-b.eqiad.wmnet boostrap by 25mbps (5x5) : T128107 T95253 |
[production] |
| 17:09 |
<gehel> |
elastic1029.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101) |
[production] |
| 16:25 |
<gehel> |
elastic1028.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101) |
[production] |
| 16:25 |
<Krinkle> |
krinkle@terbium mwscript deleteEqualMessages.php --wiki frwiki |
[production] |
| 16:18 |
<godog> |
restbase1001 bump stream throughput to 60mbps on restbase1001 |
[production] |
| 16:10 |
<jynus> |
downtime on all mariadb replication lag checks in preparation to changing its check |
[production] |
| 16:09 |
<jzerebecki@tin> |
Synchronized w/static/images/project-logos/wikitech.png: Change the wikitech favicon and logo to the actual wikitech logo a29196d359b9924719b9166dca98a474ad9a6a2b 2 of 2 (duration: 00m 29s) |
[production] |
| 16:08 |
<jzerebecki@tin> |
Synchronized w/static/favicon/wikitech.ico: Change the wikitech favicon and logo to the actual wikitech logo a29196d359b9924719b9166dca98a474ad9a6a2b 1 of 2 (duration: 00m 30s) |
[production] |
| 16:06 |
<jzerebecki@tin> |
Synchronized wmf-config/CirrusSearch-production.php: CirrusSearch: Enable popqual (quality+pageviews) scoring method for the completion suggester T127943 (duration: 00m 37s) |
[production] |
| 15:42 |
<gehel> |
elastic1027.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101) |
[production] |
| 15:12 |
<godog> |
cassandra throttle 1001, 1002, 1007-a, 1007-b, and 1010-a to 30mbps T95253 |
[production] |
| 15:03 |
<moritzm> |
uploaded openssl 1.0.2g for jessie-wikimedia |
[production] |
| 14:34 |
<gehel> |
elastic1026.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101) |
[production] |
| 13:47 |
<gehel> |
elastic1025.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101) |
[production] |
| 12:48 |
<gehel> |
elastic1024.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101) |
[production] |
| 12:22 |
<godog> |
temporary repool ms-fe1004, apply https://gerrit.wikimedia.org/r/#/c/273431 to test T128081 |
[production] |
| 12:12 |
<elukey@tin> |
Synchronized wmf-config/jobqueue-eqiad.php: Revert - Remove rdb1003 from the Redis JobQueue pool for maintenance (duration: 00m 28s) |
[production] |
| 12:07 |
<elukey@tin> |
Synchronized wmf-config/jobqueue-eqiad.php: Remove rdb1003 from the Redis JobQueue pool for maintenance (duration: 00m 32s) |
[production] |
| 12:02 |
<gehel> |
elastic1023.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101) |
[production] |
| 11:55 |
<_joe_> |
disabled notifications from the redises IPSEC checks while replication is disabled |
[production] |
| 11:52 |
<volans> |
Migrating data es2006->es2015 and es2008->es2017->es2019 T127330 |
[production] |
| 11:40 |
<moritzm> |
upgrading cp1008 to openssl 1.0.2g |
[production] |
| 11:21 |
<volans@tin> |
Synchronized wmf-config/db-codfw.php: Depool es2005,es2008 to migrate data to es2015,es2017 T127330 (duration: 00m 53s) |
[production] |
| 10:54 |
<godog> |
replicate swift unsharded -deleted containers eqiad -> codfw T128096 |
[production] |
| 10:48 |
<volans> |
Changing local replica topology for shard es3 in codfw for T127330 |
[production] |
| 10:36 |
<volans> |
Changing local replica topology for shard es2 in codfw for T127330 |
[production] |
| 10:21 |
<_joe_> |
rolling restart of strongswan on eqiad failing servers |
[production] |
| 10:17 |
<_joe_> |
restarted strongswan on mc1011 |
[production] |
| 10:07 |
<gehel> |
elastic1022.eqiad.wmnet: upgrading to 1.7.5, shipping logs to logstash (T122697, T109101) |
[production] |
| 09:57 |
<volans> |
Added es2014,es2016,es2018 to tendril [ T127330 ] |
[production] |
| 09:46 |
<jynus> |
schema change finished on all hosts (except delayed slaves) |
[production] |
| 09:21 |
<_joe_> |
puppet re-enabled everywhere, now troubleshooting ipsec issues |
[production] |
| 08:59 |
<moritzm> |
repooled scb1002 |
[production] |
| 08:49 |
<moritzm> |
repooled scb1001, depooling scb1002 for nodejs upgrade |
[production] |
| 08:35 |
<_joe_> |
disabled puppet across the main redises fleet in order to merge https://gerrit.wikimedia.org/r/271261 safely |
[production] |
| 08:33 |
<moritzm> |
depooling scb1001 for nodejs upgrade |
[production] |
| 08:27 |
<jynus> |
altering heartbeat table on all production servers |
[production] |
| 06:39 |
<ebernhardson> |
upgrade elastic1021.eqiad.wmnet to elasticsearch 1.7.5 |
[production] |
| 05:47 |
<ebernhardson> |
upgrade elastic1020.eqiad.wmnet to elasticsearch 1.7.5 |
[production] |
| 05:02 |
<ebernhardson> |
upgrade elastic1019.eqiad.wmnet to elasticseach 1.7.5 |
[production] |
| 04:05 |
<bblack> |
disabling puppet on caches for a bit, JIC |
[production] |