|
2018-05-29
ยง
|
| 16:59 |
<elukey> |
roll restart of kafka mirror maker on kafka-jumbo100* to pick up the new zookeeper settings |
[production] |
| 16:56 |
<XioNoX> |
bounced analytics1031 switchport to fix weird issue of that host not being able to receive traffic from analytics1001 |
[production] |
| 16:44 |
<elukey> |
roll restart of kafka mirror maker on kafka100[1-3] to pick up new zk settings |
[production] |
| 16:32 |
<thcipriani> |
upgrading blubber to 0.4.0 for integration machines |
[production] |
| 15:57 |
<addshore> |
really done with wb_terms related syncs now |
[production] |
| 15:52 |
<addshore@tin> |
Synchronized wmf-config/Wikibase.php: [[gerrit:435147|Revert - Dont load PropertySuggester]] T195520 (duration: 01m 19s) |
[production] |
| 15:48 |
<elukey> |
roll restart kafka on kafka-jumbo* to pick up new zookeeper settings |
[production] |
| 15:45 |
<addshore@tin> |
Synchronized php-1.32.0-wmf.5/extensions/PropertySuggester: [[gerrit:436038|Use CirrusSearch for PropertySuggester]] (duration: 01m 21s) |
[production] |
| 15:26 |
<elukey> |
restart hadoop yarn/hdfs daemons to pick up the new zookeeper settings |
[production] |
| 15:11 |
<addshore> |
Wikibase - Re enable wb_terms things window done |
[production] |
| 14:57 |
<addshore@tin> |
Synchronized php-1.32.0-wmf.5/extensions/Wikibase: [[gerrit:436007|TermSqlIndex::getMatchingTerms actually execute select]] (duration: 02m 19s) |
[production] |
| 14:57 |
<marostegui> |
Move s6 topology back to its normal status |
[production] |
| 14:49 |
<addshore@tin> |
Synchronized php-1.32.0-wmf.4/extensions/Wikibase: [[gerrit:436006|TermSqlIndex::getMatchingTerms actually execute select]] (duration: 02m 18s) |
[production] |
| 14:32 |
<addshore@tin> |
Synchronized php-1.32.0-wmf.4/extensions/Wikibase: [[gerrit:436004|Re add TermSqlIndex::getMatchingTerms select, but dont call]] (duration: 02m 18s) |
[production] |
| 14:29 |
<addshore@tin> |
Synchronized php-1.32.0-wmf.5/extensions/Wikibase: [[gerrit:436003|Re add TermSqlIndex::getMatchingTerms select, but dont call]] (duration: 02m 13s) |
[production] |
| 14:24 |
<elukey> |
roll restart kafka on kafka100[1-3] (job queues) to pick up the new zookeeper settings |
[production] |
| 14:19 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool all databases in row C - T187962 (duration: 01m 19s) |
[production] |
| 14:13 |
<addshore@tin> |
Synchronized php-1.32.0-wmf.4/extensions/Wikibase: [[gerrit:436001|track all wb_terms table access via statsd]] (duration: 02m 19s) |
[production] |
| 14:13 |
<gehel> |
comleted rolling restart of relforge for plugin upgrade - T193734 |
[production] |
| 14:10 |
<addshore@tin> |
Synchronized php-1.32.0-wmf.5/extensions/Wikibase: [[gerrit:436000|track all wb_terms table access via statsd]] (duration: 02m 21s) |
[production] |
| 14:07 |
<addshore@tin> |
Synchronized php-1.32.0-wmf.4/extensions/CentralNotice: [[gerrit:435817|Convert numerical URL parameters to numbers]] for AndyRussG (was left on tin) (duration: 01m 25s) |
[production] |
| 14:03 |
<elukey> |
swap zookeeper from conf1003 to conf1006 |
[production] |
| 13:56 |
<XioNoX> |
rolling back ns0 and ping1001 redirects - T187962 |
[production] |
| 13:47 |
<zfilipin@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:435693|Create 2 extra namespaces for bdwikimedia (T195700)]] (duration: 01m 39s) |
[production] |
| 13:42 |
<moritzm> |
upgrading remaining job runners in eqiad to hhvm-wikidiff 1.7.0 |
[production] |
| 13:39 |
<zfilipin@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:435629|Revert "Revert "Revert "Temp rate limit for arwiki due to mass vandalism""" (T192668)]] (duration: 01m 51s) |
[production] |
| 13:32 |
<volans> |
restarted ircecho |
[production] |
| 13:28 |
<volans> |
puppet run on failed hosts completed |
[production] |
| 13:25 |
<moritzm> |
powered down mw2182 for hardware diagnosis |
[production] |
| 13:19 |
<gehel> |
rolling restart of relforge for plugin upgrade - T193734 |
[production] |
| 13:16 |
<volans> |
running puppet on failed only hosts |
[production] |
| 13:12 |
<volans> |
stopped ircecho temporarily |
[production] |
| 12:54 |
<moritzm> |
installing xdg-utils security updates |
[production] |
| 11:21 |
<marostegui> |
Restar db1125 mysql - T195595 |
[production] |
| 11:14 |
<moritzm> |
upgrading snapshot hosts to hhvm-wikidiff 1.7.0 (HHVM is unused, just for completeness) |
[production] |
| 11:08 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Disable read only on s6 T194939 T187962 (duration: 01m 37s) |
[production] |
| 10:59 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Enable read only on s6 T194939 T187962 (duration: 01m 35s) |
[production] |
| 10:55 |
<XioNoX> |
Eqiad row C server move starting - T187962 |
[production] |
| 10:53 |
<XioNoX> |
Eqiad row C server move starting |
[production] |
| 10:35 |
<moritzm> |
upgrading mw1308-mw1311 to hhvm-wikidiff 1.7.0 (HHVM bytecode cache needs to be pruned during rollout) |
[production] |
| 10:09 |
<mobrovac@tin> |
Synchronized wmf-config/InitialiseSettings.php: Switch all jobs to EventBus file 2/2 - T190327 T195500 (duration: 01m 47s) |
[production] |
| 10:06 |
<mobrovac@tin> |
Synchronized wmf-config/jobqueue.php: Switch all jobs to EventBus file 1/2 - T190327 T195500 (duration: 01m 39s) |
[production] |
| 10:05 |
<ppchelko@tin> |
Finished deploy [cpjobqueue/deploy@c6dc83d]: Enable all jobs apart from exceptions for everything. T190327 (duration: 00m 58s) |
[production] |
| 10:04 |
<ppchelko@tin> |
Started deploy [cpjobqueue/deploy@c6dc83d]: Enable all jobs apart from exceptions for everything. T190327 |
[production] |
| 09:20 |
<XioNoX> |
redirect ns0 to baham - T187962 |
[production] |
| 09:16 |
<XioNoX> |
disable ping1001 redirect - T187962 |
[production] |
| 09:13 |
<marostegui> |
Downtime s6 replicas for 4 hours - T195595 |
[production] |
| 09:07 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool all databases in row C - T187962 (duration: 01m 35s) |
[production] |
| 09:05 |
<moritzm> |
upgrading labweb servers to hhvm-wikidiff 1.7.0 (HHVM bytecode cache needs to be pruned during rollout) |
[production] |
| 08:40 |
<jynus> |
performing topology changes on s6 ahead of a possible failover |
[production] |