|
2017-10-26
ยง
|
| 22:39 |
<bblack> |
restarting varnish-backend on cp1053 (mailbox lag from ongoing issues elsewhere?) |
[production] |
| 21:40 |
<bblack> |
raising backend max_connections for api.svc.eqiad.wmnet + appservers.svc.eqiad.wmnet from 1K to 10K on cp1053.eqiad.wmnet (current funnel for the bulk of the 503s) |
[production] |
| 21:32 |
<hoo@tin> |
Synchronized wmf-config/InitialiseSettings.php: Temporary disable remex html (T178632) (duration: 00m 50s) |
[production] |
| 21:30 |
<hoo@tin> |
Synchronized wmf-config/InitialiseSettings.php: Temporary disable remex html (T178632) (duration: 00m 50s) |
[production] |
| 21:00 |
<hoo> |
Fully revert all changes related to T178180 |
[production] |
| 20:58 |
<hoo@tin> |
Synchronized wmf-config/Wikibase.php: Revert "Add property for RDF mapping of external identifiers for Wikidata" (T178180) (duration: 00m 50s) |
[production] |
| 20:02 |
<ladsgroup@tin> |
Synchronized wmf-config/InitialiseSettings.php: UBN! disbale ores for wikidata (T179107) (duration: 00m 50s) |
[production] |
| 20:00 |
<ladsgroup@tin> |
Synchronized wmf-config/InitialiseSettings.php: UBN! disbale ores for wikidata (T179107) (duration: 00m 50s) |
[production] |
| 19:41 |
<awight@tin> |
Finished deploy [ores/deploy@0adae70]: Increase extractor wikidata API timeout to 15s, T179107 (duration: 07m 25s) |
[production] |
| 19:36 |
<aaron@tin> |
Started restart [jobrunner/jobrunner@a20d043]: (no justification provided) |
[production] |
| 19:34 |
<awight@tin> |
Started deploy [ores/deploy@0adae70]: Increase extractor wikidata API timeout to 15s, T179107 |
[production] |
| 19:33 |
<awight@tin> |
Finished deploy [ores/deploy@0adae70]: Increase extractor wikidata API timeout to 15s, T179107 (duration: 00m 10s) |
[production] |
| 19:33 |
<awight@tin> |
Started deploy [ores/deploy@0adae70]: Increase extractor wikidata API timeout to 15s, T179107 |
[production] |
| 19:27 |
<demon@tin> |
Synchronized php-1.31.0-wmf.5/includes/libs/filebackend/SwiftFileBackend.php: Better error grouping (duration: 00m 50s) |
[production] |
| 19:14 |
<ladsgroup@tin> |
Synchronized php-1.31.0-wmf.5/extensions/WikibaseQualityConstraints/includes/ConstraintCheck/DelegatingConstraintChecker.php: Fix sorting of NullResults (T179038) (duration: 00m 49s) |
[production] |
| 19:12 |
<ladsgroup@tin> |
Synchronized php-1.31.0-wmf.5/extensions/WikibaseQualityConstraints/tests/phpunit/DelegatingConstraintCheckerTest.php: Fix sorting of NullResults (T179038) (duration: 00m 50s) |
[production] |
| 18:52 |
<ladsgroup@tin> |
Synchronized php-1.31.0-wmf.5/extensions/Wikidata/extensions/Constraints/tests/phpunit/DelegatingConstraintCheckerTest.php: Fix sorting of NullResults (T179038) (duration: 00m 49s) |
[production] |
| 18:51 |
<ladsgroup@tin> |
Synchronized php-1.31.0-wmf.5/extensions/Wikidata/extensions/Constraints/includes/ConstraintCheck/DelegatingConstraintChecker.php: Fix sorting of NullResults (T179038) (duration: 01m 04s) |
[production] |
| 18:41 |
<bblack> |
eqiad cp servers: rolling quick depool -> repool around ethtool parameter changes for -lro,-pause |
[production] |
| 18:30 |
<urandom> |
Dropping and recreating keyspace enwiki_T_page__summary (restbase-ng cluster) |
[production] |
| 18:27 |
<bblack> |
esams cp servers: rolling quick depool -> repool around ethtool parameter changes for -lro,-pause |
[production] |
| 18:26 |
<urandom> |
T179105: Altering "enwiki_T_page__summary".data to use LZ4Compressor |
[production] |
| 17:59 |
<bblack> |
codfw cp servers: rolling quick depool -> repool around ethtool parameter changes for -lro,-pause |
[production] |
| 17:57 |
<bblack@neodymium> |
conftool action : set/pooled=no; selector: name=cp4024.ulsfo.wmnet |
[production] |
| 17:49 |
<bblack> |
ulsfo cp servers: rolling quick depool -> repool around ethtool parameter changes for -lro,-pause |
[production] |
| 17:47 |
<awight@tin> |
Finished deploy [ores/deploy@f5deb7f]: Revscoring 2 on ores1002 (non-production) (duration: 01m 45s) |
[production] |
| 17:45 |
<awight@tin> |
Started deploy [ores/deploy@f5deb7f]: Revscoring 2 on ores1002 (non-production) |
[production] |
| 17:36 |
<arlolra> |
Updated Parsoid to 8e99708a (T176728) |
[production] |
| 17:30 |
<arlolra@tin> |
Finished deploy [parsoid/deploy@4882c59]: Updating Parsoid to 8e99708a (duration: 08m 40s) |
[production] |
| 17:21 |
<arlolra@tin> |
Started deploy [parsoid/deploy@4882c59]: Updating Parsoid to 8e99708a |
[production] |
| 17:17 |
<awight@tin> |
Finished deploy [ores/deploy@971be22]: Rolling back scb1002, T175180 T178441 (duration: 00m 32s) |
[production] |
| 17:16 |
<awight@tin> |
Started deploy [ores/deploy@971be22]: Rolling back scb1002, T175180 T178441 |
[production] |
| 17:14 |
<awight@tin> |
Finished deploy [ores/deploy@971be22]: ORES w/ revscoring 2 and Celery 4, T175180 T178441 (duration: 03m 20s) |
[production] |
| 17:11 |
<awight@tin> |
Started deploy [ores/deploy@971be22]: ORES w/ revscoring 2 and Celery 4, T175180 T178441 |
[production] |
| 17:09 |
<marostegui> |
Optimize ruwiki.recentchanges on dbstore1001 - T162789 |
[production] |
| 17:08 |
<dcausse> |
[logstash] reimporting today logs |
[production] |
| 17:08 |
<dcausse> |
[logstash] deleting today logs to reapply mapping template |
[production] |
| 16:58 |
<ottomata> |
now mirroring main Kafka cluster topics to jumbo Kafka cluster, with MirrorMaker instances running on analytics-eqiad broker nodes. https://phabricator.wikimedia.org/T177216 |
[production] |
| 16:44 |
<awight@tin> |
Finished deploy [ores/deploy@971be22]: Push ORES w/ Celery 4 support to new cluster (try to rebuild venv), T178441 (duration: 01m 02s) |
[production] |
| 16:43 |
<awight@tin> |
Started deploy [ores/deploy@971be22]: Push ORES w/ Celery 4 support to new cluster (try to rebuild venv), T178441 |
[production] |
| 16:33 |
<awight@tin> |
Finished deploy [ores/deploy@971be22]: Push ORES w/ Celery 4 support to new cluster (take 3), T178441 (duration: 15m 33s) |
[production] |
| 16:21 |
<urandom> |
restarted cassandra, restbase2001-b |
[production] |
| 16:18 |
<awight@tin> |
Started deploy [ores/deploy@971be22]: Push ORES w/ Celery 4 support to new cluster (take 3), T178441 |
[production] |
| 16:15 |
<urandom> |
truncating cassandra hints in restbase-ng cluster |
[production] |
| 16:12 |
<mobrovac@tin> |
Finished deploy [restbase/deploy@860cbfe]: Revert all summaries, back to all but WP (duration: 07m 57s) |
[production] |
| 16:04 |
<mobrovac@tin> |
Started deploy [restbase/deploy@860cbfe]: Revert all summaries, back to all but WP |
[production] |
| 16:04 |
<mobrovac@tin> |
Started deploy [restbase/deploy@860cbfe]: Revert all summaries, back to all but WP |
[production] |
| 15:50 |
<mobrovac@tin> |
Finished deploy [restbase/deploy@522321a]: Double-process all summaries (duration: 09m 09s) |
[production] |
| 15:45 |
<urandom> |
restarting cassandra, restbase2001-b |
[production] |
| 15:41 |
<andrewbogott> |
upgrading dnsmasq on labnet1001 and labnet1002 |
[production] |