|
2017-10-27
§
|
| 11:36 |
<ema> |
cp4023: varnish-backend-restart for lag |
[production] |
| 11:06 |
<mobrovac@tin> |
Finished deploy [citoid/deploy@ff63420]: Update dependencies (duration: 03m 18s) |
[production] |
| 11:04 |
<marostegui> |
Optimize cebwiki.templatelinks on db1103 - T174509 |
[production] |
| 11:03 |
<mobrovac@tin> |
Started deploy [citoid/deploy@ff63420]: Update dependencies |
[production] |
| 08:56 |
<marostegui> |
Optimize table commonswiki.recentchanges on dbstore1001 - T162789 |
[production] |
| 08:11 |
<marostegui> |
Drop redundant indexes from pagelinks and templatelinks on s3 wikis only for dbstore2001 and dbstore2002 - T174509 |
[production] |
| 05:59 |
<marostegui> |
Stop MySQL on db2084 to copy s5 to db2086 - T178359 |
[production] |
| 05:55 |
<marostegui> |
dbstore1001: convert itwiki.pagelinks to TokuDB - T162789 |
[production] |
| 05:53 |
<mutante> |
uploaded parsoid_0.8.0 to releases.wikimedia.org (for Subbu - T179134) |
[production] |
| 05:46 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1060 - T174509 (duration: 00m 50s) |
[production] |
| 03:02 |
<bblack> |
cp1067, cp4026 - backend restarts, mailbox lag |
[production] |
|
2017-10-26
§
|
| 22:47 |
<demon@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: wikidata to wmf.4 |
[production] |
| 22:39 |
<bblack> |
restarting varnish-backend on cp1053 (mailbox lag from ongoing issues elsewhere?) |
[production] |
| 21:40 |
<bblack> |
raising backend max_connections for api.svc.eqiad.wmnet + appservers.svc.eqiad.wmnet from 1K to 10K on cp1053.eqiad.wmnet (current funnel for the bulk of the 503s) |
[production] |
| 21:32 |
<hoo@tin> |
Synchronized wmf-config/InitialiseSettings.php: Temporary disable remex html (T178632) (duration: 00m 50s) |
[production] |
| 21:30 |
<hoo@tin> |
Synchronized wmf-config/InitialiseSettings.php: Temporary disable remex html (T178632) (duration: 00m 50s) |
[production] |
| 21:00 |
<hoo> |
Fully revert all changes related to T178180 |
[production] |
| 20:58 |
<hoo@tin> |
Synchronized wmf-config/Wikibase.php: Revert "Add property for RDF mapping of external identifiers for Wikidata" (T178180) (duration: 00m 50s) |
[production] |
| 20:02 |
<ladsgroup@tin> |
Synchronized wmf-config/InitialiseSettings.php: UBN! disbale ores for wikidata (T179107) (duration: 00m 50s) |
[production] |
| 20:00 |
<ladsgroup@tin> |
Synchronized wmf-config/InitialiseSettings.php: UBN! disbale ores for wikidata (T179107) (duration: 00m 50s) |
[production] |
| 19:41 |
<awight@tin> |
Finished deploy [ores/deploy@0adae70]: Increase extractor wikidata API timeout to 15s, T179107 (duration: 07m 25s) |
[production] |
| 19:36 |
<aaron@tin> |
Started restart [jobrunner/jobrunner@a20d043]: (no justification provided) |
[production] |
| 19:34 |
<awight@tin> |
Started deploy [ores/deploy@0adae70]: Increase extractor wikidata API timeout to 15s, T179107 |
[production] |
| 19:33 |
<awight@tin> |
Finished deploy [ores/deploy@0adae70]: Increase extractor wikidata API timeout to 15s, T179107 (duration: 00m 10s) |
[production] |
| 19:33 |
<awight@tin> |
Started deploy [ores/deploy@0adae70]: Increase extractor wikidata API timeout to 15s, T179107 |
[production] |
| 19:27 |
<demon@tin> |
Synchronized php-1.31.0-wmf.5/includes/libs/filebackend/SwiftFileBackend.php: Better error grouping (duration: 00m 50s) |
[production] |
| 19:14 |
<ladsgroup@tin> |
Synchronized php-1.31.0-wmf.5/extensions/WikibaseQualityConstraints/includes/ConstraintCheck/DelegatingConstraintChecker.php: Fix sorting of NullResults (T179038) (duration: 00m 49s) |
[production] |
| 19:12 |
<ladsgroup@tin> |
Synchronized php-1.31.0-wmf.5/extensions/WikibaseQualityConstraints/tests/phpunit/DelegatingConstraintCheckerTest.php: Fix sorting of NullResults (T179038) (duration: 00m 50s) |
[production] |
| 18:52 |
<ladsgroup@tin> |
Synchronized php-1.31.0-wmf.5/extensions/Wikidata/extensions/Constraints/tests/phpunit/DelegatingConstraintCheckerTest.php: Fix sorting of NullResults (T179038) (duration: 00m 49s) |
[production] |
| 18:51 |
<ladsgroup@tin> |
Synchronized php-1.31.0-wmf.5/extensions/Wikidata/extensions/Constraints/includes/ConstraintCheck/DelegatingConstraintChecker.php: Fix sorting of NullResults (T179038) (duration: 01m 04s) |
[production] |
| 18:41 |
<bblack> |
eqiad cp servers: rolling quick depool -> repool around ethtool parameter changes for -lro,-pause |
[production] |
| 18:30 |
<urandom> |
Dropping and recreating keyspace enwiki_T_page__summary (restbase-ng cluster) |
[production] |
| 18:27 |
<bblack> |
esams cp servers: rolling quick depool -> repool around ethtool parameter changes for -lro,-pause |
[production] |
| 18:26 |
<urandom> |
T179105: Altering "enwiki_T_page__summary".data to use LZ4Compressor |
[production] |
| 17:59 |
<bblack> |
codfw cp servers: rolling quick depool -> repool around ethtool parameter changes for -lro,-pause |
[production] |
| 17:57 |
<bblack@neodymium> |
conftool action : set/pooled=no; selector: name=cp4024.ulsfo.wmnet |
[production] |
| 17:49 |
<bblack> |
ulsfo cp servers: rolling quick depool -> repool around ethtool parameter changes for -lro,-pause |
[production] |
| 17:47 |
<awight@tin> |
Finished deploy [ores/deploy@f5deb7f]: Revscoring 2 on ores1002 (non-production) (duration: 01m 45s) |
[production] |
| 17:45 |
<awight@tin> |
Started deploy [ores/deploy@f5deb7f]: Revscoring 2 on ores1002 (non-production) |
[production] |
| 17:36 |
<arlolra> |
Updated Parsoid to 8e99708a (T176728) |
[production] |
| 17:30 |
<arlolra@tin> |
Finished deploy [parsoid/deploy@4882c59]: Updating Parsoid to 8e99708a (duration: 08m 40s) |
[production] |
| 17:21 |
<arlolra@tin> |
Started deploy [parsoid/deploy@4882c59]: Updating Parsoid to 8e99708a |
[production] |
| 17:17 |
<awight@tin> |
Finished deploy [ores/deploy@971be22]: Rolling back scb1002, T175180 T178441 (duration: 00m 32s) |
[production] |
| 17:16 |
<awight@tin> |
Started deploy [ores/deploy@971be22]: Rolling back scb1002, T175180 T178441 |
[production] |
| 17:14 |
<awight@tin> |
Finished deploy [ores/deploy@971be22]: ORES w/ revscoring 2 and Celery 4, T175180 T178441 (duration: 03m 20s) |
[production] |
| 17:11 |
<awight@tin> |
Started deploy [ores/deploy@971be22]: ORES w/ revscoring 2 and Celery 4, T175180 T178441 |
[production] |
| 17:09 |
<marostegui> |
Optimize ruwiki.recentchanges on dbstore1001 - T162789 |
[production] |
| 17:08 |
<dcausse> |
[logstash] reimporting today logs |
[production] |
| 17:08 |
<dcausse> |
[logstash] deleting today logs to reapply mapping template |
[production] |
| 16:58 |
<ottomata> |
now mirroring main Kafka cluster topics to jumbo Kafka cluster, with MirrorMaker instances running on analytics-eqiad broker nodes. https://phabricator.wikimedia.org/T177216 |
[production] |