|
2017-10-24
§
|
| 15:48 |
<elukey> |
drop table Edit_13457736_15423246 from the log database (Eventlogging) on db104[6,7], dbstore1002 |
[production] |
| 15:39 |
<elukey> |
set net.netfilter.nf_conntrack_tcp_timeout_time_wait=65 to mw[1308-1311] - T136094 |
[production] |
| 15:09 |
<gehel> |
restarting blazegraph on all wdqs nodes for GC config change - T175919 |
[production] |
| 14:48 |
<_joe_> |
also stopping HHVM, nginx, apache2 , and disabling all the services |
[production] |
| 14:46 |
<_joe_> |
stopping the jobrunner service on mw1161-67 |
[production] |
| 14:39 |
<marostegui> |
Optimize pagelinks templatelinks and recentchanges on db1030 - T174509 https://phabricator.wikimedia.org/T177772 |
[production] |
| 14:36 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1065 - T174509 (duration: 00m 45s) |
[production] |
| 14:33 |
<marostegui> |
Optimize ores_classification on enwiki db1065 - T159753 |
[production] |
| 14:32 |
<marostegui> |
Optimize pagelinks and templatelinks on db1065 - T174509 |
[production] |
| 14:30 |
<bd808> |
Restart webservice on kubernetes |
[tools.admin] |
| 14:24 |
<gehel> |
restarting blazegraph on wdqs2001 for GC config change - T175919 |
[production] |
| 14:20 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase db1106 weight (duration: 00m 45s) |
[production] |
| 13:42 |
<marostegui@tin> |
Synchronized wmf-config/db-codfw.php: Depool db2035 - T178359 (duration: 00m 45s) |
[production] |
| 13:41 |
<marostegui> |
Stop MySQL on db2035 to copy its data to db2088 - T178359 |
[production] |
| 13:39 |
<zeljkof> |
EU SWAT finished |
[production] |
| 13:39 |
<zfilipin@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:385843|Add $wgNamespaceRobotPolicies Config for hewikisource (T178775)]] (duration: 00m 46s) |
[production] |
| 13:22 |
<chasemp> |
start admin webservice |
[tools] |
| 13:22 |
<chasemp> |
stop admin webservice |
[tools] |
| 11:19 |
<elukey> |
removed several roles mistakenly applied to puppet prefix deployment-aqs in Horizon (causing puppet failures for AQS nodes) |
[releng] |
| 10:50 |
<dcausse> |
cirrus/elasticsearch: reindexing of 167 small wikis done (T177871) |
[production] |
| 09:22 |
<marostegui> |
Stop s1 on db2092 to copy its data to db2088 - T178359 |
[production] |
| 08:35 |
<hashar> |
beta: cherry pick https://gerrit.wikimedia.org/r/#/c/386077/4 "hieradata for varnish caches" - T178841 |
[releng] |
| 08:26 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Restore db1082 original weight - T178460 (duration: 00m 45s) |
[production] |
| 07:48 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase db1082 weight - T178460 (duration: 00m 45s) |
[production] |
| 07:46 |
<marostegui> |
Stop replication in sync on db1103 and db2018 to fix data drifts - T164488 |
[production] |
| 07:40 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1078 - T164488 (duration: 00m 46s) |
[production] |
| 07:28 |
<marostegui> |
Stop replication in sync on db1078 and db1103 to fix data drifts - https://phabricator.wikimedia.org/T164488 |
[production] |
| 06:22 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1078 - T164488 (duration: 00m 45s) |
[production] |
| 06:16 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1082 with low weight - T178460 (duration: 00m 47s) |
[production] |
| 02:32 |
<l10nupdate@tin> |
ResourceLoader cache refresh completed at Tue Oct 24 02:32:25 UTC 2017 (duration 6m 33s) |
[production] |
| 02:25 |
<l10nupdate@tin> |
scap sync-l10n completed (1.31.0-wmf.4) (duration: 08m 13s) |
[production] |
|
2017-10-23
§
|
| 23:09 |
<demon@tin> |
Synchronized wmf-config/throttle.php: Working Class Movement Library (Salford) throttle rule (duration: 00m 46s) |
[production] |
| 22:59 |
<bblack> |
cp4024: varnish-backend-restart for lag |
[production] |
| 21:40 |
<XioNoX> |
increasing MTU on Zayo transit interfaces |
[production] |
| 20:33 |
<arlolra> |
Updated Parsoid to 1cefef12 (T178217) |
[production] |
| 20:29 |
<Krinkle> |
Puppet still failing, now with: "Error 400 on SERVER: Could not find data item cache::fe_transient_gb in any Hiera data file and no default supplied at /etc/puppet/modules/profile/manifests/cache/text.pp:12 on node deployment-cache-text04.deployment-prep.eqiad.wmflabs" |
[releng] |
| 20:29 |
<Krinkle> |
Previous edit failed. Horizon saved the field as blank. Presumably because the class is unknown in the current version of puppet manifests it has. Strange that it normalises in this way. |
[releng] |
| 20:28 |
<Krinkle> |
Edit horizon "Other classes" config for deployment-prep/deployment-cache-text04. Rename role::prometheus::varnish_exporter to profile::prometheus::varnish_exporter |
[releng] |
| 20:25 |
<arlolra@tin> |
Finished deploy [parsoid/deploy@0759bd1]: Updating Parsoid to 1cefef12 (duration: 11m 14s) |
[production] |
| 20:14 |
<arlolra@tin> |
Started deploy [parsoid/deploy@0759bd1]: Updating Parsoid to 1cefef12 |
[production] |
| 20:13 |
<Krinkle> |
Puppet run still failing on Beta cluster varnish: "Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find class role::prometheus::varnish_exporter" |
[releng] |
| 19:37 |
<bsitzmann@tin> |
Finished deploy [mobileapps/deploy@a654ce1]: Update mobileapps to 3628105 (T178828) (duration: 06m 59s) |
[production] |
| 19:30 |
<bsitzmann@tin> |
Started deploy [mobileapps/deploy@a654ce1]: Update mobileapps to 3628105 (T178828) |
[production] |
| 19:00 |
<joal@tin> |
Finished deploy [analytics/aqs/deploy@ad22e0c]: Upgrade AQS node modules to restbase-latests - Redeploy after config fix (duration: 02m 34s) |
[production] |
| 18:58 |
<joal@tin> |
Started deploy [analytics/aqs/deploy@ad22e0c]: Upgrade AQS node modules to restbase-latests - Redeploy after config fix |
[production] |
| 18:58 |
<joal@tin> |
Finished deploy [analytics/aqs/deploy@ad22e0c]: Upgrade AQS node modules to restbase-latests - Redeploy after config fix (duration: 05m 07s) |
[production] |
| 18:52 |
<joal@tin> |
Started deploy [analytics/aqs/deploy@ad22e0c]: Upgrade AQS node modules to restbase-latests - Redeploy after config fix |
[production] |
| 18:52 |
<joal@tin> |
Finished deploy [analytics/aqs/deploy@ad22e0c]: Upgrade AQS node modules to restbase-latests (duration: 321m 51s) |
[production] |
| 18:50 |
<joal> |
Deploying AQS after fix |
[analytics] |
| 17:56 |
<gehel@tin> |
Finished deploy [wdqs/wdqs@b84592f]: WDQS GUI update (duration: 01m 58s) |
[production] |