|
2018-02-22
ยง
|
| 16:23 |
<ppchelko@tin> |
Started deploy [cpjobqueue/deploy@ab3d002]: Enable refreshLinks for group0 wikis T185052 |
[production] |
| 16:22 |
<mobrovac@tin> |
scap failed: average error rate on 8/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details) |
[production] |
| 16:13 |
<jynus> |
tendril and dbtree database currently under maintanance |
[production] |
| 16:04 |
<ejegg> |
updated payments-wiki from fe311c2d26 to 1acfc4a9a0 |
[production] |
| 15:26 |
<ema> |
finished upgrading cache_text@ulsfo to varnish 5 |
[production] |
| 15:24 |
<elukey> |
manually removing from cp1008 and cache::misc old files related to the varnishkafka jumbo testing instance (after https://gerrit.wikimedia.org/r/413370) |
[production] |
| 14:58 |
<matthiasmullie> |
EU SWAT finished |
[production] |
| 14:52 |
<mlitn@tin> |
Synchronized wmf-config/CommonSettings.php: Enable 3D file display (duration: 01m 12s) |
[production] |
| 14:50 |
<mlitn@tin> |
Synchronized php-1.31.0-wmf.21/extensions/3D/extension.json: Remove MMV dependency for 3D (duration: 01m 12s) |
[production] |
| 14:41 |
<ottomata> |
beginning migration of webrequest_misc from Kafka analytics to jumbo: T185136 |
[production] |
| 14:40 |
<mlitn@tin> |
scap failed: average error rate on 3/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details) |
[production] |
| 14:38 |
<mlitn@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable 3D file display (duration: 01m 13s) |
[production] |
| 14:32 |
<jmm@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=mw2171.codfw.wmnet |
[production] |
| 14:26 |
<zfilipin@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:396318|Show HTML summaries on cswiki (T182321)]] (duration: 01m 13s) |
[production] |
| 13:41 |
<ema> |
bounce pybal on lvs1003 to try establish missing etcd connections (zotero, thumbor, wdqs) https://phabricator.wikimedia.org/P6730 |
[production] |
| 13:30 |
<moritzm> |
rebooting kubernetes1001 |
[production] |
| 13:21 |
<ema> |
upgrade pybal on lvs1003 to 1.14.4 |
[production] |
| 12:42 |
<_joe_> |
ended live-hacking on mwdebug1001 (T185078) |
[production] |
| 12:24 |
<_joe_> |
live-hacking ProductionServices.php on mwdebug1001 for testing (T185078) |
[production] |
| 11:50 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase traffic for db1089 and slowly repool db1076 (duration: 01m 12s) |
[production] |
| 11:40 |
<kartik@tin> |
Finished deploy [cxserver/deploy@300f728]: Update cxserver to b0404d1 (duration: 03m 37s) |
[production] |
| 11:39 |
<akosiaris> |
purge ORES from scb hosts T168073 T171851 |
[production] |
| 11:37 |
<kartik@tin> |
Started deploy [cxserver/deploy@300f728]: Update cxserver to b0404d1 |
[production] |
| 11:19 |
<_joe_> |
upgrading python-conftool on all cache hosts |
[production] |
| 10:55 |
<ema> |
upgrading python-conftool on cp5007 |
[production] |
| 10:51 |
<_joe_> |
upgrading python-conftool on cp1008 |
[production] |
| 10:42 |
<jynus> |
stop db2073 for maintenance |
[production] |
| 10:40 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase traffic for db1089 and fully repool db1104 (duration: 01m 13s) |
[production] |
| 10:37 |
<_joe_> |
benchmarking EtcdConfig failure scenarios on mwdebug1001, T185078 |
[production] |
| 10:22 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase traffic for db1104 - T186321 (duration: 01m 14s) |
[production] |
| 10:18 |
<ema> |
upgrade cache_text @ ulsfo to varnish 5 |
[production] |
| 10:13 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Depool db2073 for maintenance (duration: 01m 12s) |
[production] |
| 10:08 |
<moritzm> |
uploaded Linux 4.9.82-1~wmf1 for jessie-wikimedia to apt.wikimedia.org (retpoline-enabled kernel) |
[production] |
| 10:00 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1089 with low traffic and depool db1067 - T162807 (duration: 01m 12s) |
[production] |
| 09:59 |
<akosiaris> |
reboot kraz.wikimedia.org (irc.wikimedia.org) |
[production] |
| 09:47 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase traffic for db1104 - T186321 (duration: 01m 12s) |
[production] |
| 09:31 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Slowly repool db1104 - T186321 (duration: 01m 12s) |
[production] |
| 09:20 |
<marostegui> |
Stop MySQL on db1104 to switch its binlog to statement - T186321 |
[production] |
| 09:20 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1104 - T186321 (duration: 01m 13s) |
[production] |
| 09:19 |
<moritzm> |
rebooting multatuli |
[production] |
| 09:03 |
<ema> |
eqiad LVSs: upgrade pybal to 1.14.4 |
[production] |
| 08:48 |
<jynus> |
tendril and dbtree database currently under maintanance |
[production] |
| 08:47 |
<ema> |
codfw LVSs: upgrade pybal to 1.14.4 |
[production] |
| 08:35 |
<marostegui> |
Stop tendril database (db1011) to copy it to db1115 - tendril will be offline while the copy is in progress - T184704 |
[production] |
| 08:32 |
<ema> |
esams LVSs: upgrade pybal to 1.14.4 |
[production] |
| 08:24 |
<ema> |
ulsfo LVSs: upgrade pybal to 1.14.4 |
[production] |
| 08:05 |
<marostegui> |
Disable puppet on db1011 - T184704 |
[production] |
| 07:48 |
<krinkle@tin> |
Synchronized wmf-config/FeaturedFeedsWMF.php: I73945d7d - minor clean-up (duration: 01m 13s) |
[production] |
| 07:32 |
<_joe_> |
starting tests on mwdebug1001 again |
[production] |
| 07:32 |
<marostegui> |
Deploy schema change on db1076 - T187089 T185128 T153182 |
[production] |