|
2018-02-22
ยง
|
| 14:40 |
<mlitn@tin> |
scap failed: average error rate on 3/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details) |
[production] |
| 14:38 |
<mlitn@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable 3D file display (duration: 01m 13s) |
[production] |
| 14:32 |
<jmm@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=mw2171.codfw.wmnet |
[production] |
| 14:26 |
<zfilipin@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:396318|Show HTML summaries on cswiki (T182321)]] (duration: 01m 13s) |
[production] |
| 13:41 |
<ema> |
bounce pybal on lvs1003 to try establish missing etcd connections (zotero, thumbor, wdqs) https://phabricator.wikimedia.org/P6730 |
[production] |
| 13:30 |
<moritzm> |
rebooting kubernetes1001 |
[production] |
| 13:21 |
<ema> |
upgrade pybal on lvs1003 to 1.14.4 |
[production] |
| 12:42 |
<_joe_> |
ended live-hacking on mwdebug1001 (T185078) |
[production] |
| 12:24 |
<_joe_> |
live-hacking ProductionServices.php on mwdebug1001 for testing (T185078) |
[production] |
| 11:50 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase traffic for db1089 and slowly repool db1076 (duration: 01m 12s) |
[production] |
| 11:40 |
<kartik@tin> |
Finished deploy [cxserver/deploy@300f728]: Update cxserver to b0404d1 (duration: 03m 37s) |
[production] |
| 11:39 |
<akosiaris> |
purge ORES from scb hosts T168073 T171851 |
[production] |
| 11:37 |
<kartik@tin> |
Started deploy [cxserver/deploy@300f728]: Update cxserver to b0404d1 |
[production] |
| 11:19 |
<_joe_> |
upgrading python-conftool on all cache hosts |
[production] |
| 10:55 |
<ema> |
upgrading python-conftool on cp5007 |
[production] |
| 10:51 |
<_joe_> |
upgrading python-conftool on cp1008 |
[production] |
| 10:42 |
<jynus> |
stop db2073 for maintenance |
[production] |
| 10:40 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase traffic for db1089 and fully repool db1104 (duration: 01m 13s) |
[production] |
| 10:37 |
<_joe_> |
benchmarking EtcdConfig failure scenarios on mwdebug1001, T185078 |
[production] |
| 10:22 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase traffic for db1104 - T186321 (duration: 01m 14s) |
[production] |
| 10:18 |
<ema> |
upgrade cache_text @ ulsfo to varnish 5 |
[production] |
| 10:13 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Depool db2073 for maintenance (duration: 01m 12s) |
[production] |
| 10:08 |
<moritzm> |
uploaded Linux 4.9.82-1~wmf1 for jessie-wikimedia to apt.wikimedia.org (retpoline-enabled kernel) |
[production] |
| 10:00 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1089 with low traffic and depool db1067 - T162807 (duration: 01m 12s) |
[production] |
| 09:59 |
<akosiaris> |
reboot kraz.wikimedia.org (irc.wikimedia.org) |
[production] |
| 09:47 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase traffic for db1104 - T186321 (duration: 01m 12s) |
[production] |
| 09:31 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Slowly repool db1104 - T186321 (duration: 01m 12s) |
[production] |
| 09:20 |
<marostegui> |
Stop MySQL on db1104 to switch its binlog to statement - T186321 |
[production] |
| 09:20 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1104 - T186321 (duration: 01m 13s) |
[production] |
| 09:19 |
<moritzm> |
rebooting multatuli |
[production] |
| 09:03 |
<ema> |
eqiad LVSs: upgrade pybal to 1.14.4 |
[production] |
| 08:48 |
<jynus> |
tendril and dbtree database currently under maintanance |
[production] |
| 08:47 |
<ema> |
codfw LVSs: upgrade pybal to 1.14.4 |
[production] |
| 08:35 |
<marostegui> |
Stop tendril database (db1011) to copy it to db1115 - tendril will be offline while the copy is in progress - T184704 |
[production] |
| 08:32 |
<ema> |
esams LVSs: upgrade pybal to 1.14.4 |
[production] |
| 08:24 |
<ema> |
ulsfo LVSs: upgrade pybal to 1.14.4 |
[production] |
| 08:05 |
<marostegui> |
Disable puppet on db1011 - T184704 |
[production] |
| 07:48 |
<krinkle@tin> |
Synchronized wmf-config/FeaturedFeedsWMF.php: I73945d7d - minor clean-up (duration: 01m 13s) |
[production] |
| 07:32 |
<_joe_> |
starting tests on mwdebug1001 again |
[production] |
| 07:32 |
<marostegui> |
Deploy schema change on db1076 - T187089 T185128 T153182 |
[production] |
| 07:24 |
<marostegui> |
Stop MySQL on db1076 for mariadb and kernel upgrade + alter table |
[production] |
| 07:24 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1076 for alter table (duration: 01m 14s) |
[production] |
| 06:38 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Repool db1105 (duration: 01m 13s) |
[production] |
| 06:21 |
<marostegui> |
Stop puppet and mysql on db1011 to get ready to copy its data to db1115 - T184704 |
[production] |
| 02:24 |
<l10nupdate@tin> |
scap sync-l10n completed (1.31.0-wmf.21) (duration: 05m 53s) |
[production] |
| 01:05 |
<anomie> |
Running cleanupBlocks.php on more wikis for T187834: alswiki bgwiki bhwiki cawiki dewiki elwiki eswiki frwiki hewiki hiwiki huwiki hywiki jawiki jawikibooks jawikinews jawikiquote jawikisource jawiktionary kawiki kowiki mswiki mswiktionary rowiki sourceswiki |
[production] |
| 01:01 |
<anomie> |
Running cleanupBlocks.php on mediawikiwiki for T187834 |
[production] |
| 00:46 |
<smalyshev@tin> |
Finished deploy [wdqs/wdqs@5131080]: update whitelist to include categories namespace (duration: 03m 07s) |
[production] |
| 00:43 |
<smalyshev@tin> |
Started deploy [wdqs/wdqs@5131080]: update whitelist to include categories namespace |
[production] |
| 00:41 |
<smalyshev@tin> |
Finished deploy [wdqs/wdqs@5131080]: update whitelist to include categories namespace (duration: 00m 27s) |
[production] |