|
2018-02-22
ยง
|
| 21:11 |
<gehel> |
powercycling wdqs1004 (complete loss of network) |
[production] |
| 20:39 |
<demon@tin> |
Synchronized php-1.31.0-wmf.22/includes/libs/objectcache/WANObjectCache.php: betterer logging for cache ttl reduction, Iea029e78 (duration: 01m 13s) |
[production] |
| 19:33 |
<XioNoX> |
redirecting Facebook bots large source of traffic to codfw ( https://gerrit.wikimedia.org/r/#/c/413446/ ) |
[production] |
| 19:14 |
<akosiaris> |
rolling restart of eqiad appservers. sudo cumin -b3 -s 30 'A:mw-eqiad' 'restart-hhvm' T188019 |
[production] |
| 19:12 |
<twentyafterfour@tin> |
Synchronized php-1.31.0-wmf.22/extensions/SyntaxHighlight_GeSHi: deploy https://gerrit.wikimedia.org/r/#/c/413437/ (duration: 01m 13s) |
[production] |
| 19:09 |
<twentyafterfour@tin> |
Synchronized php-1.31.0-wmf.21/extensions/SyntaxHighlight_GeSHi: deploy https://gerrit.wikimedia.org/r/#/c/413437/ (duration: 01m 13s) |
[production] |
| 19:09 |
<twentyafterfour> |
syncing https://gerrit.wikimedia.org/r/#/c/413437/ |
[production] |
| 19:03 |
<chasemp> |
baham:~# authdns-update |
[production] |
| 19:00 |
<jynus@tin> |
Synchronized wmf-config/db-codfw.php: Repool db2073 (duration: 01m 12s) |
[production] |
| 17:23 |
<elukey> |
installed linux-perf-4.9 on phab1001 to experiment with perf tracing |
[production] |
| 17:20 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Fully repool db1076 (duration: 01m 12s) |
[production] |
| 17:05 |
<XioNoX> |
rolling back "redirecting ns2 traffic to radon" |
[production] |
| 17:02 |
<ema> |
reboot eeden with new kernel 4.9.0-0.bpo.6 |
[production] |
| 16:58 |
<XioNoX> |
redirecting ns2 traffic to radon |
[production] |
| 16:49 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase traffic for db1076 (duration: 01m 12s) |
[production] |
| 16:28 |
<ejegg> |
updated CiviCRM from b27e6a5019 to aa251f1a93 |
[production] |
| 16:26 |
<mobrovac@tin> |
Synchronized wmf-config/jobqueue.php: Use EventBus for refreshLinks in test wikis, file 2/2 - T185052 (duration: 01m 12s) |
[production] |
| 16:25 |
<mobrovac@tin> |
Synchronized wmf-config/InitialiseSettings.php: Use EventBus for refreshLinks in test wikis, file 1/2 - T185052 (duration: 01m 12s) |
[production] |
| 16:23 |
<ppchelko@tin> |
Finished deploy [cpjobqueue/deploy@ab3d002]: Enable refreshLinks for group0 wikis T185052 (duration: 00m 36s) |
[production] |
| 16:23 |
<ppchelko@tin> |
Started deploy [cpjobqueue/deploy@ab3d002]: Enable refreshLinks for group0 wikis T185052 |
[production] |
| 16:22 |
<mobrovac@tin> |
scap failed: average error rate on 8/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details) |
[production] |
| 16:13 |
<jynus> |
tendril and dbtree database currently under maintanance |
[production] |
| 16:04 |
<ejegg> |
updated payments-wiki from fe311c2d26 to 1acfc4a9a0 |
[production] |
| 15:26 |
<ema> |
finished upgrading cache_text@ulsfo to varnish 5 |
[production] |
| 15:24 |
<elukey> |
manually removing from cp1008 and cache::misc old files related to the varnishkafka jumbo testing instance (after https://gerrit.wikimedia.org/r/413370) |
[production] |
| 14:58 |
<matthiasmullie> |
EU SWAT finished |
[production] |
| 14:52 |
<mlitn@tin> |
Synchronized wmf-config/CommonSettings.php: Enable 3D file display (duration: 01m 12s) |
[production] |
| 14:50 |
<mlitn@tin> |
Synchronized php-1.31.0-wmf.21/extensions/3D/extension.json: Remove MMV dependency for 3D (duration: 01m 12s) |
[production] |
| 14:41 |
<ottomata> |
beginning migration of webrequest_misc from Kafka analytics to jumbo: T185136 |
[production] |
| 14:40 |
<mlitn@tin> |
scap failed: average error rate on 3/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details) |
[production] |
| 14:38 |
<mlitn@tin> |
Synchronized wmf-config/InitialiseSettings.php: Enable 3D file display (duration: 01m 13s) |
[production] |
| 14:32 |
<jmm@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=mw2171.codfw.wmnet |
[production] |
| 14:26 |
<zfilipin@tin> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:396318|Show HTML summaries on cswiki (T182321)]] (duration: 01m 13s) |
[production] |
| 13:41 |
<ema> |
bounce pybal on lvs1003 to try establish missing etcd connections (zotero, thumbor, wdqs) https://phabricator.wikimedia.org/P6730 |
[production] |
| 13:30 |
<moritzm> |
rebooting kubernetes1001 |
[production] |
| 13:21 |
<ema> |
upgrade pybal on lvs1003 to 1.14.4 |
[production] |
| 12:42 |
<_joe_> |
ended live-hacking on mwdebug1001 (T185078) |
[production] |
| 12:24 |
<_joe_> |
live-hacking ProductionServices.php on mwdebug1001 for testing (T185078) |
[production] |
| 11:50 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase traffic for db1089 and slowly repool db1076 (duration: 01m 12s) |
[production] |
| 11:40 |
<kartik@tin> |
Finished deploy [cxserver/deploy@300f728]: Update cxserver to b0404d1 (duration: 03m 37s) |
[production] |
| 11:39 |
<akosiaris> |
purge ORES from scb hosts T168073 T171851 |
[production] |
| 11:37 |
<kartik@tin> |
Started deploy [cxserver/deploy@300f728]: Update cxserver to b0404d1 |
[production] |
| 11:19 |
<_joe_> |
upgrading python-conftool on all cache hosts |
[production] |
| 10:55 |
<ema> |
upgrading python-conftool on cp5007 |
[production] |
| 10:51 |
<_joe_> |
upgrading python-conftool on cp1008 |
[production] |
| 10:42 |
<jynus> |
stop db2073 for maintenance |
[production] |
| 10:40 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase traffic for db1089 and fully repool db1104 (duration: 01m 13s) |
[production] |
| 10:37 |
<_joe_> |
benchmarking EtcdConfig failure scenarios on mwdebug1001, T185078 |
[production] |
| 10:22 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase traffic for db1104 - T186321 (duration: 01m 14s) |
[production] |
| 10:18 |
<ema> |
upgrade cache_text @ ulsfo to varnish 5 |
[production] |