| 2018-02-22
      
      § | 
    
  | 22:33 | <demon@tin> | Synchronized php-1.31.0-wmf.22/includes/filerepo/file/LocalFile.php: Id5cdd8ec (duration: 01m 12s) | [production] | 
            
  | 22:32 | <demon@tin> | Synchronized php-1.31.0-wmf.22/includes/externalstore/: Id5cdd8ec (duration: 01m 12s) | [production] | 
            
  | 22:30 | <demon@tin> | Synchronized php-1.31.0-wmf.22/includes/Storage/: Id5cdd8ec (duration: 01m 13s) | [production] | 
            
  | 22:16 | <maxsem@tin> | Synchronized php-1.31.0-wmf.21/extensions/SyntaxHighlight_GeSHi/: T188019 (duration: 01m 12s) | [production] | 
            
  | 22:14 | <maxsem@tin> | Synchronized php-1.31.0-wmf.22/extensions/SyntaxHighlight_GeSHi/: T188019 (duration: 01m 14s) | [production] | 
            
  | 21:51 | <demon@tin> | Synchronized php-1.31.0-wmf.22/includes/externalstore/: I9334d36e (duration: 01m 15s) | [production] | 
            
  | 21:37 | <dzahn@puppetmaster1001> | conftool action : set/pooled=no; selector: name=wdqs1004.eqiad.wmnet | [production] | 
            
  | 21:11 | <gehel> | powercycling wdqs1004 (complete loss of network) | [production] | 
            
  | 20:39 | <demon@tin> | Synchronized php-1.31.0-wmf.22/includes/libs/objectcache/WANObjectCache.php: betterer logging for cache ttl reduction, Iea029e78 (duration: 01m 13s) | [production] | 
            
  | 19:33 | <XioNoX> | redirecting Facebook bots large source of traffic to codfw ( https://gerrit.wikimedia.org/r/#/c/413446/ ) | [production] | 
            
  | 19:14 | <akosiaris> | rolling restart of eqiad appservers. sudo cumin -b3 -s 30 'A:mw-eqiad' 'restart-hhvm' T188019 | [production] | 
            
  | 19:12 | <twentyafterfour@tin> | Synchronized php-1.31.0-wmf.22/extensions/SyntaxHighlight_GeSHi: deploy https://gerrit.wikimedia.org/r/#/c/413437/ (duration: 01m 13s) | [production] | 
            
  | 19:09 | <twentyafterfour@tin> | Synchronized php-1.31.0-wmf.21/extensions/SyntaxHighlight_GeSHi: deploy https://gerrit.wikimedia.org/r/#/c/413437/ (duration: 01m 13s) | [production] | 
            
  | 19:09 | <twentyafterfour> | syncing https://gerrit.wikimedia.org/r/#/c/413437/ | [production] | 
            
  | 19:03 | <chasemp> | baham:~# authdns-update | [production] | 
            
  | 19:00 | <jynus@tin> | Synchronized wmf-config/db-codfw.php: Repool db2073 (duration: 01m 12s) | [production] | 
            
  | 17:23 | <elukey> | installed linux-perf-4.9 on phab1001 to experiment with perf tracing | [production] | 
            
  | 17:20 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Fully repool db1076 (duration: 01m 12s) | [production] | 
            
  | 17:05 | <XioNoX> | rolling back "redirecting ns2 traffic to radon" | [production] | 
            
  | 17:02 | <ema> | reboot eeden with new kernel 4.9.0-0.bpo.6 | [production] | 
            
  | 16:58 | <XioNoX> | redirecting ns2 traffic to radon | [production] | 
            
  | 16:49 | <marostegui@tin> | Synchronized wmf-config/db-eqiad.php: Increase traffic for db1076 (duration: 01m 12s) | [production] | 
            
  | 16:28 | <ejegg> | updated CiviCRM from b27e6a5019 to aa251f1a93 | [production] | 
            
  | 16:26 | <mobrovac@tin> | Synchronized wmf-config/jobqueue.php: Use EventBus for refreshLinks in test wikis, file 2/2 - T185052 (duration: 01m 12s) | [production] | 
            
  | 16:25 | <mobrovac@tin> | Synchronized wmf-config/InitialiseSettings.php: Use EventBus for refreshLinks in test wikis, file 1/2 - T185052 (duration: 01m 12s) | [production] | 
            
  | 16:23 | <ppchelko@tin> | Finished deploy [cpjobqueue/deploy@ab3d002]: Enable refreshLinks for group0 wikis T185052 (duration: 00m 36s) | [production] | 
            
  | 16:23 | <ppchelko@tin> | Started deploy [cpjobqueue/deploy@ab3d002]: Enable refreshLinks for group0 wikis T185052 | [production] | 
            
  | 16:22 | <mobrovac@tin> | scap failed: average error rate on 8/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details) | [production] | 
            
  | 16:13 | <jynus> | tendril and dbtree database currently under maintanance | [production] | 
            
  | 16:04 | <ejegg> | updated payments-wiki from fe311c2d26 to 1acfc4a9a0 | [production] | 
            
  | 15:26 | <ema> | finished upgrading cache_text@ulsfo to varnish 5 | [production] | 
            
  | 15:24 | <elukey> | manually removing from cp1008 and cache::misc old files related to the varnishkafka jumbo testing instance (after https://gerrit.wikimedia.org/r/413370) | [production] | 
            
  | 14:58 | <matthiasmullie> | EU SWAT finished | [production] | 
            
  | 14:52 | <mlitn@tin> | Synchronized wmf-config/CommonSettings.php: Enable 3D file display (duration: 01m 12s) | [production] | 
            
  | 14:50 | <mlitn@tin> | Synchronized php-1.31.0-wmf.21/extensions/3D/extension.json: Remove MMV dependency for 3D (duration: 01m 12s) | [production] | 
            
  | 14:41 | <ottomata> | beginning migration of webrequest_misc from Kafka analytics to jumbo: T185136 | [production] | 
            
  | 14:40 | <mlitn@tin> | scap failed: average error rate on 3/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details) | [production] | 
            
  | 14:38 | <mlitn@tin> | Synchronized wmf-config/InitialiseSettings.php: Enable 3D file display (duration: 01m 13s) | [production] | 
            
  | 14:32 | <jmm@puppetmaster1001> | conftool action : set/pooled=no; selector: name=mw2171.codfw.wmnet | [production] | 
            
  | 14:26 | <zfilipin@tin> | Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:396318|Show HTML summaries on cswiki (T182321)]] (duration: 01m 13s) | [production] | 
            
  | 13:41 | <ema> | bounce pybal on lvs1003 to try establish missing etcd connections (zotero, thumbor, wdqs) https://phabricator.wikimedia.org/P6730 | [production] | 
            
  | 13:30 | <moritzm> | rebooting kubernetes1001 | [production] | 
            
  | 13:21 | <ema> | upgrade pybal on lvs1003 to 1.14.4 | [production] |