| 
      
        2016-06-09
      
      §
     | 
  
    
  | 20:57 | 
  <hoo@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on nnwiki (T130997) (duration: 00m 24s) | 
  [production] | 
            
  | 20:53 | 
  <hoo@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on lvwiki (T136100) (duration: 00m 26s) | 
  [production] | 
            
  | 20:48 | 
  <hoo@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on guwiki (T136517) (duration: 00m 24s) | 
  [production] | 
            
  | 20:40 | 
  <hoo@tin> | 
  Synchronized php-1.28.0-wmf.4/extensions/Wikidata: Update ArticlePlaceholder (duration: 01m 54s) | 
  [production] | 
            
  | 20:36 | 
  <hoo@tin> | 
  Synchronized php-1.28.0-wmf.5/extensions/Wikidata: Update ArticlePlaceholder (without unrelated T136598 fixes this time) (duration: 01m 51s) | 
  [production] | 
            
  | 20:33 | 
  <aaron@tin> | 
  Synchronized php-1.28.0-wmf.5/includes/user/User.php: c3b1f80a701d61dc57ccac0c8b1dc7daf03fa925 (duration: 00m 29s) | 
  [production] | 
            
  | 19:59 | 
  <urandom> | 
  Restarting Cassandra on xenon.eqiad.wmnet (removing patched test build; restoring state) : T137474 | 
  [production] | 
            
  | 19:53 | 
  <hoo@tin> | 
  Synchronized php-1.28.0-wmf.5/extensions/Wikidata: revert, possible s5 master overload (duration: 01m 57s) | 
  [production] | 
            
  | 19:47 | 
  <hoo@tin> | 
  Synchronized php-1.28.0-wmf.5/extensions/Wikidata: Update ArticlePlaceholder (duration: 02m 04s) | 
  [production] | 
            
  | 19:44 | 
  <bearND> | 
  mobileapps deployed 71ff97c | 
  [production] | 
            
  | 19:42 | 
  <bearND> | 
  starting mobileapps deploy | 
  [production] | 
            
  | 19:11 | 
  <ejegg> | 
  updated cancel page settings on payments-wiki | 
  [production] | 
            
  | 17:43 | 
  <urandom> | 
  Restarting Cassandra on xenon.eqiad.wmnet (use exponentially decaying resevoirs for metrics histograms) : T126629 | 
  [production] | 
            
  | 17:19 | 
  <mobrovac> | 
  change-prop deploying ecfda93f09d | 
  [production] | 
            
  | 17:09 | 
  <ejegg> | 
  updated payments-wiki from 3dcf58e3b4e1d02ad4f1874a3e87e55b7e169bfe to 053aaa259382c94aa59e4d0da7317fcafab635cd | 
  [production] | 
            
  | 15:31 | 
  <elukey> | 
  added topic override retention.bytes=536870912000 to Kafka webrequest_text (T136690)  | 
  [production] | 
            
  | 15:22 | 
  <hashar> | 
  Cleaning git-daemon on gallium (was used by zuul-merger) T137418 | 
  [production] | 
            
  | 15:19 | 
  <aude@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: Add *.nara.gov to wgCopyUploadDomains (duration: 00m 40s) | 
  [production] | 
            
  | 14:47 | 
  <mobrovac> | 
  change-prop stopped on scb1002 | 
  [production] | 
            
  | 14:38 | 
  <elukey> | 
  Tested temp setting retention.bytes=2G for Analytics kafka topic webrequest_misc | 
  [production] | 
            
  | 14:37 | 
  <hashar> | 
  Removing zuul-merger from gallium | 
  [production] | 
            
  | 14:33 | 
  <hashar> | 
  stopped / disabled zuul-merger on gallium T137418 | 
  [production] | 
            
  | 14:12 | 
  <mobrovac> | 
  change-prop restarting on scb1001 for update | 
  [production] | 
            
  | 14:07 | 
  <urandom> | 
  Re-enabling puppet on xenon.eqiad.wmnet, forcing a run, and restarting Cassandra : T137419 | 
  [production] | 
            
  | 13:52 | 
  <mobrovac> | 
  change-prop restarting on scb1002 for update | 
  [production] | 
            
  | 13:45 | 
  <mobrovac> | 
  change-prop deploying 2161403c | 
  [production] | 
            
  | 13:26 | 
  <urandom> | 
  Restarting Cassandra on xenon.eqiad.wmnet to apply 2G file cache : T137419 | 
  [production] | 
            
  | 12:50 | 
  <urandom> | 
  Restarting Cassandra on xenon.eqiad.wmnet : T126629 | 
  [production] | 
            
  | 12:33 | 
  <tgr@tin> | 
  Synchronized php-1.28.0-wmf.5/extensions/LdapAuthentication/LdapPrimaryAuthenticationProvider.php: deploy [[gerrit:293459]] to fix wikitech API login / morebots (T137377) (duration: 00m 47s) | 
  [production] | 
            
  | 12:19 | 
  <tgr> | 
  deploying [[gerrit:293459]] to fix morebots (T137377) | 
  [production] | 
            
  | 12:11 | 
  <urandom> | 
  Restarting Cassandra on xenon.eqiad.wmnet : T126629 | 
  [production] | 
            
  | 12:06 | 
  <urandom> | 
  Temporarily disabling puppet on xenon.eqiad.wmnet to test settings : T126629 | 
  [production] | 
            
  | 11:33 | 
  <Amir1> | 
  manually restarting ores-uwsgi and celery-ores-worker in scb100[12] | 
  [production] | 
            
  | 10:51 | 
  <urandom> | 
  Restarting Cassandra on {cerium,praseodymium}.eqiad.wmnet (RESTBase staging) : T126629 | 
  [production] | 
            
  | 09:16 | 
  <gehel> | 
  lowering disk high watermark to rebalance disk usage on elasticsearch eqiad cluster | 
  [production] | 
            
  | 09:05 | 
  <Amir1> | 
  restarting uwsgi-ores celery-ores-worker in scb1001 and scb1002 | 
  [production] | 
            
  | 08:54 | 
  <moritzm> | 
  installing libtasn security updates | 
  [production] | 
            
  | 08:38 | 
  <moritzm> | 
  rolling restart of app server canaries for libtasn security update | 
  [production] | 
            
  | 07:22 | 
  <moritzm> | 
  removed /var/log/logstash/logstash.1 on logstash1001, logspam (similar to the what is described in https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/144) depleted the space on the root partition | 
  [production] | 
            
  | 02:55 | 
  <l10nupdate@tin> | 
  ResourceLoader cache refresh completed at Thu Jun  9 02:55:20 UTC 2016 (duration 6m 19s) | 
  [production] | 
            
  | 02:53 | 
  <mutante> | 
  ms-be2012 ran out of disk due to huge syslog, deleted log, restarted rsyslogd | 
  [production] | 
            
  | 02:49 | 
  <mwdeploy@tin> | 
  scap sync-l10n completed (1.28.0-wmf.5) (duration: 11m 02s) | 
  [production] | 
            
  | 02:26 | 
  <mwdeploy@tin> | 
  scap sync-l10n completed (1.28.0-wmf.4) (duration: 11m 00s) | 
  [production] | 
            
  | 00:16 | 
  <ejegg> | 
  updated payments from de11bfe2273d0bcaa0e713389b2d91e8b3567a1d to c6ea2ca699aa78a90f64c77d860f69421e9a7782 | 
  [production] | 
            
  | 00:15 | 
  <mutante> | 
  restarted the log bot | 
  [production] | 
            
  | 00:11 | 
  <mutante> | 
  restarted the log bot | 
  [production] | 
            
  | 00:06 | 
  <mutante> | 
  meta T46791 | 
  [production] | 
            
  | 00:03 | 
  <twentyafterfour> | 
  Preparing to deploy phabricator update. Tagged release/2016-06-08/1 | 
  [production] |