| 
      
        2016-04-11
      
      §
     | 
  
    
  | 21:01 | 
  <bearND> | 
  mobileapps deployed 6ef3054 | 
  [production] | 
            
  | 20:55 | 
  <csteipp@tin> | 
  Synchronized wmf-config/CommonSettings-labs.php: retry oath in labs (duration: 00m 29s) | 
  [production] | 
            
  | 20:53 | 
  <csteipp@tin> | 
  Synchronized wmf-config/InitialiseSettings-labs.php: retry oath in labs (duration: 00m 32s) | 
  [production] | 
            
  | 20:43 | 
  <bearND> | 
  starting mobileapps deploy | 
  [production] | 
            
  | 20:41 | 
  <Krinkle> | 
  mwscript deleteEqualMessages.php --wiki plwiki (T45917) | 
  [production] | 
            
  | 20:19 | 
  <arlolra> | 
  updated Parsoid to version e3766b79 | 
  [production] | 
            
  | 20:09 | 
  <arlolra> | 
  synced code; restarted Parsoid on wtp1001.eqiad as a canary | 
  [production] | 
            
  | 20:05 | 
  <ebernhardson> | 
  temporarily changing elasticsearch high watermark to 75% to rebalance cluster | 
  [production] | 
            
  | 20:03 | 
  <volans> | 
  Deploy and use Puppet certs for TLS on cross-dc replica for shard s5 T111654 | 
  [production] | 
            
  | 20:00 | 
  <mutante> | 
  antimony - shred and delete git.wikimedia.org SSL key | 
  [production] | 
            
  | 19:57 | 
  <volans@tin> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1049 to deploy Pupper certs for TLS - T111654 (duration: 00m 28s) | 
  [production] | 
            
  | 19:55 | 
  <mutante> | 
  antimony - remove Apache package and config | 
  [production] | 
            
  | 19:53 | 
  <volans@tin> | 
  Synchronized wmf-config/db-eqiad.php: Depool db1049 to deploy Pupper certs for TLS - T111654 (duration: 02m 25s) | 
  [production] | 
            
  | 19:52 | 
  <mutante> | 
  powercycled mw1115 | 
  [production] | 
            
  | 19:41 | 
  <mutante> | 
  antimony (gitblit) - stop Apache  | 
  [production] | 
            
  | 19:15 | 
  <ejegg> | 
  extende globalcollect audit timeout from 90 min to 120 min for duration of NL campaign | 
  [production] | 
            
  | 19:01 | 
  <ori@tin> | 
  Synchronized php-1.27.0-wmf.20/extensions/WikimediaEvents: I672624e9fc30: Collect impact of proposed ResourceLoader feature-test in statsd (duration: 00m 34s) | 
  [production] | 
            
  | 18:08 | 
  <catrope@tin> | 
  Synchronized php-1.27.0-wmf.20/includes/RevisionList.php: Undo oversight live hack (duration: 00m 25s) | 
  [production] | 
            
  | 17:49 | 
  <catrope@tin> | 
  Synchronized php-1.27.0-wmf.20/includes/RevisionList.php: Attempt to fix oversight timeouts (duration: 00m 28s) | 
  [production] | 
            
  | 17:44 | 
  <gehel> | 
  rebooting wdqs1002 for kernel upgrade | 
  [production] | 
            
  | 17:36 | 
  <gehel> | 
  rebooting wdqs1001 | 
  [production] | 
            
  | 17:28 | 
  <gehel> | 
  deploying latest WDQS version | 
  [production] | 
            
  | 16:16 | 
  <ejegg> | 
  updated payments from f71d3fbc7b8331b0427748b3fd358b5e2fc626fa to f09297028acace67588c2de845b754e2ace75c97 | 
  [production] | 
            
  | 15:59 | 
  <thcipriani@tin> | 
  Synchronized wmf-config/CommonSettings.php: SWAT: Match JsonConfig change [[gerrit:282447]] (duration: 00m 25s) | 
  [production] | 
            
  | 15:54 | 
  <thcipriani@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Enable Echo survey on French-language wikis" (duration: 00m 26s) | 
  [production] | 
            
  | 15:40 | 
  <thcipriani@tin> | 
  Synchronized php-1.27.0-wmf.20/extensions/MobileApp/config/config.json: SWAT: Roll out RESTBase usage to Android production app: 100% [[gerrit:282434]] (duration: 00m 27s) | 
  [production] | 
            
  | 15:29 | 
  <thcipriani@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Echo survey on French-language wikis [[gerrit:282414]] (duration: 00m 25s) | 
  [production] | 
            
  | 15:27 | 
  <elukey> | 
  restarted hhvm on mw1190 (hhvm-dump-debug saved in /tmp/hhvm.19289.bt) | 
  [production] | 
            
  | 15:24 | 
  <thcipriani@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: [Cleanup] Remove VisualEditor AutoAccountEnable config now unused [[gerrit:280870]] (duration: 00m 25s) | 
  [production] | 
            
  | 15:20 | 
  <thcipriani@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: [Cleanup] Remove VisualEditor experimental config 2 of 2 [[gerrit:280869]] (duration: 00m 28s) | 
  [production] | 
            
  | 15:20 | 
  <thcipriani@tin> | 
  Synchronized wmf-config/CommonSettings.php: SWAT: [Cleanup] Remove VisualEditor experimental config 1 of 2 [[gerrit:280869]] (duration: 00m 28s) | 
  [production] | 
            
  | 15:13 | 
  <thcipriani@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor on the Project ("Wikipedya") of htwiki [[gerrit:281263]] (duration: 00m 32s) | 
  [production] | 
            
  | 15:10 | 
  <thcipriani@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: Add flood group to ladwiki [[gerrit:282201]] (duration: 00m 29s) | 
  [production] | 
            
  | 15:05 | 
  <thcipriani@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: Add new namespaces and new aliases for newikibooks [[gerrit:281443]] (duration: 00m 45s) | 
  [production] | 
            
  | 14:09 | 
  <elukey> | 
  mw1140 powercycled (ssh not usable, can't access as root via console too) | 
  [production] | 
            
  | 11:14 | 
  <elukey> | 
  mw1148 powercycled, not responsive to ssh | 
  [production] | 
            
  | 10:27 | 
  <elukey> | 
  powercycled mw1144.eqiad.wmnet  | 
  [production] | 
            
  | 08:44 | 
  <volans> | 
  Re-enabling Puppet on cluster mysql and parsercache to deploy change 282385, T111654 | 
  [production] | 
            
  | 08:30 | 
  <godog> | 
  bootstrap restbase1014-b T128107 | 
  [production] | 
            
  | 08:28 | 
  <godog> | 
  start raid expansion on restbase2005 T127951 | 
  [production] | 
            
  | 08:19 | 
  <godog> | 
  repool restbase2002, depool restbase2005 | 
  [production] | 
            
  | 08:18 | 
  <volans> | 
  Disabling Puppet on cluster mysql and parsercache to merge and test change 282385 on db2040, T111654 | 
  [production] | 
            
  | 07:05 | 
  <moritzm> | 
  restarted hhvm on mw1143, mw1201 and mw1213 | 
  [production] | 
            
  | 04:42 | 
  <Krinkle> | 
  mwscript deleteEqualMessages.php --wiki shwiki (T45917) | 
  [production] | 
            
  | 04:34 | 
  <MaxSem> | 
  ran `mwscript extensions/CentralAuth/maintenance/createLocalAccount.php --wiki=enwiki 'Corvin Victor Paul'` due to CA bug | 
  [production] | 
            
  | 02:33 | 
  <l10nupdate@tin> | 
  ResourceLoader cache refresh completed at Mon Apr 11 02:33:52 UTC 2016 (duration 8m 32s) | 
  [production] | 
            
  | 02:25 | 
  <mwdeploy@tin> | 
  sync-l10n completed (1.27.0-wmf.20) (duration: 11m 22s) | 
  [production] |