| 2015-08-29
      
      § | 
    
  | 15:26 | <jynus> | killing idle mysql connections from phabricator and setting wait and interactive timeout to 60 | [production] | 
            
  | 09:30 | <jynus> | SCAP failed, cannot depool db1028 | [production] | 
            
  | 09:28 | <jynus@tin> | Synchronized wmf-config/db-eqiad.php: Depool db1028, return ES servers back from maintenance (duration: 00m 03s) | [production] | 
            
  | 09:28 | <jynus@tin> | Synchronized wmf-config/db-eqiad.php: Depool db1028, return ES servers back from maintenance (duration: 00m 03s) | [production] | 
            
  | 09:05 | <jynus> | about to depool db1028 due to disk issue | [production] | 
            
  | 04:17 | <l10nupdate@tin> | ResourceLoader cache refresh completed at Sat Aug 29 04:17:55 UTC 2015 (duration 17m 54s) | [production] | 
            
  | 02:24 | <l10nupdate@tin> | LocalisationUpdate completed (1.26wmf20) at 2015-08-29 02:24:01+00:00 | [production] | 
            
  | 02:21 | <l10nupdate@tin> | Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 05m 48s) | [production] | 
            
  
    | 2015-08-28
      
      § | 
    
  | 23:44 | <krenair@tin> | Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234679/ (duration: 06m 56s) | [production] | 
            
  | 22:51 | <bd808@tin> | Synchronized wmf-config/CommonSettings-labs.php: Use ffmpeg instead of avconv on labs beta (I250fe33) (duration: 06m 05s) | [production] | 
            
  | 22:05 | <ori> | disabling puppet on tin for a few minutes to test an ssh-agent-proxy change | [production] | 
            
  | 20:04 | <catrope@tin> | Synchronized php-1.26wmf20/resources/src/mediawiki.legacy/shared.css: T110716 (duration: 00m 12s) | [production] | 
            
  | 18:09 | <robh> | updating ldap-codfw cert | [production] | 
            
  | 17:10 | <catrope@tin> | Synchronized php-1.26wmf20/extensions/Flow/includes/Parsoid/Utils.php: T110676 (duration: 00m 13s) | [production] | 
            
  | 17:08 | <urandom> | bouncing Cassandra on restbase1001 to apply default (puppet-managed) settings | [production] | 
            
  | 16:03 | <chasemp> | ferm for elasticsearch10(0[8-9|1[0-13]) | [production] | 
            
  | 15:23 | <chasemp> | ferm for elasticsearch10[14-17] | [production] | 
            
  | 11:09 | <aude@tin> | Synchronized php-1.26wmf20/extensions/Wikidata/Wikidata.php: Sync entry point - updated to work on Jenkins together with ContentTranslation (duration: 00m 12s) | [production] | 
            
  | 10:29 | <godog> | reenable puppet on ms-fe1, ferm changes will go out on monday | [production] | 
            
  | 09:48 | <jynus> | Cloning es1001 database into es1012 | [production] | 
            
  | 09:45 | <moritzm> | enabled ferm for swift on esams | [production] | 
            
  | 09:28 | <moritzm> | enabled ferm on strontium puppetmaster backend | [production] | 
            
  | 09:00 | <moritzm> | enabled ferm on rhodium puppetmaster backend | [production] | 
            
  | 08:29 | <moritzm> | uploaded debdeploy 0.0.3 to carbon | [production] | 
            
  | 08:23 | <jynus@tin> | Synchronized wmf-config/db-eqiad.php: Depool es1001, increas weight of es1011, pool es1014 for the first time (duration: 00m 13s) | [production] | 
            
  | 05:59 | <l10nupdate@tin> | ResourceLoader cache refresh completed at Fri Aug 28 05:59:09 UTC 2015 (duration 59m 8s) | [production] | 
            
  | 04:58 | <ori@tin> | Synchronized php-1.26wmf20/includes/parser/Parser.php: 754b222daf: Add ParserOutput cache and expiry times to NewPP report (duration: 00m 13s) | [production] | 
            
  | 02:41 | <l10nupdate@tin> | LocalisationUpdate completed (1.26wmf20) at 2015-08-28 02:41:26+00:00 | [production] | 
            
  | 02:35 | <l10nupdate@tin> | Synchronized php-1.26wmf20/cache/l10n: l10nupdate for 1.26wmf20 (duration: 10m 47s) | [production] | 
            
  | 01:59 | <TimStarling> | on ruthenium: started parsoid_vd which was previously killed by oom-killer | [production] | 
            
  | 01:58 | <TimStarling> | on ruthenium, reduced parsoid-rt-client concurrency from 16 to 8 since it was OOM and oom-killer was killing random things | [production] | 
            
  | 01:37 | <TimStarling> | on ruthenium restarted parsoid-rt-client and parsoid-vd-client | [production] | 
            
  | 00:24 | <mutante> | powercycled mw2027 | [production] | 
            
  | 00:19 | <krenair@tin> | Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/234450/ (duration: 01m 14s) | [production] | 
            
  | 00:05 | <krenair@tin> | Synchronized wmf-config/mobile.php: live hack to make previous commit work (duration: 01m 14s) | [production] | 
            
  | 00:05 | <Krenair> | Another codfw host broke: mw2027 | [production] | 
            
  | 00:01 | <krenair@tin> | Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/234330/ (duration: 00m 13s) | [production] | 
            
  
    | 2015-08-27
      
      § | 
    
  | 23:58 | <krenair@tin> | Synchronized php-1.26wmf20/extensions/MobileFrontend/includes/MobileFormatter.php: https://gerrit.wikimedia.org/r/#/c/234331/1 (duration: 00m 12s) | [production] | 
            
  | 23:57 | <krenair@tin> | Synchronized php-1.26wmf20/extensions/MobileFrontend/includes/config/Experimental.php: https://gerrit.wikimedia.org/r/#/c/234331/1 (duration: 00m 14s) | [production] | 
            
  | 23:55 | <krenair@tin> | Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/233439/ (duration: 00m 12s) | [production] | 
            
  | 23:29 | <krenair@tin> | Synchronized php-1.26wmf20/extensions/Gadgets/extension.json: touch (duration: 00m 13s) | [production] | 
            
  | 23:24 | <krenair@tin> | Synchronized php-1.26wmf20/includes/DefaultSettings.php: https://gerrit.wikimedia.org/r/#/c/234328/ (duration: 00m 12s) | [production] | 
            
  | 23:24 | <krenair@tin> | Synchronized php-1.26wmf20/includes/registration/ExtensionProcessor.php: https://gerrit.wikimedia.org/r/#/c/234328/ (duration: 00m 12s) | [production] | 
            
  | 23:23 | <krenair@tin> | Synchronized php-1.26wmf20/includes/MWNamespace.php: https://gerrit.wikimedia.org/r/#/c/234328/ (duration: 00m 13s) | [production] | 
            
  | 23:15 | <krenair@tin> | Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/234009/ (duration: 00m 13s) | [production] | 
            
  | 23:04 | <krenair@tin> | Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/233100/ (duration: 00m 12s) | [production] | 
            
  | 20:11 | <chasemp> | ferm setup on elasticsearch10(1[8-9|2[0-3]) | [production] | 
            
  | 20:06 | <twentyafterfour@tin> | rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedia wikis to 1.26wmf20 | [production] | 
            
  | 19:57 | <twentyafterfour@tin> | Synchronized php-1.26wmf20/includes/media/XMP.php: deploy fix for T89532 on 1.26wmf20 (duration: 00m 13s) | [production] | 
            
  | 18:16 | <chasemp> | setting up ferm on elastic1027-31 | [production] |