| 2016-02-27
      
      § | 
    
  | 19:32 | <ori@tin> | Synchronized php-1.27.0-wmf.14/includes/user/User.php: Revert I43cde3a48: Prevent duplicate memcached lookups for user record (T128246) (duration: 00m 56s) | [production] | 
            
  | 04:15 | <ori> | On silver: commented out exception from I3eae28a58 to work around T128246 | [production] | 
            
  | 03:53 | <bd808> | test | [production] | 
            
  | 03:47 | <bd808> | Tried to fix CAS error for Labslogbot | [production] | 
            
  | 03:36 | <bd808> | Backfilled missing missing https://wmflabs.org/sal/production data (2016-02-21T14:39 to 2016-02-24T16:53) | [production] | 
            
  | 03:14 | <urandom> | bootstrap of restbase1008-a.eqiad.wmnet complete; begining `nodetool cleanup' of 1003, 1004-a, and 1008-b : T95253 | [production] | 
            
  | 02:31 | <l10nupdate@tin> | ResourceLoader cache refresh completed at Sat Feb 27 02:31:10 UTC 2016 (duration 7m 44s) | [production] | 
            
  | 02:23 | <mwdeploy@tin> | sync-l10n completed (1.27.0-wmf.14) (duration: 10m 24s) | [production] | 
            
  | 01:50 | <twentyafterfour> | updated phabricator/tools to https://phabricator.wikimedia.org/rPHTO1433a20886a4168d41b81b5ecf5e29687678f2e2 (applying fix for T127524 ) | [production] | 
            
  | 00:52 | <aaron@tin> | Synchronized php-1.27.0-wmf.14/includes/filebackend/FileBackendMultiWrite.php: 7ca624b8b93 (duration: 00m 47s) | [production] | 
            
  
    | 2016-02-26
      
      § | 
    
  | 22:20 | <mutante> | powercycle mw1130 | [production] | 
            
  | 20:42 | <demon@tin> | Synchronized wmf-config/InitialiseSettings.php: update wmf address in echo footer thingie (duration: 00m 59s) | [production] | 
            
  | 20:29 | <hashar> | Restarting Jenkins | [production] | 
            
  | 20:04 | <urandom> | issuing test repair on cerium (restbase staging), keyspace  : T108611 | [production] | 
            
  | 18:41 | <ori@tin> | Synchronized php-1.27.0-wmf.14/includes/user/User.php: I43cde3a48: Prevent duplicate memcached lookups for user record (duration: 01m 02s) | [production] | 
            
  | 16:46 | <chasemp> | labstore1001 'mdadm --manage /dev/md126 --add /dev/sdaf' | [production] | 
            
  | 15:51 | <jynus> | shutting down mariadb on db2030 to clone from db1009 | [production] | 
            
  | 15:24 | <urandom> | forcing puppet run on restbase1009.eqiad.wmnet | [production] | 
            
  | 15:24 | <urandom> | re-enabling puppet on restbase1009.eqiad.wmnet | [production] | 
            
  | 15:20 | <jynus> | performing backup of m5-master mysql data | [production] | 
            
  | 15:17 | <urandom> | blocking CQL native port on restbase1009.eqiad.wmnet : https://phabricator.wikimedia.org/P2677 | [production] | 
            
  | 15:14 | <urandom> | disabling puppet on restbase1009.eqiad to preserve local changes during a quick experiment | [production] | 
            
  | 15:03 | <hashar> | Switched MediaWiki core npm test to Nodepool instance T119143 | [production] | 
            
  | 13:59 | <krinkle@tin> | Synchronized wmf-config/InitialiseSettings.php: T99096: Enable wmgUseWmfstatic on remaining wikis (duration: 00m 50s) | [production] | 
            
  | 13:54 | <moritzm> | rebooting lithium for kernel update | [production] | 
            
  | 13:26 | <godog> | launch swiftrepl continuous replication for unsharded containers on ms-fe1003 T128096 | [production] | 
            
  | 12:31 | <elukey> | added mc1017/mc1018 back to the redis/memcached pools after maintenance | [production] | 
            
  | 11:42 | <godog> | run swiftrepl eqiad -> codfw for unsharded containers | [production] | 
            
  | 11:01 | <elukey> | removed mc1018/1017 from the redis memcached pools for maintenance | [production] | 
            
  | 09:46 | <elukey> | mc1016.eqiad re-added to the memcached/redis pools after maintenance | [production] | 
            
  | 08:12 | <elukey> | removed mc1016.eqiad from the redis/memcached pools for maintenance | [production] | 
            
  | 08:01 | <moritzm> | blacklisting aufs kernel module | [production] | 
            
  | 02:32 | <l10nupdate@tin> | ResourceLoader cache refresh completed at Fri Feb 26 02:32:19 UTC 2016 (duration 7m 42s) | [production] | 
            
  | 02:24 | <mwdeploy@tin> | sync-l10n completed (1.27.0-wmf.14) (duration: 10m 34s) | [production] | 
            
  | 01:06 | <catrope@tin> | Synchronized wmf-config/InitialiseSettings.php: Lower survey rate again (duration: 01m 05s) | [production] | 
            
  | 00:33 | <catrope@tin> | Synchronized php-1.27.0-wmf.14/extensions/MobileFrontend/: SWAT (duration: 01m 05s) | [production] | 
            
  | 00:31 | <catrope@tin> | Synchronized wmf-config/InitialiseSettings.php: Raise file upload limit to 2047MB (duration: 01m 02s) | [production] | 
            
  | 00:22 | <catrope@tin> | Synchronized wmf-config/CommonSettings.php: Add plumbing for wmgUseGraphWithJsonNamespace (duration: 01m 03s) | [production] | 
            
  | 00:21 | <catrope@tin> | Synchronized wmf-config/InitialiseSettings.php: Add wmgUseGraphWithJsonNamespace (duration: 01m 04s) | [production] | 
            
  | 00:15 | <catrope@tin> | Synchronized wmf-config/InitialiseSettings.php: Run reader segmentation survey at 1:500 to test DNT (duration: 01m 21s) | [production] | 
            
  
    | 2016-02-25
      
      § | 
    
  | 23:16 | <bblack> | turning puppet back on for cp*, pushing changes through https://gerrit.wikimedia.org/r/#/c/273385/ | [production] | 
            
  | 22:33 | <bblack> | disabling puppet on caches for more scary VCL merges | [production] | 
            
  | 22:33 | <ori@tin> | Synchronized php-1.27.0-wmf.14/extensions/CentralAuth: I2cfcbf98f3: Reduce memcache traffic for central session storage (duration: 01m 21s) | [production] | 
            
  | 22:06 | <bblack> | turning puppet back on for cp*, pushing changes through https://gerrit.wikimedia.org/r/273217 to all | [production] | 
            
  | 21:59 | <thcipriani@tin> | Synchronized php-1.27.0-wmf.14/extensions/CirrusSearch/includes/InterwikiSearcher.php: Fix undefined variable $term in InterwikiSearcher [[gerrit:273369]] (duration: 01m 08s) | [production] | 
            
  | 21:49 | <ori> | Upgraded Grafana to v3.0.0-pre1. | [production] | 
            
  | 21:28 | <thcipriani@tin> | rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.27.0-wmf.14 | [production] | 
            
  | 21:21 | <ebernhardson@tin> | Synchronized php-1.27.0-wmf.14/extensions/CirrusSearch/includes/Searcher.php: Update file that wasnt synced properly (duration: 01m 50s) | [production] | 
            
  | 20:57 | <bblack> | disabling puppet on caches for scarier VCL merges | [production] | 
            
  | 20:46 | <urandom> | starting bootstrap of restbase1008-a T119935 | [production] |