| 2012-10-15
      
      § | 
    
  | 16:06 | <logmsgbot_> | reedy Finished syncing Wikimedia installation... : Build localisation cache for 1.21wmf2 | [production] | 
            
  | 15:43 | <ottomata> | stopping puppet on oxygen, looking for cause of recent packet loss | [production] | 
            
  | 15:19 | <logmsgbot_> | reedy synchronized php-1.21wmf2/extensions/MobileFrontend | [production] | 
            
  | 15:17 | <logmsgbot_> | reedy synchronized php-1.21wmf1/extensions/MobileFrontend | [production] | 
            
  | 15:12 | <logmsgbot_> | reedy Started syncing Wikimedia installation... : Build localisation cache for 1.21wmf2 | [production] | 
            
  | 14:56 | <logmsgbot_> | reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.21wmf2 | [production] | 
            
  | 14:55 | <logmsgbot_> | reedy synchronized wmf-config/ | [production] | 
            
  | 14:52 | <logmsgbot_> | reedy synchronized php-1.21wmf2/  'Initial sync' | [production] | 
            
  | 13:35 | <Reedy> | Killing php-1.20wmf11 l10n cache from apaches | [production] | 
            
  | 13:34 | <Reedy> | Killing php-1.20wmf11 l10n cache from apaches | [production] | 
            
  | 13:31 | <Reedy> | Pointing php to php-1.21wmf1 | [production] | 
            
  | 13:29 | <logmsgbot_> | reedy synchronized wmf-config/throttle.php  'Updated from gitrepo' | [production] | 
            
  | 13:28 | <logmsgbot_> | reedy synchronized wmf-config/CommonSettings.php  'Remove extdist conditional for mount path' | [production] | 
            
  | 11:49 | <mark> | Deployed range_offset_limit 64 MB change (from 0) to all upload backend squids | [production] | 
            
  | 11:24 | <paravoid> | restarting pdns @ ns2 | [production] | 
            
  | 11:16 | <paravoid> | switching European bits back to esams | [production] | 
            
  | 09:49 | <paravoid> | restarting all imagescaler apaches, did not recover after swift outage | [production] | 
            
  | 09:24 | <paravoid> | live-hacking swift on ms-be10, ms-be12, ms-be1 to remove fadvise calls | [production] | 
            
  | 07:53 | <Tim> | increasing worker count to 100 on all swift backends, via puppet | [production] | 
            
  | 07:37 | <Tim> | ms-be11 showed an immediate improvement in bandwidth out, but wchan still indicates that 48 is not enough, increasing to 100 | [production] | 
            
  | 07:33 | <Tim> | experimentally doubling the worker count on ms-be11 since wchan indicates that the worker pool is exhausted | [production] | 
            
  | 05:49 | <apergos> | ms-be12 reporting errors on disk /dev/sdi, I've rt'ed it an unmounted it in the meantime | [production] | 
            
  | 04:34 | <Tim> | on ms-fe2: restarted swift-proxy due to mem leak | [production] | 
            
  | 04:28 | <Tim> | on ms-fe3: restarting swift-proxy due to swap | [production] | 
            
  | 04:26 | <LeslieCarr> | restarted swift proxy on ms-fe4 - it was in swap doom | [production] | 
            
  | 02:25 | <logmsgbot_> | LocalisationUpdate completed (1.21wmf1) at Mon Oct 15 02:25:12 UTC 2012 | [production] | 
            
  
    | 2012-10-13
      
      § | 
    
  | 21:39 | <logmsgbot_> | preilly Finished syncing Wikimedia installation... : update zero rated mobile access | [production] | 
            
  | 21:18 | <logmsgbot_> | preilly Started syncing Wikimedia installation... : update zero rated mobile access | [production] | 
            
  | 21:06 | <logmsgbot_> | preilly synchronized php-1.21wmf1/extensions/MobileFrontend  'update for landing page' | [production] | 
            
  | 18:51 | <logmsgbot_> | reedy synchronized php-1.21wmf1/extensions/Collection  'Revert to master' | [production] | 
            
  | 18:41 | <logmsgbot_> | reedy synchronized php-1.21wmf1/extensions/Collection  'Revert to original again..' | [production] | 
            
  | 18:38 | <logmsgbot_> | reedy synchronized php-1.21wmf1/extensions/Collection  'Revert to f980f2380b830e712edfdbce065784d1bcba82fd' | [production] | 
            
  | 18:36 | <logmsgbot_> | reedy synchronized php-1.21wmf1/extensions/Collection  'Revert to 4f898444ce5d9c2c9a5491f2303db15a3143a5c5' | [production] | 
            
  | 18:34 | <logmsgbot_> | reedy synchronized php-1.21wmf1/extensions/Collection  'Revert to 72b7d4d645489dd092605a2183a944fac33cb40b' | [production] | 
            
  | 18:26 | <logmsgbot_> | reedy synchronized php-1.21wmf1/extensions/Collection  'Revert to original' | [production] | 
            
  | 18:25 | <logmsgbot_> | reedy synchronized php-1.21wmf1/extensions/Collection  'Revert to 8415a4564bf429b76e113368a3e5e1c19fb84a47' | [production] | 
            
  | 18:23 | <logmsgbot_> | reedy synchronized php-1.21wmf1/extensions/Collection  'Revert to 72b7d4d645489dd092605a2183a944fac33cb40b' | [production] | 
            
  | 18:21 | <logmsgbot_> | reedy synchronized php-1.21wmf1/extensions/Collection  'Revert to 9f37005e78a068821c45d9d313394c6728f55b4a' | [production] | 
            
  | 16:34 | <logmsgbot_> | reedy synchronized php-1.21wmf1/extensions/Collection | [production] | 
            
  | 16:27 | <logmsgbot_> | reedy synchronized php-1.21wmf1/extensions/Collection | [production] | 
            
  | 13:17 | <logmsgbot_> | reedy synchronized wmf-config/throttle.php | [production] | 
            
  | 02:00 | <logmsgbot_> | LocalisationUpdate failed: git pull of extensions failed | [production] |