| 2010-01-25
      
      § | 
    
  | 19:01 | <Fred> | adjusted apache.conf on srv225 to make it work again... | [production] | 
            
  | 16:07 | <Rob> | powering srv224 & srv225 back up | [production] | 
            
  | 16:01 | <Rob> | shutting down srv225 & srv224 to swap their power cables from single to Y split. | [production] | 
            
  | 15:45 | <Rob> | srv196 memory replaced, back online and in service | [production] | 
            
  | 15:45 | <root> | synchronized php-1.5/wmf-config/mc.php  'pushing out change to two hosts that were down a few minutes ago' | [production] | 
            
  | 15:40 | <Rob> | shutting down db28 to swap memory around in diagnostics (bad memory in system) | [production] | 
            
  | 15:25 | <root> | synchronized php-1.5/wmf-config/mc.php  'took down srv196 for hardware replacement' | [production] | 
            
  | 15:20 | <Rob> | shutting down srv196 to replace faulty memory dimm | [production] | 
            
  | 12:19 | <Tim> | started recompressTracked.sh on hume | [production] | 
            
  
    | 2010-01-22
      
      § | 
    
  | 23:08 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative/ClickTracking/ClickTracking.hooks.php  'Deploy r61401' | [production] | 
            
  | 23:07 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative/CollapsibleTabs/CollapsibleTabs.hooks.php  'Deploy r61403' | [production] | 
            
  | 21:29 | <brion> | freed some space on srv218 removing old upload temp files from /tmp but / is still pretty tight (445m free) | [production] | 
            
  | 20:19 | <catrope> | synchronized php-1.5/extensions/WikimediaMessages/WikimediaLicenseTexts.i18n.php  'Deploying r61387' | [production] | 
            
  | 19:05 | <catrope> | synchronized php-1.5/wmf-config/InitialiseSettings.php  'bug 22201: Add WZ: alias on itwiktionary' | [production] | 
            
  | 15:22 | <catrope> | synchronized php-1.5/wmf-config/InitialiseSettings.php  'Enable $wgEnotifWatchlist on usabilitywiki' | [production] | 
            
  | 11:38 | <mark> | Deployed new gmond 3.0.3-2 package that conflicts with the incompatible ganglia 3.1 packages in Ubuntu, to avoid problems in our upcoming upgrade | [production] | 
            
  
    | 2010-01-20
      
      § | 
    
  | 20:36 | <root> | synchronized php-1.5/wmf-config/mc.php | [production] | 
            
  | 16:38 | <Fred> | killed long running queries created by civicrm on db9. | [production] | 
            
  | 13:56 | <domas> | did lots of changes on wikitech wiki. :) | [production] | 
            
  | 13:55 | <domas> | test! | [production] | 
            
  | 02:38 | <catrope> | synchronized php-1.5/wmf-config/InitialiseSettings.php  'Disable API logging on enwiki' | [production] | 
            
  | 01:20 | <tfinc> | synchronized php-1.5/wmf-config/InitialiseSettings.php  'Enabling api logging for enwiki' | [production] | 
            
  | 01:16 | <catrope> | synchronized php-1.5/wmf-config/InitialiseSettings.php  'Disabling API request logging on dewiki' | [production] | 
            
  | 00:03 | <RoanKattouw> | Strike my last, was actually *enabling* it on dewiki | [production] | 
            
  | 00:03 | <catrope> | synchronized php-1.5/wmf-config/InitialiseSettings.php  'Disable API logging test on dewiki' | [production] | 
            
  | 00:02 | <catrope> | synchronized php-1.5/wmf-config/InitialiseSettings.php  'Disable API logging test on frwiki' | [production] | 
            
  
    | 2010-01-19
      
      § | 
    
  | 21:37 | <catrope> | synchronized php-1.5/wmf-config/InitialiseSettings.php  'Enabling API request logging on frwiki' | [production] | 
            
  | 21:36 | <catrope> | synchronized php-1.5/wmf-config/CommonSettings.php  'Enabling API request logging on frwiki' | [production] | 
            
  | 20:34 | <Fred> | bugzilla upgrade completed. It might be up to an hour before attachments can be added due to DNS propagation. | [production] | 
            
  | 19:41 | <Fred> | Bugzilla as you know it is going down. Be back shortly. | [production] | 
            
  | 18:21 | <Rob> | updated blog.wikimedia.org wp-stats plugin to 1.6.1 | [production] | 
            
  | 16:03 | <Rob> | ssds installed in db28, during boot it displays a memory error, investigating. | [production] | 
            
  | 15:20 | <Rob> | db28 offline while disks are swapped to solid state for testing. | [production] |