| 2009-06-08
      
      § | 
    
  | 16:48 | <tstarling> | synchronized php-1.5/db.php | [production] | 
            
  | 16:47 | <tstarling> | synchronized php-1.5/db.php | [production] | 
            
  | 16:19 | <tstarling> | synchronized php-1.5/db.php  'repooled the current batch of servers. db24 is still lagged.' | [production] | 
            
  | 16:07 | <tstarling> | synchronized php-1.5/db.php | [production] | 
            
  | 10:48 | <tstarling> | synchronized php-1.5/db.php  'depooled db4, db7, db23, db24, db18, db21 for schema updates. db12 will do the enwiki query groups.' | [production] | 
            
  | 10:38 | <tstarling> | synchronized php-1.5/db.php  'repooled db12, db3, db8, db15' | [production] | 
            
  | 10:11 | <tstarling> | synchronized php-1.5/db.php  'repooled db5, db17' | [production] | 
            
  | 10:01 | <Tim> | adding AbuseFilter tables to all wikis that don't have them | [production] | 
            
  | 09:59 | <tstarling> | synchronized php-1.5/extensions/AbuseFilter/abusefilter.tables.sql | [production] | 
            
  | 08:54 | <tstarling> | synchronized php-1.5/db.php  'removed db12 from query groups (was 1%)' | [production] | 
            
  | 07:27 | <Tim> | depooled db12, db3, db8, db15, db17 for schema updates | [production] | 
            
  | 07:27 | <tstarling> | synchronized php-1.5/db.php | [production] | 
            
  | 06:46 | <Tim> | running DB updates on db5. All updates today are done by maintenance/update-2009-06-08.php running in a screen on zwinger | [production] | 
            
  | 06:16 | <tstarling> | synchronized php-1.5/maintenance/archives/patch-log_user_text.sql | [production] | 
            
  | 06:12 | <tstarling> | synchronized php-1.5/maintenance/archives/patch-log_user_text.sql | [production] | 
            
  | 06:01 | <Tim> | depooled db5 for schema update. Makes a good guinea pig since it has the lowest disk free space. | [production] | 
            
  | 06:00 | <tstarling> | synchronized php-1.5/db.php | [production] | 
            
  | 05:51 | <tstarling> | synchronized php-1.5/db.php  'repooled db29, depooled on Feb 2 by Domas "for some testing"' | [production] | 
            
  | 05:43 | <tstarling> | synchronized php-1.5/db.php | [production] | 
            
  | 05:38 | <Tim> | fixed nagios configuration, had many errors preventing sync | [production] | 
            
  | 05:23 | <Tim> | disk space critical on storage2. Deleted ~600 GB of files from 2008: all 2008 backups except those that come from wikis that are not in all.dblist | [production] | 
            
  | 05:04 | <Tim> | stopped ES slave on srv171, disk critical. ms2/ms3 have been reasonably stable replacements. | [production] | 
            
  | 04:58 | <Tim> | cleaned up binlogs on db13, was disk space critical | [production] | 
            
  
    | 2009-06-04
      
      § | 
    
  | 18:48 | <Rob> | all active memcached servers now online | [production] | 
            
  | 18:47 | <robh> | synchronized php-1.5/mc-pmtpa.php  'decommissioned srv90 due to bad fans' | [production] | 
            
  | 18:46 | <Rob> | fixing memcached | [production] | 
            
  | 18:43 | <Rob> | pulled srv90 and srv67 for decommissioning | [production] | 
            
  | 18:43 | <mark> | Rebooting iris which has a bad disk | [production] | 
            
  | 18:31 | <Rob> | replaced disk in both sq26 and sq47 | [production] | 
            
  | 18:01 | <Rob> | shutdown sq47 for bad disk | [production] | 
            
  | 18:01 | <Rob> | shutting down sq26 to replace bad disk | [production] | 
            
  | 17:01 | <Fred> | temporarily re-enabled deletion for OTRS while the Junk queue is getting cleaned. | [production] | 
            
  | 13:09 | <mark> | Moved back traffic to esams | [production] |