| 2010-10-24
      
      § | 
    
  | 20:44 | <mark> | Shutting down mayflower for decommissioning | [production] | 
            
  | 20:40 | <mark> | Changed root password on lily, iris, ragweed, mayflower and sage | [production] | 
            
  | 20:37 | <mark> | Changed root password on browne and srv2 | [production] | 
            
  | 19:31 | <RobH> | installed python-openssl on spence for testing | [production] | 
            
  | 19:11 | <tomaszf> | removing downtime with init script on loudon | [production] | 
            
  | 18:21 | <midom> | synchronized php-1.5/wmf-config/db.php | [production] | 
            
  | 18:20 | <midom> | synchronized php-1.5/wmf-config/db.php | [production] | 
            
  | 17:44 | <Ryan_Lane> | reinstalling ersch for openstack testing | [production] | 
            
  | 17:29 | <mark> | Changed root password hash in puppet; being deployed to all systems in the next 30-60 minutes | [production] | 
            
  | 17:21 | <midom> | synchronized php-1.5/wmf-config/db.php | [production] | 
            
  | 16:56 | <Ryan_Lane> | reinstalling alsted, to try stable openstack from scratch | [production] | 
            
  | 16:43 | <Ryan_Lane> | rebooting ersch and alsted | [production] | 
            
  | 14:26 | <midom> | synchronized php-1.5/wmf-config/db.php | [production] | 
            
  | 14:26 | <midom> | synchronized php-1.5/wmf-config/db.php | [production] | 
            
  | 12:34 | <mark> | Restarted varnish on knsq5 | [production] | 
            
  | 12:33 | <mark> | Restarted varnish on knsq1 | [production] | 
            
  | 12:31 | <mark> | Powercycled knsq4 | [production] | 
            
  | 12:31 | <mark> | Restarted varnish on knsq2 | [production] | 
            
  | 12:31 | <mark> | Removed puppet lock on knsq2 | [production] | 
            
  
    | 2010-10-23
      
      § | 
    
  | 20:48 | <domas> | powercycled srv154, saw mgmt network problems yet again | [production] | 
            
  | 20:30 | <tfinc> | synchronized php-1.5/wmf-config/InitialiseSettings.php  'Turning of CentralNotice for qualitywiki' | [production] | 
            
  | 18:32 | <RobH> | modifying the sync-common-all to take out logging while I am futzing with it. | [production] | 
            
  | 18:27 | <root> | ran sync-common-all | [production] | 
            
  | 18:24 | <root> | ran sync-common-all | [production] | 
            
  | 18:21 | <root> | ran sync-common-all | [production] | 
            
  | 18:19 | <root> | ran sync-common-all | [production] | 
            
  | 18:17 | <root> | ran sync-common-all | [production] | 
            
  | 18:13 | <domas> | increased maxclients 10->20 on scalers | [production] | 
            
  | 18:10 | <root> | ran sync-common-all | [production] | 
            
  | 17:37 | <Ryan_Lane> | updating the dns entry for svn.wikimedia.org | [production] | 
            
  | 17:37 | <Ryan_Lane> | rsynd'd /svnroot to formey | [production] | 
            
  | 17:37 | <Ryan_Lane> | remounted /svnroot ro on mayflower | [production] | 
            
  | 17:37 | <Ryan_Lane> | moving svn from mayflower to formey | [production] | 
            
  | 17:03 | <domas> | restarted apaches that failed to start after apt-get upgrade :( | [production] | 
            
  | 17:01 | <root> | ran sync-common-all | [production] | 
            
  | 16:58 | <RobH> | working on rsync settings and running crap as root for testing, disregard | [production] | 
            
  | 16:57 | <tomaszf> | upgrading activemq to 5.4.1 on erzurumi | [production] | 
            
  | 16:51 | <tstarling> | synchronized php-1.5/includes/memcached-client.php  'profiling memcached compression' | [production] | 
            
  | 16:48 | <tstarling> | synchronized php-1.5/includes/memcached-client.php  'profiling memcached compression' | [production] | 
            
  | 16:46 | <catrope> | synchronized php-1.5/extensions/CodeReview/backend/RepoStats.php  'r75268' | [production] | 
            
  | 16:36 | <root> | ran sync-common-all | [production] | 
            
  | 16:33 | <domas> | upgrading scalers to new imagemagick and whatever is pending | [production] | 
            
  | 16:20 | <RobH> | having to run the sync as root to fix permissions with roan, not normal use | [production] | 
            
  | 16:19 | <root> | ran sync-common-all | [production] | 
            
  | 16:10 | <root> | ran sync-common-all | [production] | 
            
  | 15:22 | <mark> | Deployed wikimedia.vcl (varnish) which fixes a memleak in the geoip code | [production] |