| 
      
        2010-10-24
      
      §
     | 
  
    
  | 20:44 | 
  <mark> | 
  Shutting down mayflower for decommissioning | 
  [production] | 
            
  | 20:40 | 
  <mark> | 
  Changed root password on lily, iris, ragweed, mayflower and sage | 
  [production] | 
            
  | 20:37 | 
  <mark> | 
  Changed root password on browne and srv2 | 
  [production] | 
            
  | 19:31 | 
  <RobH> | 
  installed python-openssl on spence for testing | 
  [production] | 
            
  | 19:11 | 
  <tomaszf> | 
  removing downtime with init script on loudon | 
  [production] | 
            
  | 18:21 | 
  <midom> | 
  synchronized php-1.5/wmf-config/db.php  | 
  [production] | 
            
  | 18:20 | 
  <midom> | 
  synchronized php-1.5/wmf-config/db.php  | 
  [production] | 
            
  | 17:44 | 
  <Ryan_Lane> | 
  reinstalling ersch for openstack testing | 
  [production] | 
            
  | 17:29 | 
  <mark> | 
  Changed root password hash in puppet; being deployed to all systems in the next 30-60 minutes | 
  [production] | 
            
  | 17:21 | 
  <midom> | 
  synchronized php-1.5/wmf-config/db.php  | 
  [production] | 
            
  | 16:56 | 
  <Ryan_Lane> | 
  reinstalling alsted, to try stable openstack from scratch | 
  [production] | 
            
  | 16:43 | 
  <Ryan_Lane> | 
  rebooting ersch and alsted | 
  [production] | 
            
  | 14:26 | 
  <midom> | 
  synchronized php-1.5/wmf-config/db.php  | 
  [production] | 
            
  | 14:26 | 
  <midom> | 
  synchronized php-1.5/wmf-config/db.php  | 
  [production] | 
            
  | 12:34 | 
  <mark> | 
  Restarted varnish on knsq5 | 
  [production] | 
            
  | 12:33 | 
  <mark> | 
  Restarted varnish on knsq1 | 
  [production] | 
            
  | 12:31 | 
  <mark> | 
  Powercycled knsq4 | 
  [production] | 
            
  | 12:31 | 
  <mark> | 
  Restarted varnish on knsq2 | 
  [production] | 
            
  | 12:31 | 
  <mark> | 
  Removed puppet lock on knsq2 | 
  [production] | 
            
  
    | 
      
        2010-10-23
      
      §
     | 
  
    
  | 20:48 | 
  <domas> | 
  powercycled srv154, saw mgmt network problems yet again | 
  [production] | 
            
  | 20:30 | 
  <tfinc> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  'Turning of CentralNotice for qualitywiki' | 
  [production] | 
            
  | 18:32 | 
  <RobH> | 
  modifying the sync-common-all to take out logging while I am futzing with it. | 
  [production] | 
            
  | 18:27 | 
  <root> | 
  ran sync-common-all  | 
  [production] | 
            
  | 18:24 | 
  <root> | 
  ran sync-common-all  | 
  [production] | 
            
  | 18:21 | 
  <root> | 
  ran sync-common-all  | 
  [production] | 
            
  | 18:19 | 
  <root> | 
  ran sync-common-all  | 
  [production] | 
            
  | 18:17 | 
  <root> | 
  ran sync-common-all  | 
  [production] | 
            
  | 18:13 | 
  <domas> | 
  increased maxclients 10->20 on scalers | 
  [production] | 
            
  | 18:10 | 
  <root> | 
  ran sync-common-all  | 
  [production] | 
            
  | 17:37 | 
  <Ryan_Lane> | 
  updating the dns entry for svn.wikimedia.org | 
  [production] | 
            
  | 17:37 | 
  <Ryan_Lane> | 
  rsynd'd /svnroot to formey | 
  [production] | 
            
  | 17:37 | 
  <Ryan_Lane> | 
  remounted /svnroot ro on mayflower | 
  [production] | 
            
  | 17:37 | 
  <Ryan_Lane> | 
  moving svn from mayflower to formey | 
  [production] | 
            
  | 17:03 | 
  <domas> | 
  restarted apaches that failed to start after apt-get upgrade :( | 
  [production] | 
            
  | 17:01 | 
  <root> | 
  ran sync-common-all  | 
  [production] | 
            
  | 16:58 | 
  <RobH> | 
  working on rsync settings and running crap as root for testing, disregard | 
  [production] | 
            
  | 16:57 | 
  <tomaszf> | 
  upgrading activemq to 5.4.1 on erzurumi | 
  [production] | 
            
  | 16:51 | 
  <tstarling> | 
  synchronized php-1.5/includes/memcached-client.php  'profiling memcached compression' | 
  [production] | 
            
  | 16:48 | 
  <tstarling> | 
  synchronized php-1.5/includes/memcached-client.php  'profiling memcached compression' | 
  [production] | 
            
  | 16:46 | 
  <catrope> | 
  synchronized php-1.5/extensions/CodeReview/backend/RepoStats.php  'r75268' | 
  [production] | 
            
  | 16:36 | 
  <root> | 
  ran sync-common-all  | 
  [production] | 
            
  | 16:33 | 
  <domas> | 
  upgrading scalers to new imagemagick and whatever is pending | 
  [production] | 
            
  | 16:20 | 
  <RobH> | 
  having to run the sync as root to fix permissions with roan, not normal use | 
  [production] | 
            
  | 16:19 | 
  <root> | 
  ran sync-common-all  | 
  [production] | 
            
  | 16:10 | 
  <root> | 
  ran sync-common-all  | 
  [production] |