| 
      
        2010-01-28
      
      §
     | 
  
    
  | 22:58 | 
  <domas> | 
  test | 
  [production] | 
            
  | 20:15 | 
  <Rob> | 
  racked sq51-sq68, all drac accessible, wired, ready for install | 
  [production] | 
            
  | 19:05 | 
  <Rob> | 
  updated dns for management ranges on new squid servers | 
  [production] | 
            
  | 13:56 | 
  <mark> | 
  Added DNS entries for 36 new Squid servers | 
  [production] | 
            
  | 13:28 | 
  <mark> | 
  Storage1 went down due to I/O failure | 
  [production] | 
            
  | 13:00 | 
  <catrope> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  'Remove aliases made obsolete by r61632' | 
  [production] | 
            
  | 12:55 | 
  <catrope> | 
  synchronized php-1.5/languages/messages/MessagesMl.php  'Deploying r61632' | 
  [production] | 
            
  | 10:57 | 
  <aaron> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  'supressredirect for rollbackers on ruwikisource' | 
  [production] | 
            
  | 09:30 | 
  <aaron> | 
  synchronized php-1.5/wmf-config/flaggedrevs.php  'Removed redundant huwiki settings' | 
  [production] | 
            
  
    | 
      
        2010-01-27
      
      §
     | 
  
    
  | 18:04 | 
  <mark> | 
  Fixed ExtensionDistributor, which broke due to the image server migration | 
  [production] | 
            
  | 18:02 | 
  <mark> | 
  synchronized php-1.5/extensions/ExtensionDistributor/svn-invoker.php  's/upload5/upload6/' | 
  [production] | 
            
  | 16:10 | 
  <mark> | 
  Added a timeout to puppet's apt-get update execution; hopefully this will prevent things from getting stuck for weeks | 
  [production] | 
            
  | 16:08 | 
  <mark> | 
  Shutdown sq20 once again, please decommission this broken box | 
  [production] | 
            
  | 16:05 | 
  <mark> | 
  Fixed puppet on db7 | 
  [production] | 
            
  | 16:05 | 
  <mark> | 
  Killed stuck apt-get update processes on all servers | 
  [production] | 
            
  | 15:58 | 
  <mark> | 
  Made puppet manage NFS mounts /mnt/upload6, /mnt/thumbs and the absence of /mnt/upload5 on all application servers and image scalers | 
  [production] | 
            
  | 13:37 | 
  <catrope> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  'Fix Category talk alias on mlwiki' | 
  [production] | 
            
  | 11:57 | 
  <catrope> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  'Also add a Unicode 5.1 alias for Category talk on mlwiki' | 
  [production] | 
            
  | 11:52 | 
  <catrope> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  'bug 22286: Unicode 5.1 namespace alias for Category on mlwiki' | 
  [production] | 
            
  | 11:46 | 
  <catrope> | 
  synchronized php-1.5/includes/api/ApiQueryBacklinks.php  'Deploying r61571' | 
  [production] | 
            
  | 08:12 | 
  <andrew> | 
  synchronized php-1.5/extensions/LiquidThreads/lqt.js  | 
  [production] | 
            
  | 08:11 | 
  <andrew> | 
  synchronized php-1.5/extensions/LiquidThreads_alpha/lqt.js  | 
  [production] | 
            
  
    | 
      
        2010-01-26
      
      §
     | 
  
    
  | 23:44 | 
  <fvassard> | 
  synchronized php-1.5/wmf-config/CommonSettings.php  'Enable file uploads.' | 
  [production] | 
            
  | 23:34 | 
  <Fred> | 
  restarted morebot | 
  [production] | 
            
  | 21:10 | 
  <atglenn> | 
  copying uploaded data since last replication, from ms1 to ms7 | 
  [production] | 
            
  | 21:05 | 
  <atglenn> | 
  turned off file  uploads temporarily | 
  [production] | 
            
  | 21:04 | 
  <ariel> | 
  synchronized wmf-deployment/wmf-config/CommonSettings.php  | 
  [production] | 
            
  | 20:50 | 
  <Rob> | 
  rebooted srv125 per dc task entry by fred | 
  [production] | 
            
  | 20:35 | 
  <tomaszf> | 
  enabling global file uploads notice | 
  [production] | 
            
  | 20:29 | 
  <catrope> | 
  synchronized php-1.5/extensions/GlobalUsage/GlobalUsage.php  'Deploying r61534' | 
  [production] | 
            
  | 20:29 | 
  <catrope> | 
  synchronized php-1.5/extensions/GlobalUsage/GlobalUsageHooks.php  'Deploying r61534' | 
  [production] | 
            
  | 18:38 | 
  <Rob> | 
  shutting down pdf2 to relocate its rack position. | 
  [production] | 
            
  | 18:37 | 
  <Rob> | 
  !log mobile2 port not moved yet, as its port assignment on wikitech is invalid | 
  [production] | 
            
  | 18:08 | 
  <Rob> | 
  shutting down mobile2 to relocate its rack location | 
  [production] | 
            
  | 18:07 | 
  <Rob> | 
  bayes back online | 
  [production] | 
            
  | 18:00 | 
  <Rob> | 
  bayes management lom had ip issues, resolved, still working on host | 
  [production] | 
            
  | 17:52 | 
  <Rob> | 
  bayes is not coming back online properly, checking. | 
  [production] | 
            
  | 17:47 | 
  <Rob> | 
  bayes rack location moved and updated in racktables, powering back up | 
  [production] | 
            
  | 17:34 | 
  <Rob> | 
  shutting down bayes to move its rack location from b2-pmtpa to a2-pmtpa | 
  [production] | 
            
  | 14:42 | 
  <catrope> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  'bug 22274: Set $wgAutoConfirmCount to 10 on itwiktionary' | 
  [production] | 
            
  | 00:56 | 
  <Tim> | 
  on hume: increased RCT proc count, reduced max lag to 1s, increased lag check frequency | 
  [production] |