| 2009-07-17
      
      § | 
    
  | 23:42 | <brion> | added a 404 page and recovered index.php for our temp download.wikimedia.org | [production] | 
            
  | 22:05 | <brion> | set wikitech to use vector skin by default :D | [production] | 
            
  | 22:03 | <Andrew> | Fixed morebots, which was relying on a fragile version check. Just deleted it :) | [production] | 
            
  | 22:02 | <Andrew> | testing | [production] | 
            
  | 20:43 | <brion> | fixed paths for noc.wikimedia.org/conf file highlighting | [production] | 
            
  | 20:38 | <domas> | ms2 has broken disks.. | [production] | 
            
  | 20:31 | <brion> | We're going to see about setting up the previously-idle ms5 so we can get our thumbnailing on | [production] | 
            
  | 20:01 | <brion> | rob's poking raid rebuild on storage2 (dumps server) | [production] | 
            
  | 19:03 | <RobH_A90> | eiximenis and dobson pulled for solid state drive testing, do not use for other tasks | [production] | 
            
  | 18:28 | <brion> | synchronized wmf-deployment/wmf-config/InitialiseSettings.php  'enabling vector for rtl' | [production] | 
            
  | 18:25 | <atglenn> | started mass move out of the way of thumbnail dirs and replacing with symlinks to ms4 | [production] | 
            
  | 18:25 | <brion> | synchronized wmf-deployment/wmf-config/CommonSettings.php  'bump style version' | [production] | 
            
  | 18:25 | <brion> | ran sync-common-all | [production] | 
            
  | 18:24 | <brion> | running sync-common-all for UI updates. need to poke the style ver too :) | [production] | 
            
  | 18:08 | <brion> | svn up'ing wmf-deployment for test.wikipedia.org. Merged UI fixes from usability team | [production] | 
            
  | 18:03 | <Fred> | spun a couple more apache server into image scalers: srv219..srv224. | [production] | 
            
  | 17:28 | <rainman-sr> | putting new location of initialisesettings to lsearch-global-2.1.conf so the incremental updater works again | [production] | 
            
  | 17:20 | <Fred> | srv224 is now an image_scaler. Adjusted on lvs3, ganglia and dsh's node_list. | [production] | 
            
  | 17:14 | <Fred> | db20 back online | [production] | 
            
  | 16:50 | <Fred> | rebooting db20 as it is in a "state" | [production] | 
            
  | 16:45 | <brion> | looks like we've lost internal /home NFS, which makes some of our internal services very unhappy. investigating... | [production] | 
            
  | 16:43 | <brion> | ganglia out. | [production] | 
            
  | 13:44 | <apergos1> | doing next round of removals on ms1 (/export/upload/wikipedia/en/thumb/2) to keep ahead of the game | [production] | 
            
  | 04:15 | <apergos> | starting removal of /export/upload/wikipedia/en/thumb/1 on ms1 (moved away and symlink to ms4 done already) for more space | [production] | 
            
  | 03:54 | <brion> | synchronized wmf-deployment/wmf-config/CommonSettings.php  'Disabling sitenotice from maintenance' | [production] | 
            
  | 03:29 | <brion> | reenabling uploads & image deletion/undeletion | [production] | 
            
  | 03:29 | <brion> | synchronized wmf-deployment/wmf-config/CommonSettings.php | [production] | 
            
  | 03:28 | <brion> | remounting ms1 on apaches | [production] | 
            
  | 00:49 | <atglenn> | only about 1gb gain on each so doing all of  /export/upload/wikipedia/en/thumb/0 | [production] | 
            
  | 00:39 | <atglenn> | removing more directories in /export/upload/wikipedia/en/thumb/0 on ms1 and replacing with symlinks to ms4 | [production] | 
            
  | 00:30 | <brion> | synchronized wmf-deployment/includes/specials/SpecialUpload.php | [production] | 
            
  | 00:30 | <brion> | synchronized wmf-deployment/includes/ImagePage.php | [production] | 
            
  | 00:27 | <brion> | synchronized wmf-deployment/wmf-config/CommonSettings.php | [production] | 
            
  | 00:20 | <brion> | temporarily disabling image delete/rename during maintenance | [production] |