| 2011-03-30
      
      § | 
    
  | 04:59 | <tstarling> | synchronized php-1.17/includes/GlobalFunctions.php  'live hack: send site notice cache to memcached not ehcache' | [production] | 
            
  | 04:58 | <tstarling> | synchronized php-1.17/includes/MessageCache.php  'another one' | [production] | 
            
  | 04:55 | <tstarling> | synchronized php-1.17/includes/Skin.php  'live hack to send sidebar cache to memcached not ehcache' | [production] | 
            
  | 04:50 | <tstarling> | synchronized php-1.17/wmf-config/InitialiseSettings.php  'enabling ehcache client on testwiki and test2wiki' | [production] | 
            
  | 04:46 | <Tim> | installed ehcache under tomcat on db40. Documented at [[Ehcache]]. | [production] | 
            
  | 03:49 | <Tim> | did apt-get upgrade on db40, now rebooting to get new kernel | [production] | 
            
  | 01:11 | <mark> | Reinstalled ms5 with Ubuntu 10.04, created software RAID10 array /dev/md1 out of 48 drives | [production] | 
            
  
    | 2011-03-29
      
      § | 
    
  | 20:44 | <mark> | Moved ms5 to internal vlan 2 | [production] | 
            
  | 20:34 | <mark> | Pushed DNS for ms5.pmtpa.wmnet | [production] | 
            
  | 18:56 | <mark> | Changed NFS client params to rsize=8192,wsize=8192 on srv228:/mnt/thumbs | [production] | 
            
  | 15:37 | <ariel> | synchronized php-1.17/wmf-config/CommonSettings.php  'increase account creation throttle value for el wiki for editing workshop' | [production] | 
            
  | 10:41 | <apergos> | rsync on dataset2 for White Cat one last time, please shoot in three days (check with user first) | [production] | 
            
  | 00:42 | <Ryan_Lane> | enabled ssl on RT, all non-http now forces to https for RT | [production] | 
            
  | 00:42 | <Ryan_Lane> | fixed puppet issue on erzurumi by adding the wikidev group. | [production] | 
            
  
    | 2011-03-28
      
      § | 
    
  | 23:37 | <Ryan_Lane> | spence died; powercycling it | [production] | 
            
  | 23:06 | <awjrichards> | synchronized php-1.17/extensions/UserDailyContribs/api/ApiUserDailyContribs.php  'Deploying bug fixes to UserDailyContribs API (r84935)' | [production] | 
            
  | 22:41 | <Ryan_Lane> | added a certificate check for ldaps on nfs1/2 | [production] | 
            
  | 22:36 | <Ryan_Lane> | pushed the internal CA certificate to spence, so that we can monitor the LDAP certificates | [production] | 
            
  | 21:43 | <awjr> | running sync-comon all to deploy api updates to extension UserDailyContrib | [production] | 
            
  | 21:43 | <awjrichards> | ran sync-common-all | [production] | 
            
  | 21:38 | <awjrichards> | synchronized php-1.17/extensions/UserDailyContribs/api  'Attempting to create UserDailyContribs/api dir using sync-file' | [production] | 
            
  | 21:07 | <Ryan_Lane> | pushed out a cert check for secure.wikimedia.org | [production] | 
            
  | 19:17 | <Ryan_Lane> | changing weight of mobile1 to 10 | [production] | 
            
  | 17:11 | <catrope> | synchronized php-1.17/wmf-config/InitialiseSettings.php  'Disable Vector/SectionEditLinks on enwiki' | [production] | 
            
  | 16:58 | <catrope> | synchronized php-1.17/includes/api/ApiQueryCategoryMembers.php  'r84908' | [production] | 
            
  | 14:58 | <apergos> | was seeing sluggish response from scalers, ms4, running out of swap space | [production] | 
            
  | 14:54 | <apergos> | restarting web server on ms4... | [production] | 
            
  | 05:58 | <tstarling> | synchronized php-1.17/wmf-config/InitialiseSettings.php  'disabling ehcache' | [production] | 
            
  | 05:57 | <Tim> | ehcache broke, disabling client | [production] | 
            
  | 05:39 | <tstarling> | synchronized php-1.17/wmf-config/InitialiseSettings.php | [production] | 
            
  | 05:35 | <Tim> | db40 CPU usage is 5% of all cores, disk util <1%, network ~1%, request rate ~58% of weekly peak. Enabling ehcache on all wikis. | [production] | 
            
  | 05:32 | <Tim> | fixing torrus per wikitech instructions | [production] | 
            
  | 05:08 | <tstarling> | synchronized php-1.17/wmf-config/InitialiseSettings.php  'enabling ehcache on enwiki' | [production] | 
            
  | 05:07 | <Tim> | resource utilisation on db40 is a nothingth of a percent, moving on to stage 4 | [production] | 
            
  | 05:04 | <tstarling> | synchronized php-1.17/wmf-config/InitialiseSettings.php  'enabling ehcache for enwikiquote' | [production] | 
            
  | 04:56 | <tstarling> | synchronized php-1.17/wmf-config/CommonSettings.php  'put ehcache first so it will get more load' | [production] | 
            
  | 04:45 | <tstarling> | synchronized php-1.17/includes/objectcache/EhcacheBagOStuff.php  'curl fix' | [production] | 
            
  | 04:38 | <tstarling> | synchronized php-1.17/wmf-config/InitialiseSettings.php  'enable ehcache on test and test2' | [production] | 
            
  | 04:36 | <tstarling> | synchronized php-1.17/includes/objectcache/EhcacheBagOStuff.php  'don't use CURLOPT_TIMEOUT_MS' | [production] | 
            
  | 04:28 | <tstarling> | synchronized php-1.17/wmf-config/InitialiseSettings.php  'ehcache setup' | [production] | 
            
  | 04:28 | <Tim> | fixed sync on searchidx1 | [production] | 
            
  | 04:27 | <tstarling> | synchronized php-1.17/wmf-config/CommonSettings.php  'test sync' | [production] | 
            
  | 04:27 | <tstarling> | synchronized php-1.17/wmf-config/CommonSettings.php  'test sync' | [production] | 
            
  | 04:26 | <tstarling> | synchronized php-1.17/wmf-config/CommonSettings.php  'ehcache setup' | [production] |