| 2011-03-28
      
      § | 
    
  | 23:37 | <Ryan_Lane> | spence died; powercycling it | [production] | 
            
  | 23:06 | <awjrichards> | synchronized php-1.17/extensions/UserDailyContribs/api/ApiUserDailyContribs.php  'Deploying bug fixes to UserDailyContribs API (r84935)' | [production] | 
            
  | 22:41 | <Ryan_Lane> | added a certificate check for ldaps on nfs1/2 | [production] | 
            
  | 22:36 | <Ryan_Lane> | pushed the internal CA certificate to spence, so that we can monitor the LDAP certificates | [production] | 
            
  | 21:43 | <awjr> | running sync-comon all to deploy api updates to extension UserDailyContrib | [production] | 
            
  | 21:43 | <awjrichards> | ran sync-common-all | [production] | 
            
  | 21:38 | <awjrichards> | synchronized php-1.17/extensions/UserDailyContribs/api  'Attempting to create UserDailyContribs/api dir using sync-file' | [production] | 
            
  | 21:07 | <Ryan_Lane> | pushed out a cert check for secure.wikimedia.org | [production] | 
            
  | 19:17 | <Ryan_Lane> | changing weight of mobile1 to 10 | [production] | 
            
  | 17:11 | <catrope> | synchronized php-1.17/wmf-config/InitialiseSettings.php  'Disable Vector/SectionEditLinks on enwiki' | [production] | 
            
  | 16:58 | <catrope> | synchronized php-1.17/includes/api/ApiQueryCategoryMembers.php  'r84908' | [production] | 
            
  | 14:58 | <apergos> | was seeing sluggish response from scalers, ms4, running out of swap space | [production] | 
            
  | 14:54 | <apergos> | restarting web server on ms4... | [production] | 
            
  | 05:58 | <tstarling> | synchronized php-1.17/wmf-config/InitialiseSettings.php  'disabling ehcache' | [production] | 
            
  | 05:57 | <Tim> | ehcache broke, disabling client | [production] | 
            
  | 05:39 | <tstarling> | synchronized php-1.17/wmf-config/InitialiseSettings.php | [production] | 
            
  | 05:35 | <Tim> | db40 CPU usage is 5% of all cores, disk util <1%, network ~1%, request rate ~58% of weekly peak. Enabling ehcache on all wikis. | [production] | 
            
  | 05:32 | <Tim> | fixing torrus per wikitech instructions | [production] | 
            
  | 05:08 | <tstarling> | synchronized php-1.17/wmf-config/InitialiseSettings.php  'enabling ehcache on enwiki' | [production] | 
            
  | 05:07 | <Tim> | resource utilisation on db40 is a nothingth of a percent, moving on to stage 4 | [production] | 
            
  | 05:04 | <tstarling> | synchronized php-1.17/wmf-config/InitialiseSettings.php  'enabling ehcache for enwikiquote' | [production] | 
            
  | 04:56 | <tstarling> | synchronized php-1.17/wmf-config/CommonSettings.php  'put ehcache first so it will get more load' | [production] | 
            
  | 04:45 | <tstarling> | synchronized php-1.17/includes/objectcache/EhcacheBagOStuff.php  'curl fix' | [production] | 
            
  | 04:38 | <tstarling> | synchronized php-1.17/wmf-config/InitialiseSettings.php  'enable ehcache on test and test2' | [production] | 
            
  | 04:36 | <tstarling> | synchronized php-1.17/includes/objectcache/EhcacheBagOStuff.php  'don't use CURLOPT_TIMEOUT_MS' | [production] | 
            
  | 04:28 | <tstarling> | synchronized php-1.17/wmf-config/InitialiseSettings.php  'ehcache setup' | [production] | 
            
  | 04:28 | <Tim> | fixed sync on searchidx1 | [production] | 
            
  | 04:27 | <tstarling> | synchronized php-1.17/wmf-config/CommonSettings.php  'test sync' | [production] | 
            
  | 04:27 | <tstarling> | synchronized php-1.17/wmf-config/CommonSettings.php  'test sync' | [production] | 
            
  | 04:26 | <tstarling> | synchronized php-1.17/wmf-config/CommonSettings.php  'ehcache setup' | [production] | 
            
  
    | 2011-03-27
      
      § | 
    
  | 22:28 | <Ryan_Lane> | restarting apache on singer | [production] | 
            
  | 22:19 | <Ryan_Lane> | pushing new star cert to singer | [production] | 
            
  | 21:55 | <RoanKattouw> | SSL certificate for secure.wikimedia.org expired about an hour ago, no ops around. I've just texted Ryan | [production] | 
            
  | 19:52 | <demon> | synchronized php-1.17/extensions/ProofreadPage/ProofreadPage_body.php  'r84871' | [production] | 
            
  | 14:14 | <catrope> | synchronized php-1.17/includes/resourceloader/ResourceLoaderModule.php  'r84845, hopefully fixes catchable fatal error about makeCombinedStyles()' | [production] | 
            
  | 10:55 | <hashar:> | on [[searchidx1]] wmf-config path does not exist triggering an error on code sync | [production] | 
            
  | 10:54 | <hashar:> | auto defers SocialProfile, editor_trends & WikiSnaps | [production] | 
            
  | 10:54 | <hashar> | synchronized php-1.17/wmf-config/codereview.php | [production] | 
            
  
    | 2011-03-26
      
      § | 
    
  | 21:41 | <ariel> | synchronized php-1.17/extensions/Narayam/Narayam.i18n.php  'remove BOM from file, was being written to stdout' | [production] | 
            
  | 19:13 | <apergos> | running rsync for White Cat again to pick up en wiki 20110115 hist files, on dataset2 in screen session as root, please shoot after 3 days (check with user first in channel) | [production] | 
            
  | 15:05 | <mark> | synchronized php-1.17/wmf-config/db.php  'Repooled db38 with lower load (100 instead of 400)' | [production] | 
            
  | 14:57 | <mark> | synchronized php-1.17/wmf-config/db.php  'Depooling db38' | [production] | 
            
  | 10:55 | <midom> | synchronized php-1.17/wmf-config/mc.php  'srv206 to srv243 swap' | [production] | 
            
  | 10:53 | <domas> | I misread before. memcached check was broken for 23 days, not 23 hours. duct-taped by creating symlink php-1.17 -> wmf-deployment on spence, not sure for how long :) | [production] |