| 
      
        2011-03-28
      
      §
     | 
  
    
  | 16:58 | 
  <catrope> | 
  synchronized php-1.17/includes/api/ApiQueryCategoryMembers.php  'r84908' | 
  [production] | 
            
  | 14:58 | 
  <apergos> | 
  was seeing sluggish response from scalers, ms4, running out of swap space  | 
  [production] | 
            
  | 14:54 | 
  <apergos> | 
  restarting web server on ms4...  | 
  [production] | 
            
  | 05:58 | 
  <tstarling> | 
  synchronized php-1.17/wmf-config/InitialiseSettings.php  'disabling ehcache' | 
  [production] | 
            
  | 05:57 | 
  <Tim> | 
  ehcache broke, disabling client | 
  [production] | 
            
  | 05:39 | 
  <tstarling> | 
  synchronized php-1.17/wmf-config/InitialiseSettings.php  | 
  [production] | 
            
  | 05:35 | 
  <Tim> | 
  db40 CPU usage is 5% of all cores, disk util <1%, network ~1%, request rate ~58% of weekly peak. Enabling ehcache on all wikis. | 
  [production] | 
            
  | 05:32 | 
  <Tim> | 
  fixing torrus per wikitech instructions | 
  [production] | 
            
  | 05:08 | 
  <tstarling> | 
  synchronized php-1.17/wmf-config/InitialiseSettings.php  'enabling ehcache on enwiki' | 
  [production] | 
            
  | 05:07 | 
  <Tim> | 
  resource utilisation on db40 is a nothingth of a percent, moving on to stage 4 | 
  [production] | 
            
  | 05:04 | 
  <tstarling> | 
  synchronized php-1.17/wmf-config/InitialiseSettings.php  'enabling ehcache for enwikiquote' | 
  [production] | 
            
  | 04:56 | 
  <tstarling> | 
  synchronized php-1.17/wmf-config/CommonSettings.php  'put ehcache first so it will get more load' | 
  [production] | 
            
  | 04:45 | 
  <tstarling> | 
  synchronized php-1.17/includes/objectcache/EhcacheBagOStuff.php  'curl fix' | 
  [production] | 
            
  | 04:38 | 
  <tstarling> | 
  synchronized php-1.17/wmf-config/InitialiseSettings.php  'enable ehcache on test and test2' | 
  [production] | 
            
  | 04:36 | 
  <tstarling> | 
  synchronized php-1.17/includes/objectcache/EhcacheBagOStuff.php  'don't use CURLOPT_TIMEOUT_MS' | 
  [production] | 
            
  | 04:28 | 
  <tstarling> | 
  synchronized php-1.17/wmf-config/InitialiseSettings.php  'ehcache setup' | 
  [production] | 
            
  | 04:28 | 
  <Tim> | 
  fixed sync on searchidx1 | 
  [production] | 
            
  | 04:27 | 
  <tstarling> | 
  synchronized php-1.17/wmf-config/CommonSettings.php  'test sync' | 
  [production] | 
            
  | 04:27 | 
  <tstarling> | 
  synchronized php-1.17/wmf-config/CommonSettings.php  'test sync' | 
  [production] | 
            
  | 04:26 | 
  <tstarling> | 
  synchronized php-1.17/wmf-config/CommonSettings.php  'ehcache setup' | 
  [production] | 
            
  
    | 
      
        2011-03-27
      
      §
     | 
  
    
  | 22:28 | 
  <Ryan_Lane> | 
  restarting apache on singer | 
  [production] | 
            
  | 22:19 | 
  <Ryan_Lane> | 
  pushing new star cert to singer | 
  [production] | 
            
  | 21:55 | 
  <RoanKattouw> | 
  SSL certificate for secure.wikimedia.org expired about an hour ago, no ops around. I've just texted Ryan | 
  [production] | 
            
  | 19:52 | 
  <demon> | 
  synchronized php-1.17/extensions/ProofreadPage/ProofreadPage_body.php  'r84871' | 
  [production] | 
            
  | 14:14 | 
  <catrope> | 
  synchronized php-1.17/includes/resourceloader/ResourceLoaderModule.php  'r84845, hopefully fixes catchable fatal error about makeCombinedStyles()' | 
  [production] | 
            
  | 10:55 | 
  <hashar:> | 
  on [[searchidx1]] wmf-config path does not exist triggering an error on code sync | 
  [production] | 
            
  | 10:54 | 
  <hashar:> | 
  auto defers SocialProfile, editor_trends & WikiSnaps | 
  [production] | 
            
  | 10:54 | 
  <hashar> | 
  synchronized php-1.17/wmf-config/codereview.php  | 
  [production] | 
            
  
    | 
      
        2011-03-26
      
      §
     | 
  
    
  | 21:41 | 
  <ariel> | 
  synchronized php-1.17/extensions/Narayam/Narayam.i18n.php  'remove BOM from file, was being written to stdout' | 
  [production] | 
            
  | 19:13 | 
  <apergos> | 
  running rsync for White Cat again to pick up en wiki 20110115 hist files, on dataset2 in screen session as root, please shoot after 3 days (check with user first in channel) | 
  [production] | 
            
  | 15:05 | 
  <mark> | 
  synchronized php-1.17/wmf-config/db.php  'Repooled db38 with lower load (100 instead of 400)' | 
  [production] | 
            
  | 14:57 | 
  <mark> | 
  synchronized php-1.17/wmf-config/db.php  'Depooling db38' | 
  [production] | 
            
  | 10:55 | 
  <midom> | 
  synchronized php-1.17/wmf-config/mc.php  'srv206 to srv243 swap' | 
  [production] | 
            
  | 10:53 | 
  <domas> | 
  I misread before. memcached check was broken for 23 days, not 23 hours. duct-taped by creating symlink php-1.17 -> wmf-deployment on spence, not sure for how long :) | 
  [production] | 
            
  | 10:50 | 
  <domas> | 
  added 'pkill -9 -f smssend' hourly crontab on spence... | 
  [production] | 
            
  
    | 
      
        2011-03-25
      
      §
     | 
  
    
  | 15:36 | 
  <mark> | 
  Upgraded csw1-esams and br1-knams to IronWare 2.7.3a | 
  [production] | 
            
  | 15:31 | 
  <mark> | 
  Reloading br1-knams | 
  [production] | 
            
  | 15:09 | 
  <mark> | 
  Reloading csw1-esams | 
  [production] | 
            
  | 15:09 | 
  <demon> | 
  synchronized php-1.17/includes/HTMLForm.php  'r84748' | 
  [production] | 
            
  | 14:53 | 
  <mark> | 
  Upgraded JUNOS to 10.0S10.0 on csw2-esams | 
  [production] | 
            
  | 13:18 | 
  <mark> | 
  Moving esams traffic to pmtpa for upcoming router firmware upgrade/reload | 
  [production] | 
            
  | 04:49 | 
  <Tim> | 
  reniced apaches to nice level 0 again | 
  [production] | 
            
  | 03:53 | 
  <aaron> | 
  synchronized php-1.17/wmf-config/InitialiseSettings.php  | 
  [production] | 
            
  | 03:51 | 
  <aaron> | 
  synchronized php-1.17/wmf-config/abusefilter.php  'Moved abusefilter rights here from InitialiseSettings.php' | 
  [production] | 
            
  | 03:36 | 
  <root> | 
  synchronizing Wikimedia installation... Revision: 84728:  | 
  [production] | 
            
  | 03:36 | 
  <Tim> | 
  svn up/scap to r84727 | 
  [production] | 
            
  | 02:45 | 
  <aaron> | 
  synchronized php-1.17/wmf-config/InitialiseSettings.php  'Added missing autopatrolled group to siwiki' | 
  [production] | 
            
  | 02:30 | 
  <aaron> | 
  synchronized php-1.17/wmf-config/InitialiseSettings.php  'Added missing rollbacker group to siwiki' | 
  [production] | 
            
  | 02:07 | 
  <aaron> | 
  synchronized php-1.17/wmf-config/flaggedrevs.php  'DO NOT PUT ABUSEFILTER SETTINGS IN HERE' | 
  [production] | 
            
  | 02:06 | 
  <aaron> | 
  synchronized php-1.17/wmf-config/InitialiseSettings.php  'Copy abusefilter right settings from flaggedrevs.php to here, were they belong' | 
  [production] |