| 2010-06-08
      
      § | 
    
  | 23:27 | <tfinc> | synchronized php-1.5/wmf-config/CommonSettings.php  'Adding global support page to tracking' | [production] | 
            
  | 23:05 | <tfinc> | synchronized php-1.5/extensions/ContributionReporting/ContributionTrackingStatistics_body.php  'adding range support  to Special:ContributionTrackingStatistics' | [production] | 
            
  | 22:36 | <tfinc> | synchronized php-1.5/wmf-config/InitialiseSettings.php  'turning off fr redirect till fix is in place' | [production] | 
            
  | 22:08 | <fvassard> | synchronized php-1.5/wmf-config/InitialiseSettings.php  'enabling mobile redirect on FRwiki' | [production] | 
            
  | 21:34 | <Fred> | mobile LVS setup. Traffic to mobile now balanced between mobile1 (30%) and mobile2 (70%) | [production] | 
            
  | 19:58 | <Rob> | updated civicrm certificate file, had to apache2ctl restart on grosley | [production] | 
            
  | 19:03 | <catrope> | synchronized php-1.5/wmf-config/CommonSettings.php  'Add .omniplan to extension whitelist for private wikis. Requested by Pete' | [production] | 
            
  | 18:56 | <Fred> | new wikimedia-task-dns-auth deployed to nameservers. langlist regenerated. *lang*.m.wikipedia.org is now cnamed to m.wikimedia.org | [production] | 
            
  | 16:41 | <Rob> | pushed updates to wordpress plugins on singer | [production] | 
            
  | 16:40 | <Fred> | I foobared. need to redirect traffic to mobile2 so mobile1 can be re-imaged. | [production] | 
            
  | 15:20 | <Rob> | updated singer and dns records for austin's setup of wm10reg | [production] | 
            
  | 01:29 | <Tim> | running throughput test on mobile2 | [production] | 
            
  
    | 2010-06-07
      
      § | 
    
  | 23:12 | <Fred> | modified puppet receipe for mobile to allow for easier passenger gem update. | [production] | 
            
  | 22:04 | <catrope> | synchronized php-1.5/wmf-config/CommonSettings.php  'Bump style version appendix' | [production] | 
            
  | 22:04 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js  'r67568' | [production] | 
            
  | 20:55 | <aaron> | synchronized php-1.5/wmf-config/flaggedrevs.php  'rollback for de_labs editors' | [production] | 
            
  | 19:46 | <tfinc> | synchronized php-1.5/extensions/CentralNotice/SpecialNoticeTemplate.php  'Picking up fix for tmp_id' | [production] | 
            
  | 19:40 | <catrope> | synchronized php-1.5/extensions/CentralNotice/SpecialNoticeTemplate.php  'r67558' | [production] | 
            
  | 19:12 | <tfinc> | synchronized php-1.5/extensions/CentralNotice/SpecialNoticeTemplate.php  'Picking up pagination from r67553' | [production] | 
            
  | 18:56 | <tfinc> | synchronized php-1.5/extensions/CentralNotice/SpecialNoticeText.php  'Picking up pagination from r67553' | [production] | 
            
  | 16:57 | <Fred> | Limiting message size to 5M for lists. | [production] | 
            
  | 16:43 | <Tim> | moved mediawiki-cvs archive files away to prevent further archiving | [production] | 
            
  | 16:26 | <Tim> | disabled archiving on problematic list mediawiki-cvs | [production] | 
            
  | 14:07 | <andrew> | synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php | [production] | 
            
  | 14:04 | <andrew> | synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php | [production] | 
            
  | 13:49 | <andrew> | synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php | [production] | 
            
  | 13:48 | <andrew> | synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php | [production] | 
            
  | 13:46 | <andrew> | synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php | [production] | 
            
  | 07:43 | <Tim> | on lily: fixed broken sources.list and upgraded python, installed python symbols | [production] | 
            
  | 07:25 | <Tim> | on lily: restarted mailman (including ArchRunner) | [production] | 
            
  | 07:07 | <Tim> | on lily: there are three instances of ArchRunner which are in a tight loop using CPU but not doing any syscalls, presumably dead. Renicing them, will try to make backtraces | [production] | 
            
  | 03:41 | <Tim> | srv281 back up and resynced. However it crashed after being up only 3 days and shows more machine check errors. Suggest RMA. | [production] | 
            
  | 03:25 | <Fred> | setup ganglia for lily. | [production] | 
            
  | 03:07 | <Tim> | srv281 down for 41 hours, trying reboot | [production] | 
            
  | 02:54 | <Fred> | upgraded gmond on lily. Modified gmond.conf to handle new modular arch. | [production] | 
            
  | 02:46 | <Tim> | on lily: experimentally disabling broken bayes feature in spamassassin | [production] | 
            
  | 02:35 | <Tim> | on lily: restarting spamd and fixing lock file errors | [production] | 
            
  | 01:25 | <Tim> | on mobile1: Reduced PassengerMaxRequests to 5000 to reduce memory leakage | [production] | 
            
  
    | 2010-06-06
      
      § | 
    
  | 11:33 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php  'r67456' | [production] | 
            
  | 11:33 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative/css/combined.min.css  'r67456' | [production] | 
            
  | 07:33 | <Tim> | on mobile1: use PassengerMaxRequests instead of PassengerPoolIdleTime, to avoid oscillation between 200 and 30 processes, once every half an hour due to unknown slowdown | [production] | 
            
  | 03:10 | <Tim> | serve a 404 error for requests to the mobile server for domains other than *.m.wikipedia.org. DNS points here for lots of domains. | [production] | 
            
  | 02:14 | <Tim> | installed a redirect from en.m.wikipedia.com to .org | [production] | 
            
  | 01:18 | <Tim> | changed the mobile1 apache access log format to something more useful (and squid-like) | [production] | 
            
  | 00:47 | <Tim> | on mobile1: installed logrotate script for apache2 (via puppet) | [production] |