| 
      
        2010-06-08
      
      §
     | 
  
    
  | 23:05 | 
  <tfinc> | 
  synchronized php-1.5/extensions/ContributionReporting/ContributionTrackingStatistics_body.php  'adding range support  to Special:ContributionTrackingStatistics' | 
  [production] | 
            
  | 22:36 | 
  <tfinc> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  'turning off fr redirect till fix is in place' | 
  [production] | 
            
  | 22:08 | 
  <fvassard> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  'enabling mobile redirect on FRwiki' | 
  [production] | 
            
  | 21:34 | 
  <Fred> | 
  mobile LVS setup. Traffic to mobile now balanced between mobile1 (30%) and mobile2 (70%) | 
  [production] | 
            
  | 19:58 | 
  <Rob> | 
  updated civicrm certificate file, had to apache2ctl restart on grosley | 
  [production] | 
            
  | 19:03 | 
  <catrope> | 
  synchronized php-1.5/wmf-config/CommonSettings.php  'Add .omniplan to extension whitelist for private wikis. Requested by Pete' | 
  [production] | 
            
  | 18:56 | 
  <Fred> | 
  new wikimedia-task-dns-auth deployed to nameservers. langlist regenerated. *lang*.m.wikipedia.org is now cnamed to m.wikimedia.org | 
  [production] | 
            
  | 16:41 | 
  <Rob> | 
  pushed updates to wordpress plugins on singer | 
  [production] | 
            
  | 16:40 | 
  <Fred> | 
  I foobared. need to redirect traffic to mobile2 so mobile1 can be re-imaged.  | 
  [production] | 
            
  | 15:20 | 
  <Rob> | 
  updated singer and dns records for austin's setup of wm10reg | 
  [production] | 
            
  | 01:29 | 
  <Tim> | 
  running throughput test on mobile2 | 
  [production] | 
            
  
    | 
      
        2010-06-07
      
      §
     | 
  
    
  | 23:12 | 
  <Fred> | 
  modified puppet receipe for mobile to allow for easier passenger gem update. | 
  [production] | 
            
  | 22:04 | 
  <catrope> | 
  synchronized php-1.5/wmf-config/CommonSettings.php  'Bump style version appendix' | 
  [production] | 
            
  | 22:04 | 
  <catrope> | 
  synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js  'r67568' | 
  [production] | 
            
  | 20:55 | 
  <aaron> | 
  synchronized php-1.5/wmf-config/flaggedrevs.php  'rollback for de_labs editors' | 
  [production] | 
            
  | 19:46 | 
  <tfinc> | 
  synchronized php-1.5/extensions/CentralNotice/SpecialNoticeTemplate.php  'Picking up fix for tmp_id' | 
  [production] | 
            
  | 19:40 | 
  <catrope> | 
  synchronized php-1.5/extensions/CentralNotice/SpecialNoticeTemplate.php  'r67558' | 
  [production] | 
            
  | 19:12 | 
  <tfinc> | 
  synchronized php-1.5/extensions/CentralNotice/SpecialNoticeTemplate.php  'Picking up pagination from r67553' | 
  [production] | 
            
  | 18:56 | 
  <tfinc> | 
  synchronized php-1.5/extensions/CentralNotice/SpecialNoticeText.php  'Picking up pagination from r67553' | 
  [production] | 
            
  | 16:57 | 
  <Fred> | 
  Limiting message size to 5M for lists. | 
  [production] | 
            
  | 16:43 | 
  <Tim> | 
  moved mediawiki-cvs archive files away to prevent further archiving | 
  [production] | 
            
  | 16:26 | 
  <Tim> | 
  disabled archiving on problematic list mediawiki-cvs | 
  [production] | 
            
  | 14:07 | 
  <andrew> | 
  synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php  | 
  [production] | 
            
  | 14:04 | 
  <andrew> | 
  synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php  | 
  [production] | 
            
  | 13:49 | 
  <andrew> | 
  synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php  | 
  [production] | 
            
  | 13:48 | 
  <andrew> | 
  synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php  | 
  [production] | 
            
  | 13:46 | 
  <andrew> | 
  synchronized php-1.5/extensions/StrategyWiki/ActiveStrategy/ActiveStrategy_body.php  | 
  [production] | 
            
  | 07:43 | 
  <Tim> | 
  on lily: fixed broken sources.list and upgraded python, installed python symbols | 
  [production] | 
            
  | 07:25 | 
  <Tim> | 
  on lily: restarted mailman (including ArchRunner) | 
  [production] | 
            
  | 07:07 | 
  <Tim> | 
  on lily: there are three instances of ArchRunner which are in a tight loop using CPU but not doing any syscalls, presumably dead. Renicing them, will try to make backtraces | 
  [production] | 
            
  | 03:41 | 
  <Tim> | 
  srv281 back up and resynced. However it crashed after being up only 3 days and shows more machine check errors. Suggest RMA. | 
  [production] | 
            
  | 03:25 | 
  <Fred> | 
  setup ganglia for lily.  | 
  [production] | 
            
  | 03:07 | 
  <Tim> | 
  srv281 down for 41 hours, trying reboot | 
  [production] | 
            
  | 02:54 | 
  <Fred> | 
  upgraded gmond on lily. Modified gmond.conf to handle new modular arch.  | 
  [production] | 
            
  | 02:46 | 
  <Tim> | 
  on lily: experimentally disabling broken bayes feature in spamassassin | 
  [production] | 
            
  | 02:35 | 
  <Tim> | 
  on lily: restarting spamd and fixing lock file errors | 
  [production] | 
            
  | 01:25 | 
  <Tim> | 
  on mobile1: Reduced PassengerMaxRequests to 5000 to reduce memory leakage | 
  [production] | 
            
  
    | 
      
        2010-06-06
      
      §
     | 
  
    
  | 11:33 | 
  <catrope> | 
  synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php  'r67456' | 
  [production] | 
            
  | 11:33 | 
  <catrope> | 
  synchronized php-1.5/extensions/UsabilityInitiative/css/combined.min.css  'r67456' | 
  [production] | 
            
  | 07:33 | 
  <Tim> | 
  on mobile1: use PassengerMaxRequests instead of PassengerPoolIdleTime, to avoid oscillation between 200 and 30 processes, once every half an hour due to unknown slowdown | 
  [production] | 
            
  | 03:10 | 
  <Tim> | 
  serve a 404 error for requests to the mobile server for domains other than *.m.wikipedia.org. DNS points here for lots of domains. | 
  [production] | 
            
  | 02:14 | 
  <Tim> | 
  installed a redirect from en.m.wikipedia.com to .org | 
  [production] | 
            
  | 01:18 | 
  <Tim> | 
  changed the mobile1 apache access log format to something more useful (and squid-like) | 
  [production] | 
            
  | 00:47 | 
  <Tim> | 
  on mobile1: installed logrotate script for apache2 (via puppet) | 
  [production] |