| 2011-10-10
      
      § | 
    
  | 22:28 | <mutante> | sq71,sq72,sq73 - dist-upgraded/kernel,rebooted | [production] | 
            
  | 21:11 | <LocalisationUpdate> | completed (1.18) at Mon Oct 10 21:14:28 UTC 2011 | [production] | 
            
  | 21:09 | <LocalisationUpdate> | failed | [production] | 
            
  | 21:03 | <LocalisationUpdate> | completed (1.18) at Mon Oct 10 21:05:53 UTC 2011 | [production] | 
            
  | 20:47 | <Guest48655> | sq66,sq68 - upgraded kernel/reboot | [production] | 
            
  | 20:34 | <LocalisationUpdate> | completed (1.18) at Mon Oct 10 20:36:46 UTC 2011 | [production] | 
            
  | 20:29 | <LocalisationUpdate> | completed (1.18) at Mon Oct 10 20:32:22 UTC 2011 | [production] | 
            
  | 20:17 | <LocalisationUpdate> | completed (1.18) at Mon Oct 10 20:20:06 UTC 2011 | [production] | 
            
  | 20:04 | <notpeter> | putting srv290-srv301 back into api pool | [production] | 
            
  | 19:58 | <reedy> | synchronized php-1.18/maintenance/wmf/clearMessageBlobs.php | [production] | 
            
  | 19:50 | <LocalisationUpdate> | completed (1.18) at Mon Oct 10 19:53:32 UTC 2011 | [production] | 
            
  | 19:25 | <LocalisationUpdate> | completed (1.18) at Mon Oct 10 19:28:09 UTC 2011 | [production] | 
            
  | 18:24 | <Jeff_Green> | spence gets a cleansing reboot | [production] | 
            
  | 17:48 | <reedy> | synchronized wmf-config/InitialiseSettings.php  'Enable ClickTracking on zhwiki as prequisite for [[bugzilla:31578|bug 31578]] Install AF' | [production] | 
            
  | 17:33 | <notpeter> | removing srv290-srv301 from api lb pool for upgrades to lucid | [production] | 
            
  | 17:09 | <RobH> | memcached started on srv256, should clear spence check | [production] | 
            
  | 16:55 | <notpeter> | putting mw62-mw74 into api pool | [production] | 
            
  | 16:16 | <RobH> | mw1102 voltage errors, poking around system, it may possibly reboot | [production] | 
            
  | 14:15 | <notpeter> | restarting nagios and ganglia on spence | [production] | 
            
  | 02:23 | <LocalisationUpdate> | completed (1.18) at Mon Oct 10 02:25:33 UTC 2011 | [production] | 
            
  | 00:47 | <Jeff_Green> | srv290, srv292, srv294, srv298 power cycled, see RT #1685 for more info | [production] | 
            
  
    | 2011-10-08
      
      § | 
    
  | 02:20 | <LocalisationUpdate> | completed (1.18) at Sat Oct  8 02:23:10 UTC 2011 | [production] | 
            
  | 01:27 | <mutante> | sq63,sq64,sq65 - upgraded kernel/reboot (earlier today, with lots of time in between( | [production] | 
            
  | 00:53 | <robla> | synchronized php-1.18/tests/  'syncing tests for [[rev:99292|r99292]]' | [production] | 
            
  | 00:43 | <robla> | synchronized php-1.18/languages/classes/LanguageTr.php  'deployed [[rev:99292|r99292]] to fix [[bugzilla:31490|bug 31490]] (Turkish lcfirst/ucfirst prob)' | [production] | 
            
  | 00:19 | <aaron> | synchronized php-1.18/includes/parser  'deployed [[rev:99265|r99265]]' | [production] | 
            
  | 00:07 | <mutante> | /dev/mapper/static-uncompressed (local on hume) is 100% full! | [production] | 
            
  | 00:04 | <aaron> | synchronized php-1.18/extensions/GlobalUsage/GlobalUsageQuery.php  'deployed [[rev:99284|r99284]]' | [production] | 
            
  
    | 2011-10-07
      
      § | 
    
  | 22:50 | <mutante> | installing security updates on fenari | [production] | 
            
  | 22:27 | <aaron> | synchronized php-1.18/resources/jquery/jquery.js  'deployed [[rev:99264|r99264]]' | [production] | 
            
  | 22:12 | <neilk> | synchronized php-1.18/extensions/UploadWizard/resources/mw.UploadWizardLicenseInput.js  'fix issue with UploadWizard next button on licensing step and jquery 1.6.4' | [production] | 
            
  | 22:11 | <aaron> | synchronized php-1.18/includes/media/Bitmap.php  'deployred [[rev:99274|r99274]]' | [production] | 
            
  | 21:22 | <maplebed> | ganglios is installed on spence; nagios can now query ganglia metrics. | [production] | 
            
  | 20:30 | <RobH> | srv284 os install in progress | [production] | 
            
  | 19:26 | <aaron> | synchronized php-1.18/maintenance/addwiki.php | [production] | 
            
  | 19:15 | <aaron> | synchronized php-1.18/maintenance/addwiki.php  '[[rev:99238|r99238]]' | [production] | 
            
  | 19:13 | <RobH> | updated netboot.cfg for cp1001-1042 on brewster | [production] | 
            
  | 19:12 | <RobH> | installing os on cp1001-cp1042 per RT 1679 | [production] | 
            
  | 18:25 | <aaron> | rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.18 | [production] | 
            
  | 16:12 | <RobH> | mw59-mw74 OS installed, awaiting puppetization | [production] | 
            
  | 15:47 | <RobH> | installing os on mw59-mw75 | [production] | 
            
  | 14:58 | <mark> | Merged lots of puppet changes, gonna hide now | [production] | 
            
  | 14:03 | <RobH> | updated dns for bd.wikimedia.org, project still not live | [production] | 
            
  | 13:52 | <notpeter> | stopping apache and job queue runners on all external store boxes | [production] | 
            
  | 13:15 | <notpeter> | powercycling srv266, as this is one of our favorite pastimes. | [production] | 
            
  | 12:21 | <catrope> | synchronized wmf-config/CommonSettings.php  'Replace expired account creation throttle exemption for an India event with one for a Brazil event' | [production] | 
            
  | 05:46 | <aaron> | synchronized php-1.18/extensions/GlobalBlocking/SpecialGlobalBlockStatus.php  '[[rev:99198|r99198]]' | [production] |