| 2011-09-21
      
      § | 
    
  | 18:03 | <notpeter> | removing srv282 from apaches pool for upgrade to lucid | [production] | 
            
  | 18:00 | <notpeter> | putting srv283, srv285, and srv286 back in the apaches pool | [production] | 
            
  | 17:47 | <maplebed> | stopping mysql on ms1 to create a replica in eqiad | [production] | 
            
  | 16:17 | <Jeff_Green> | locked out ssh from fenari, etc. to payments[1-4], leaving only the new bastion host | [production] | 
            
  | 15:39 | <catrope> | synchronized wmf-config/InitialiseSettings.php  'Add Babel settings for Italian wikis' | [production] | 
            
  | 15:37 | <RobH> | updating dns for datacenter cameras | [production] | 
            
  | 15:32 | <catrope> | synchronized php-1.18/extensions/Babel/Babel.i18n.php  '[[rev:97741|r97741]]' | [production] | 
            
  | 15:31 | <catrope> | synchronized php-1.17/extensions/Babel/Babel.i18n.php  '[[rev:97741|r97741]]' | [production] | 
            
  | 14:29 | <robh> | es1003 hdd replaced per rt#1453 | [production] | 
            
  | 14:05 | <catrope> | synchronized php-1.18/extensions/WikiEditor/WikiEditor.php  '[[rev:97734|r97734]]' | [production] | 
            
  | 13:45 | <notpeter> | removing srv283 from apaches pool for upgrade to lucid | [production] | 
            
  | 13:43 | <catrope> | synchronized php-1.18/extensions/WikimediaMessages/WikimediaMessages.php  '[[rev:97730|r97730]]' | [production] | 
            
  | 13:32 | <catrope> | synchronized wmf-config/InitialiseSettings.php  'Enable Babel extension on all wikis' | [production] | 
            
  | 13:23 | <catrope> | synchronized php-1.17/includes/Message.php  '[[rev:97729|r97729]]' | [production] | 
            
  | 13:12 | <notpeter> | removing srv285 from apaches pool for upgrade to lucid | [production] | 
            
  | 13:06 | <notpeter> | putting srv286 back in apaches pool | [production] | 
            
  | 13:06 | <catrope> | synchronized php-1.18/extensions/Babel/Babel.i18n.php  '[[rev:97728|r97728]]' | [production] | 
            
  | 13:06 | <catrope> | synchronized php-1.18/extensions/Babel/Babel.class.php  '[[rev:97728|r97728]]' | [production] | 
            
  | 13:06 | <catrope> | synchronized php-1.17/extensions/Babel/Babel.i18n.php  '[[rev:97728|r97728]]' | [production] | 
            
  | 13:06 | <catrope> | synchronized php-1.17/extensions/Babel/Babel.class.php  '[[rev:97728|r97728]]' | [production] | 
            
  | 13:03 | <catrope> | synchronized wmf-config/InitialiseSettings.php  'Add config for Babel, pulled from wikipage on meta by SPQRobin' | [production] | 
            
  | 12:53 | <catrope> | synchronized php-1.17/includes/OutputPage.php  '[[rev:97723|r97723]]' | [production] | 
            
  | 12:53 | <catrope> | synchronized php-1.17/includes/parser/ParserOutput.php  '[[rev:97723|r97723]]' | [production] | 
            
  | 12:43 | <catrope> | synchronized php-1.18/includes/OutputPage.php  '[[rev:97722|r97722]]' | [production] | 
            
  | 12:41 | <catrope> | synchronized php-1.18/includes/parser/ParserOutput.php  '[[rev:97722|r97722]]' | [production] | 
            
  | 12:22 | <catrope> | synchronizing Wikimedia installation... : Add Babel to extension-list and enable it on testwiki and test2wiki | [production] | 
            
  | 12:18 | <notpeter> | removing srv286 from apaches pool for upgrade to lucid | [production] | 
            
  | 12:16 | <catrope> | synchronizing Wikimedia installation... : Pushing out code for Babel, not enabled yet | [production] | 
            
  | 10:27 | <catrope> | synchronized wmf-config/CommonSettings.php  'revert test' | [production] | 
            
  | 10:25 | <catrope> | synchronized wmf-config/CommonSettings.php  'Add group so I can briefly view Special:MoodBar' | [production] | 
            
  | 03:12 | <aaron> | synchronized wmf-config/flaggedrevs.php  'removed unused ReaderFeedback settings from some wikis' | [production] | 
            
  | 03:09 | <aaron> | synchronized wmf-config/flaggedrevs.php  'removed flaggedrevs_labswikimedia config duplication' | [production] | 
            
  | 02:23 | <LocalisationUpdate> | completed (1.18) at Wed Sep 21 02:26:17 UTC 2011 | [production] | 
            
  | 02:20 | <LocalisationUpdate> | completed (1.17) at Wed Sep 21 02:23:08 UTC 2011 | [production] | 
            
  | 02:17 | <Tim> | rebooting srv182: ssh hangs after connecting, nothing on serial console | [production] | 
            
  | 02:14 | <tstarling> | synchronized php-1.18/includes/profiler/ProfilerSimple.php  '[[rev:97690|r97690]]' | [production] | 
            
  | 02:12 | <Tim> | cleaning up /tmp on srv215, which is almost out of disk space in its root partition | [production] | 
            
  | 01:06 | <asher> | synchronized php-1.17/includes/ExternalStoreDB.php  '[[bugzilla:31052|bug 31052]] : live hack to support reading from old non-slave external store servers' | [production] | 
            
  | 00:07 | <maplebed> | shutting down ms1 to replicate to eqiad.  It's already out of rotation. | [production] | 
            
  
    | 2011-09-20
      
      § | 
    
  | 23:31 | <reedy> | synchronizing Wikimedia installation... : Pushing out 1.18wmf1 merges, mainly image rotation fixes | [production] | 
            
  | 23:30 | <preilly> | synchronized php/extensions/MobileFrontend/views/information/disable.html.php  'emergency fix for disable link endless loop' | [production] | 
            
  | 23:29 | <preilly> | synchronized php/extensions/MobileFrontend/views/information/disable.html.php  'emergency fix for disable link endless loop' | [production] | 
            
  | 23:26 | <preilly> | emergency fix for disable link endless loop on mobile frontend | [production] | 
            
  | 23:21 | <maplebed> | increased ganglia's tmpfs from 700M to 1G and restarted gmetad.  it's all happy again, but we've lost ~8hrs ganglia data. | [production] | 
            
  | 23:08 | <maplebed> | ganglia's tmpfs is out of space. | [production] | 
            
  | 22:55 | <asher> | synchronized wmf-config/db.php  'experimenting with an externalstore cluster in readOnlyBySection' | [production] | 
            
  | 22:50 | <notpeter> | putting srv287 back in the apaches poolwq | [production] | 
            
  | 22:13 | <notpeter> | removing srv287 from apaches pool to upgrade to lucid | [production] | 
            
  | 22:05 | <asher> | synchronized php-1.17/includes/resourceloader/ResourceLoaderFileModule.php  'live hack - prevent resourceloader from serving errors when a db master is set to read-only' | [production] | 
            
  | 21:05 | <binasher> | invalidated varnish mobile cache due to cached pages linking to broken javascript | [production] |