| 2012-10-31
      
      § | 
    
  | 14:26 | <paravoid> | temporarily depooling ms-fe4 to apply a temporary local hack | [production] | 
            
  | 14:12 | <paravoid> | putting ms-fe4 back in the pool | [production] | 
            
  | 13:27 | <mark> | Restarted cp1021 with persistent storage | [production] | 
            
  | 13:26 | <paravoid> | depooling ms-fe4 to upgrade to precise/swift 1.7.4 | [production] | 
            
  | 11:35 | <paravoid> | putting ms-fe3 back in the pool | [production] | 
            
  | 11:28 | <^demon> | restarted gerrit services on manganese & formey to pick up replication changes | [production] | 
            
  | 07:59 | <paravoid> | depooling ms-fe3 to upgrade to precise/swift 1.7.4 | [production] | 
            
  | 07:09 | <apergos> | restarted lucene search on search1015 and search 1003 | [production] | 
            
  | 02:47 | <logmsgbot_> | LocalisationUpdate completed (1.21wmf3) at Wed Oct 31 02:47:40 UTC 2012 | [production] | 
            
  | 02:28 | <logmsgbot_> | LocalisationUpdate completed (1.21wmf2) at Wed Oct 31 02:28:50 UTC 2012 | [production] | 
            
  | 02:05 | <logmsgbot_> | reedy synchronized wmf-config/CommonSettings.php | [production] | 
            
  | 02:01 | <logmsgbot_> | reedy synchronized wmf-config/InitialiseSettings.php  'Re-enable TMH' | [production] | 
            
  | 01:43 | <logmsgbot_> | reedy synchronized wmf-config/ | [production] | 
            
  | 01:23 | <logmsgbot_> | reedy synchronized wmf-config/InitialiseSettings.php  'Disable TMH everywhere (and mwembed at the same time)' | [production] | 
            
  | 01:15 | <K4-713> | synchronized the payments cluster to 85e9007aef3e75 | [production] | 
            
  | 01:06 | <logmsgbot_> | awjrichards synchronized php-1.21wmf3/extensions/MobileFrontend/javascripts/actions/mf-edit.js  'touch file' | [production] | 
            
  | 00:58 | <logmsgbot_> | reedy synchronized php-1.21wmf3/includes/job/JobQueueDB.php | [production] | 
            
  | 00:57 | <logmsgbot_> | awjrichards synchronized php-1.21wmf2/extensions/MobileFrontend/javascripts/actions/mf-edit.js  'touch file' | [production] | 
            
  | 00:42 | <logmsgbot_> | kaldari synchronized php-1.21wmf2/extensions/CentralNotice  'Update CentralNotice on meta and en.wiki to fix banner override bug' | [production] | 
            
  | 00:30 | <logmsgbot_> | maxsem Finished syncing Wikimedia installation... : | [production] | 
            
  
    | 2012-10-30
      
      § | 
    
  | 23:44 | <logmsgbot_> | maxsem Started syncing Wikimedia installation... : | [production] | 
            
  | 23:08 | <mutante> | replaced wikipedia SSL cert with new one by GeoTrust | [production] | 
            
  | 22:16 | <logmsgbot_> | kaldari synchronized php-1.21wmf3/extensions/CentralNotice  'autoloading ApiCentralNoticeAllocations for CentralNotice' | [production] | 
            
  | 22:15 | <LeslieCarr> | failing overRE mastership on cr2-eqiad to upgrade re1 | [production] | 
            
  | 22:02 | <logmsgbot_> | asher synchronized wmf-config/CommonSettings.php  'memcached logging group' | [production] | 
            
  | 22:00 | <logmsgbot_> | asher synchronized wmf-config/mc.php  'testing MemcachedPeclBagOStuff on test2wiki (serverlist fix)' | [production] | 
            
  | 21:57 | <logmsgbot_> | asher synchronized wmf-config/mc.php  'testing MemcachedPeclBagOStuff on test2wiki' | [production] | 
            
  | 21:47 | <mutante> | restarting nginx on ssl hosts for cert update | [production] | 
            
  | 21:43 | <logmsgbot_> | reedy synchronized wmf-config/InitialiseSettings.php | [production] | 
            
  | 21:30 | <logmsgbot_> | kaldari synchronized php-1.21wmf3/extensions/CentralNotice  'updating centralNotice to autoload ApiCentralNoticeAllocations class' | [production] | 
            
  | 21:24 | <logmsgbot_> | kaldari synchronized php-1.21wmf2/extensions/CentralNotice  'updating centralNotice to autoload ApiCentralNoticeAllocations class' | [production] | 
            
  | 21:23 | <logmsgbot_> | kaldari Finished syncing Wikimedia installation... : | [production] | 
            
  | 21:14 | <logmsgbot_> | kaldari synchronized php-1.21wmf2/extensions/CentralNotice  'updating centralNotice to autoload ApiCentralNoticeAllocations class' | [production] | 
            
  | 21:09 | <logmsgbot_> | kaldari synchronized php-1.21wmf2/extensions/CentralNotice/CentralNotice.php  'updating centralNotice to autoload ApiCentralNoticeAllocations class' | [production] | 
            
  | 20:54 | <Jeff_Green> | stopping mysql on db1013 for cloning | [production] | 
            
  | 20:37 | <logmsgbot_> | kaldari Started syncing Wikimedia installation... : | [production] | 
            
  | 20:21 | <logmsgbot_> | kaldari synchronized wmf-config/CommonSettings.php  'updating wgCentralBannerDispatcher for CentralNotice' | [production] | 
            
  | 19:46 | <logmsgbot_> | kaldari synchronized wmf-config/CommonSettings.php  'updating settings for CentralNotice banner loading' | [production] | 
            
  | 18:31 | <binasher> | completed centralnotice patch-bucketing.sql migration | [production] | 
            
  | 18:24 | <mutante> | giving sbernardin racktables permissions | [production] | 
            
  | 17:53 | <mark> | Installed Precise on new upload esams servers cp3003-cp3010 | [production] | 
            
  | 17:20 | <mutante> | running sync-apache on srv199 only | [production] | 
            
  | 17:19 | <logmsgbot_> | reedy synchronized wmf-config/CommonSettings.php  'wgMaxAnimatedGifArea = 2.5e7' | [production] | 
            
  | 17:15 | <logmsgbot_> | reedy synchronized wmf-config/omgtestfile | [production] | 
            
  | 17:06 | <Reedy> | Running sync-common on srv199 | [production] | 
            
  | 17:01 | <Reedy> | Running sync-common on srv194 | [production] | 
            
  | 16:54 | <logmsgbot_> | reedy synchronized wmf-config/ExtensionMessages-1.21wmf3.php | [production] | 
            
  | 16:38 | <apergos> | stpped stray jobrunner on srv278 | [production] | 
            
  | 14:38 | <paravoid> | powercycling srv229, dead for unknown reason so far | [production] | 
            
  | 10:08 | <logmsgbot_> | nikerabbit synchronized php-1.21wmf2/extensions/NewUserMessage/NewUserMessage.class.php  'I0f93ee53' | [production] |