| 
      
        2012-11-16
      
      §
     | 
  
    
  | 03:11 | 
  <Tim> | 
  deleted corrupted memcached value enwiki:gadgets-definition:6 | 
  [production] | 
            
  | 02:48 | 
  <logmsgbot_> | 
  LocalisationUpdate completed (1.21wmf3) at Fri Nov 16 02:48:37 UTC 2012 | 
  [production] | 
            
  | 02:37 | 
  <RoanKattouw> | 
  Forcing puppet run on wtp1 to reinstall Parsoid | 
  [production] | 
            
  | 02:28 | 
  <logmsgbot_> | 
  LocalisationUpdate completed (1.21wmf4) at Fri Nov 16 02:28:52 UTC 2012 | 
  [production] | 
            
  | 01:49 | 
  <logmsgbot_> | 
  aaron synchronized php-1.21wmf4/thumb.php  'remove hacks' | 
  [production] | 
            
  | 01:34 | 
  <logmsgbot_> | 
  aaron synchronized php-1.21wmf4/thumb.php  'more debugging' | 
  [production] | 
            
  | 01:33 | 
  <logmsgbot_> | 
  aaron synchronized php-1.21wmf4/thumb.php | 
  [production] | 
            
  | 01:32 | 
  <logmsgbot_> | 
  aaron synchronized php-1.21wmf4/thumb.php | 
  [production] | 
            
  | 01:17 | 
  <logmsgbot_> | 
  aaron synchronized php-1.21wmf4/thumb.php | 
  [production] | 
            
  | 01:12 | 
  <logmsgbot_> | 
  aaron synchronized php-1.21wmf4/thumb.php  'debug logging' | 
  [production] | 
            
  | 00:35 | 
  <logmsgbot_> | 
  tstarling synchronized wmf-config/InitialiseSettings.php | 
  [production] | 
            
  | 00:35 | 
  <logmsgbot_> | 
  tstarling synchronized wmf-config/CommonSettings.php | 
  [production] | 
            
  | 00:30 | 
  <logmsgbot_> | 
  tstarling synchronized wmf-config/InitialiseSettings.php | 
  [production] | 
            
  | 00:29 | 
  <logmsgbot_> | 
  tstarling synchronized wmf-config/InitialiseSettings.php | 
  [production] | 
            
  | 00:28 | 
  <logmsgbot_> | 
  tstarling synchronized small.dblist | 
  [production] | 
            
  | 00:21 | 
  <logmsgbot_> | 
  tstarling synchronized php-1.21wmf4/extensions/UniversalLanguageSelector | 
  [production] | 
            
  
    | 
      
        2012-11-15
      
      §
     | 
  
    
  | 22:41 | 
  <logmsgbot_> | 
  reedy synchronized php-1.21wmf4/extensions/ReaderFeedback/ | 
  [production] | 
            
  | 22:36 | 
  <logmsgbot_> | 
  reedy synchronized php-1.21wmf4/extensions/WikimediaIncubator/ | 
  [production] | 
            
  | 22:29 | 
  <Reedy> | 
  Ran sync-docroot | 
  [production] | 
            
  | 22:08 | 
  <notpeter> | 
  repooling srv25-srv257 | 
  [production] | 
            
  | 22:06 | 
  <notpeter> | 
  repooling srv190-srv192, srv225-srv230 | 
  [production] | 
            
  | 21:33 | 
  <binasher> | 
  starting apache on srv249 | 
  [production] | 
            
  | 21:32 | 
  <binasher> | 
  stopping apache on srv249 | 
  [production] | 
            
  | 21:30 | 
  <binasher> | 
  ran chmod 755 /usr/local/apache/common-local/wmf-config on srv249 | 
  [production] | 
            
  | 20:46 | 
  <Jeff_Green> | 
  adjusted key_buffer_size for fundraisingdbs from 1M to 1G | 
  [production] | 
            
  | 20:42 | 
  <notpeter> | 
  removing srv225-srv230 for upgrade to precise | 
  [production] | 
            
  | 20:34 | 
  <notpeter> | 
  reimaging srv190-srv192 as regular apaches | 
  [production] | 
            
  | 20:09 | 
  <logmsgbot_> | 
  reedy synchronized php-1.21wmf3/ | 
  [production] | 
            
  | 20:05 | 
  <logmsgbot_> | 
  reedy synchronized php-1.21wmf4/ | 
  [production] | 
            
  | 19:37 | 
  <notpeter> | 
  repooling srv200-srv213 | 
  [production] | 
            
  | 19:23 | 
  <notpeter> | 
  removing srv251-srv257 for upgrade to precise | 
  [production] | 
            
  | 19:19 | 
  <logmsgbot_> | 
  reedy synchronized wmf-config/InitialiseSettings.php | 
  [production] | 
            
  | 18:59 | 
  <paravoid> | 
  upgrading srv{190,219,220,221,222,223,224}.pmtpa.wmnet for libtiff vulnerabilities (USN-1631-1) | 
  [production] | 
            
  | 17:40 | 
  <cmjohnson1> | 
  wtp1  raid1 config is wrong; bringing server down to fix | 
  [production] | 
            
  | 17:12 | 
  <notpeter> | 
  removing srv248, srv249, and srv250 from various pools for upgrade to precise | 
  [production] | 
            
  | 16:47 | 
  <paravoid> | 
  running 'reprepro update' to fetch ceph packages | 
  [production] | 
            
  | 16:47 | 
  <cmjohnson1> | 
  shutting down srv250 to verify drac config | 
  [production] | 
            
  | 15:01 | 
  <logmsgbot_> | 
  reedy synchronized php-1.21wmf4/extensions/SiteMatrix/ | 
  [production] | 
            
  | 14:32 | 
  <notpeter> | 
  removing srv200-srv213 from pybal for upgrades to precise (and this time I mean it) | 
  [production] | 
            
  | 13:48 | 
  <logmsgbot_> | 
  py synchronized wmf-config/db.php  'pulling es2 for conversion to inno' | 
  [production] | 
            
  | 11:26 | 
  <logmsgbot_> | 
  kaldari synchronized php-1.21wmf4/extensions/UploadWizard/resources/mw.UploadWizardUploadInterface.js  'refixing for Safari, part 2' | 
  [production] | 
            
  | 11:26 | 
  <logmsgbot_> | 
  kaldari synchronized php-1.21wmf4/extensions/UploadWizard/resources/mw.UploadWizardDetails.js  'refixing for Safari, part 1' | 
  [production] | 
            
  | 11:15 | 
  <logmsgbot_> | 
  kaldari synchronized php-1.21wmf4/extensions/UploadWizard/resources/mw.UploadWizardUploadInterface.js  'refixing for Safari, part 2' | 
  [production] | 
            
  | 11:14 | 
  <logmsgbot_> | 
  kaldari synchronized php-1.21wmf4/extensions/UploadWizard/resources/mw.UploadWizardDetails.js  'refixing for Safari, part 1' | 
  [production] | 
            
  | 11:04 | 
  <logmsgbot_> | 
  kaldari synchronized php-1.21wmf4/extensions/UploadWizard/resources/mw.UploadWizardDetails.js  'rolling back my fix for Safari, part 1' | 
  [production] | 
            
  | 11:04 | 
  <logmsgbot_> | 
  kaldari synchronized php-1.21wmf4/extensions/UploadWizard/resources/mw.UploadWizardUploadInterface.js  'rolling back my fix for Safari, part 2' | 
  [production] | 
            
  | 10:48 | 
  <logmsgbot_> | 
  kaldari synchronized php-1.21wmf4/extensions/UploadWizard/resources/mw.UploadWizardUploadInterface.js  'syncing UploadWizard fix for Safari, part 2' | 
  [production] | 
            
  | 10:47 | 
  <logmsgbot_> | 
  kaldari synchronized php-1.21wmf4/extensions/UploadWizard/resources/mw.UploadWizardDetails.js  'syncing UploadWizard fix for Safari, part 1' | 
  [production] | 
            
  | 09:14 | 
  <logmsgbot_> | 
  aaron synchronized php-1.21wmf4/maintenance/nextJobDB.php  'deployed 2e0c24df43588bd4ceba5522bde1b3f06fbd05b0' | 
  [production] | 
            
  | 07:25 | 
  <Nemo_bis> | 
  Job queue still suspect since the wmf4 deploy, job runners activity halved; see [[bugzilla:41656|bug 41656]]  | 
  [production] |