| 2012-10-22
      
      § | 
    
  | 20:21 | <Reedy> | Running p[opulateFileArchiveSha1.php on enwiki | [production] | 
            
  | 19:06 | <logmsgbot_> | reedy synchronized php-1.21wmf2/includes/api/ApiParse.php | [production] | 
            
  | 18:53 | <binasher> | streaming hotbackup of db1038 to db65, preparing to replace to db22 | [production] | 
            
  | 18:28 | <logmsgbot_> | reedy synchronized php-1.21wmf2/skins/CologneBlue.php | [production] | 
            
  | 18:16 | <logmsgbot_> | reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf2 | [production] | 
            
  | 16:49 | <logmsgbot_> | py gracefulled all apaches | [production] | 
            
  | 16:47 | <Jeff_Green> | stopping mysql on db1025 to clone it to db1013 | [production] | 
            
  | 16:02 | <logmsgbot_> | reedy synchronized wmf-config/ | [production] | 
            
  | 16:01 | <logmsgbot_> | reedy synchronized php-1.21wmf2/extensions/ExtensionDistributor | [production] | 
            
  | 15:48 | <logmsgbot_> | reedy synchronized php-1.21wmf2/extensions/ExtensionDistributor | [production] | 
            
  | 15:43 | <notpeter> | re-adding srv219 and srv220 to rendering pool | [production] | 
            
  | 15:04 | <logmsgbot_> | reedy synchronized php-1.21wmf2/extensions/ExtensionDistributor | [production] | 
            
  | 13:32 | <notpeter> | removing srv219 and srv220 from rendering pool for upgarde to precise | [production] | 
            
  | 09:35 | <mark> | Built new varnish 3.0.3plus~rc1-wm2 packages and inserted them into the precise-wikimedia APT repository | [production] | 
            
  | 02:51 | <logmsgbot_> | LocalisationUpdate completed (1.21wmf1) at Mon Oct 22 02:51:32 UTC 2012 | [production] | 
            
  | 02:31 | <logmsgbot_> | LocalisationUpdate completed (1.21wmf2) at Mon Oct 22 02:30:57 UTC 2012 | [production] | 
            
  
    | 2012-10-21
      
      § | 
    
  | 22:50 | <logmsgbot_> | tstarling synchronized php-1.21wmf2/bin/ulimit4.sh | [production] | 
            
  | 22:39 | <Reedy> | populateFilearchiveSha1.php finished on commonswiki | [production] | 
            
  | 14:18 | <Reedy> | Running mwscript populateFilearchiveSha1.php commonswiki | [production] | 
            
  | 02:49 | <logmsgbot_> | LocalisationUpdate completed (1.21wmf1) at Sun Oct 21 02:49:42 UTC 2012 | [production] | 
            
  | 02:28 | <logmsgbot_> | LocalisationUpdate completed (1.21wmf2) at Sun Oct 21 02:28:50 UTC 2012 | [production] | 
            
  | 01:35 | <paravoid> | powercycling srv219, swap-died while trying to get a core dump | [production] | 
            
  | 01:34 | <logmsgbot_> | reedy synchronized wmf-config/CommonSettings.php  'wgMaxImageArea to 20MP' | [production] | 
            
  | 01:33 | <paravoid> | powercycling srv220/srv224, swapdeath | [production] | 
            
  
    | 2012-10-20
      
      § | 
    
  | 18:51 | <Reedy> | Ran populateFilearchiveSha1.php on metawiki | [production] | 
            
  | 18:21 | <paravoid> | powercycling srv221 & srv222, both dead since ~5h ago | [production] | 
            
  | 17:55 | <RobH_> | updated otrs motd per requst by otrs admins (it referenced a now closed request for comment) | [production] | 
            
  | 17:42 | <Reedy> | Running foreachwiki cleanupUploadStash.php | [production] | 
            
  | 03:13 | <K4-713> | updated the payments cluster to 97e1ba74a39376 | [production] | 
            
  | 02:41 | <logmsgbot_> | LocalisationUpdate completed (1.21wmf1) at Sat Oct 20 02:41:10 UTC 2012 | [production] | 
            
  | 02:21 | <logmsgbot_> | LocalisationUpdate completed (1.21wmf2) at Sat Oct 20 02:21:41 UTC 2012 | [production] | 
            
  | 01:36 | <K4-713> | updated the payments cluster to bb30dab91a7afe862 | [production] | 
            
  | 00:46 | <mutante> | upgrade Bugzilla to 4.0.8 (security patch) | [production] | 
            
  
    | 2012-10-19
      
      § | 
    
  | 23:36 | <logmsgbot_> | reedy synchronized wmf-config/CommonSettings.php  'wgMaxImageArea to 17MP' | [production] | 
            
  | 23:31 | <logmsgbot_> | reedy synchronized php-1.21wmf2/extensions/SwiftCloudFiles | [production] | 
            
  | 23:30 | <LeslieCarr> | shutting down sfo-monitor1 | [production] | 
            
  | 23:27 | <logmsgbot_> | reedy synchronized php-1.21wmf2/includes/api/ApiParse.php | [production] | 
            
  | 23:15 | <binasher> | returning srv194/200 to apache lvs pool | [production] | 
            
  | 22:50 | <mutante> | authdns-update, renaming travel-guide-lb to wikivoyage-lb | [production] | 
            
  | 21:10 | <binasher> | pulled srv194 and srv200 from lvs for memc testing | [production] | 
            
  | 21:02 | <binasher> | upgrading percona-toolkit to 2.1.5 | [production] | 
            
  | 20:18 | <logmsgbot_> | reedy synchronized php-1.21wmf2/includes/api/ApiParse.php | [production] | 
            
  | 19:33 | <logmsgbot_> | aaron synchronized php-1.21wmf2/includes/filebackend/FileBackendStore.php  'deployed 65e47037f6e9da2e81db2bc1374cdac375a82db5' | [production] | 
            
  | 19:26 | <paravoid> | powercycling srv223 too, locked up, nothing in serial console | [production] | 
            
  | 19:25 | <paravoid> | powercycling srv221, locked up, nothing in serial console | [production] | 
            
  | 16:21 | <RobH> | msw-a2-eqiad back online | [production] | 
            
  | 15:54 | <RobH> | msw-a2-eqiad is coming down for replacement per RT 3683.  all mgmt connectivity (only) will be lost to items in A2-EQIAD | [production] | 
            
  | 14:43 | <paravoid> | putting ms-fe1 back into the pool | [production] | 
            
  | 14:14 | <logmsgbot_> | reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commonswiki to 1.21wmf2 | [production] | 
            
  | 13:59 | <logmsgbot_> | reedy rebuilt wikiversions.cdb and synchronized wikiversions files: metawiki to 1.21wmf2 | [production] |