| 2012-08-22
      
      § | 
    
  | 06:14 | <asher> | synchronized wmf-config/db.php  'returing db38 as an s1 slave' | [production] | 
            
  | 06:02 | <binasher> | shutting down mysql on db38 and rebooting for kernel upgrade | [production] | 
            
  | 05:50 | <asher> | synchronized wmf-config/db.php  'setting s1 writeable on new master' | [production] | 
            
  | 05:49 | <asher> | synchronized wmf-config/db.php  'setting s1 to read-only, switching master to db63' | [production] | 
            
  | 05:47 | <binasher> | preparing to switch s1 master to db63 (upgrade to 96GB ram and precise) in a minute | [production] | 
            
  | 05:29 | <tstarling> | synchronized wmf-config/InitialiseSettings.php  'enabling Scribunto on mediawiki.org' | [production] | 
            
  | 05:07 | <Tim> | on ms7, running a script to remerge the split math directories back into /mnt/upload6/math, using rsync | [production] | 
            
  | 02:55 | <LocalisationUpdate> | completed (1.20wmf10) at Wed Aug 22 02:55:22 UTC 2012 | [production] | 
            
  | 02:28 | <LocalisationUpdate> | completed (1.20wmf9) at Wed Aug 22 02:27:55 UTC 2012 | [production] | 
            
  | 00:19 | <Tim> | running updateCollation.php for ptwiki in a screen on hume | [production] | 
            
  | 00:15 | <tstarling> | synchronized wmf-config/InitialiseSettings.php  'setting $wgCategoryCollation=uca-default on ptwiki' | [production] | 
            
  | 00:00 | <binasher> | completed user.user_email index migrations | [production] | 
            
  
    | 2012-08-21
      
      § | 
    
  | 23:46 | <maplebed> | deployed change to swift proxy rewrite to only send thumb 404s to the image scalers gerrit change 20916 | [production] | 
            
  | 23:44 | <aaron> | synchronized wmf-config/filebackend.php  'Enabled token/url caching for swift.' | [production] | 
            
  | 23:26 | <Tim> | updated email address for enwiki user account "Giano" | [production] | 
            
  | 23:18 | <binasher> | running user.user_email index migration on s6-7 | [production] | 
            
  | 23:16 | <binasher> | running user.user_email index migration on s5 | [production] | 
            
  | 23:14 | <binasher> | truncated enwiki.flagged(images|templates) | [production] | 
            
  | 23:11 | <binasher> | running user.user_email index migration on s4 | [production] | 
            
  | 22:44 | <asher> | synchronized wmf-config/db.php  'returning db52 as an s2 slave' | [production] | 
            
  | 22:24 | <binasher> | s2 master swap - new repl position MASTER_LOG_FILE='db54-bin.000302', MASTER_LOG_POS=870283287 | [production] | 
            
  | 22:23 | <asher> | synchronized wmf-config/db.php  'setting s2 to writeable' | [production] | 
            
  | 22:22 | <asher> | synchronized wmf-config/db.php  'setting s2 to read-only for master swap to db54' | [production] | 
            
  | 22:09 | <reedy> | synchronized php-1.20wmf10/extensions/EducationProgram | [production] | 
            
  | 21:59 | <binasher> | running purgeParserCache.php for all dbs | [production] | 
            
  | 21:26 | <binasher> | resumed s3 user.user_email migrations | [production] | 
            
  | 20:34 | <Jeff_Green> | hand-tweezed spence:/etc/nagios/nsca_payments.cfg b/c puppet is broken for spence, restarted nagios on spence | [production] | 
            
  | 20:15 | <Jeff_Green> | added hosts boron/indium to nagios's nsca config | [production] | 
            
  | 19:39 | <reedy> | synchronized php-1.20wmf10/resources/  'jQuery UI 1.8.23' | [production] | 
            
  | 19:26 | <paravoid> | rolling back sq51 switch to originals | [production] | 
            
  | 18:52 | <paravoid> | putting sq51 (modified with originals pointing to swift) back into the backend pool | [production] | 
            
  | 18:33 | <RobH> | authdns-update for mc1009-1016 mgmt | [production] | 
            
  | 18:16 | <RobH> | db1047 back online, presends sdb and sdc at 3597.3 GB each | [production] | 
            
  | 18:12 | <RobH> | db1047 disk shelf added, two raid10 arrays of 12 disks each, booting now into OS. | [production] | 
            
  | 18:00 | <RobH> | db1047 shutting down | [production] | 
            
  | 17:58 | <RobH> | db1047 mysql shutting down (going to add new disk shelf to server) | [production] | 
            
  | 16:44 | <AaronSchulz> | Running setZoneAccess.php for all wikis | [production] | 
            
  | 16:40 | <apergos> | shot morebots so the restarter script would restart it | [production] | 
            
  | 05:41 | <binasher> | halting s3 user migration on hiwiktionary, will resume in the morning (pst) | [production] | 
            
  | 04:02 | <binasher> | adding user.user_email index on s3 | [production] | 
            
  | 03:44 | <binasher> | adding user.user_email index on s2 | [production] | 
            
  | 02:37 | <LocalisationUpdate> | completed (1.20wmf10) at Tue Aug 21 02:37:10 UTC 2012 | [production] | 
            
  | 02:21 | <LocalisationUpdate> | completed (1.20wmf9) at Tue Aug 21 02:21:32 UTC 2012 | [production] | 
            
  | 01:47 | <binasher> | adding user.user_email index to enwiki via osc | [production] | 
            
  | 00:32 | <maplebed> | removed ms-be6 from rotation due to bad memory and repeated kernel panics | [production] | 
            
  | 00:23 | <mutante> | fenari - install a couple lib* package upgrades | [production] | 
            
  | 00:10 | <preilly> | synchronized php-1.20wmf10/extensions/MobileFrontend  'weekly deploy' | [production] | 
            
  | 00:09 | <preilly> | synchronized php-1.20wmf9/extensions/MobileFrontend  'weekly deploy' | [production] |