| 2012-07-17
      
      § | 
    
  | 23:24 | <Tim> | on srv193: removed core dump files, disabled core dumping, restarted apache | [production] | 
            
  | 19:52 | <mlitn> | Finished syncing Wikimedia installation... : | [production] | 
            
  | 19:09 | <mlitn> | Started syncing Wikimedia installation... : | [production] | 
            
  | 15:18 | <mark> | lvs1002 is back up and idling | [production] | 
            
  | 15:15 | <mark> | Fixed serial console redirection after boot to OFF on lvs1002 | [production] | 
            
  | 15:06 | <mark> | lvs4 is back up and serving traffic | [production] | 
            
  | 14:52 | <mark> | Reinstalling lvs4 with Ubuntu Precise | [production] | 
            
  | 14:48 | <mark> | Stopped PyBal on lvs4, failing over traffic to lvs3 | [production] | 
            
  | 14:47 | <mark> | Reinstalling lvs1002 with Ubuntu Precise | [production] | 
            
  | 14:46 | <mark> | Stopped PyBal on lvs1002, failing over traffic to lvs1005 | [production] | 
            
  | 14:44 | <mark> | lvs1001 is back up and serving traffic | [production] | 
            
  | 14:11 | <mark> | Stopped PyBal on lvs1001, failing over traffic to lvs1004 | [production] | 
            
  | 13:09 | <maplebed> | changed auth URL for swift to use load balancer rather than round robin DNS | [production] | 
            
  | 13:08 | <mark> | lvs1003 is back up and serving traffic | [production] | 
            
  | 13:07 | <maplebed> | changed order on ganglia swift view graphs to group by metric rather than host | [production] | 
            
  | 13:02 | <reedy> | synchronized wmf-config/ | [production] | 
            
  | 12:53 | <mark> | Fixed boot order on lvs1003 | [production] | 
            
  | 12:34 | <mark> | Reinstalling lvs1003 with Ubuntu Precise | [production] | 
            
  | 12:28 | <mark> | Stopped PyBal on lvs1003, failing over traffic to lvs1006 | [production] | 
            
  | 12:10 | <mark> | amslvs1 is back up and serving traffic | [production] | 
            
  | 11:44 | <mark> | Reinstalling amslvs1 with Ubuntu Precise | [production] | 
            
  | 11:40 | <mark> | Stopped PyBal on amslvs1, failing over traffic to amslvs3 | [production] | 
            
  | 11:32 | <mark> | amslvs2 is back up and serving traffic | [production] | 
            
  | 10:47 | <mark> | Reinstalling amslvs2 with Ubuntu Precise | [production] | 
            
  | 10:36 | <mark> | Stopped PyBal on amslvs2, failing over traffic to amslvs4 | [production] | 
            
  | 09:17 | <hashar> | deployment-prep sync upload6 dirs again. root@deployment-nfs-memc:$ rsync -a --progress --inplace /mnt/export/upload6 /data/project/upload6 | [production] | 
            
  | 07:30 | <Tim> | testing envvars/apache2.conf change on srv193 | [production] | 
            
  | 07:09 | <Tim> | restarted apache on srv193 | [production] | 
            
  | 02:58 | <Tim> | graceful restart of all apaches | [production] | 
            
  | 02:52 | <reedy> | synchronized wmf-config/CommonSettings.php | [production] | 
            
  | 02:47 | <LocalisationUpdate> | completed (1.20wmf6) at Tue Jul 17 02:47:06 UTC 2012 | [production] | 
            
  | 02:23 | <LocalisationUpdate> | completed (1.20wmf7) at Tue Jul 17 02:23:41 UTC 2012 | [production] | 
            
  | 00:44 | <Tim> | deploying new redirects.conf | [production] | 
            
  | 00:28 | <Tim> | testing new redirects.conf on mw1 | [production] |