| 2012-07-18
      
      § | 
    
  | 22:02 | <notpeter> | ok, changed my mind one more time. removing srv194 from apaches pool for precise testing | [production] | 
            
  | 21:41 | <notpeter> | scratch that. removing srv289 from apaches pool for precise testing, not mw1 | [production] | 
            
  | 21:39 | <notpeter> | removing mw1 from apaches pool to do precise test install | [production] | 
            
  | 21:28 | <notpeter> | adding php5 packages to precise-wikimedia repo | [production] | 
            
  | 19:45 | <cmjohnson1> | srv281 powering down for HW checks | [production] | 
            
  | 19:09 | <cmjohnson1> | srv278 powering down for hardware problems -- random reboot system already depooled | [production] | 
            
  | 18:05 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: 295 remaining wikis to 1.20wmf7 | [production] | 
            
  | 17:37 | <cmjohnson1> | mw8 powering down to replace DIMM A3 rt 3273 | [production] | 
            
  | 17:25 | <cmjohnson1> | search32 powering down to replace DIMM B1 rt 3076 | [production] | 
            
  | 16:55 | <reedy> | synchronized php-1.20wmf7/includes/api/ApiQuerySiteinfo.php | [production] | 
            
  | 12:52 | <pp-pdf1> | upgraded mwlib to 0.13.11, restarted all services | [production] | 
            
  | 07:58 | <hashar> | gallium: added Firefox 14 to Testswarm, disabled Firefox 13. | [production] | 
            
  | 02:47 | <LocalisationUpdate> | completed (1.20wmf6) at Wed Jul 18 02:47:23 UTC 2012 | [production] | 
            
  | 02:24 | <LocalisationUpdate> | completed (1.20wmf7) at Wed Jul 18 02:24:14 UTC 2012 | [production] | 
            
  
    | 2012-07-17
      
      § | 
    
  | 23:24 | <Tim> | on srv193: removed core dump files, disabled core dumping, restarted apache | [production] | 
            
  | 19:52 | <mlitn> | Finished syncing Wikimedia installation... : | [production] | 
            
  | 19:09 | <mlitn> | Started syncing Wikimedia installation... : | [production] | 
            
  | 15:18 | <mark> | lvs1002 is back up and idling | [production] | 
            
  | 15:15 | <mark> | Fixed serial console redirection after boot to OFF on lvs1002 | [production] | 
            
  | 15:06 | <mark> | lvs4 is back up and serving traffic | [production] | 
            
  | 14:52 | <mark> | Reinstalling lvs4 with Ubuntu Precise | [production] | 
            
  | 14:48 | <mark> | Stopped PyBal on lvs4, failing over traffic to lvs3 | [production] | 
            
  | 14:47 | <mark> | Reinstalling lvs1002 with Ubuntu Precise | [production] | 
            
  | 14:46 | <mark> | Stopped PyBal on lvs1002, failing over traffic to lvs1005 | [production] | 
            
  | 14:44 | <mark> | lvs1001 is back up and serving traffic | [production] | 
            
  | 14:11 | <mark> | Stopped PyBal on lvs1001, failing over traffic to lvs1004 | [production] | 
            
  | 13:09 | <maplebed> | changed auth URL for swift to use load balancer rather than round robin DNS | [production] | 
            
  | 13:08 | <mark> | lvs1003 is back up and serving traffic | [production] | 
            
  | 13:07 | <maplebed> | changed order on ganglia swift view graphs to group by metric rather than host | [production] | 
            
  | 13:02 | <reedy> | synchronized wmf-config/ | [production] | 
            
  | 12:53 | <mark> | Fixed boot order on lvs1003 | [production] | 
            
  | 12:34 | <mark> | Reinstalling lvs1003 with Ubuntu Precise | [production] | 
            
  | 12:28 | <mark> | Stopped PyBal on lvs1003, failing over traffic to lvs1006 | [production] | 
            
  | 12:10 | <mark> | amslvs1 is back up and serving traffic | [production] | 
            
  | 11:44 | <mark> | Reinstalling amslvs1 with Ubuntu Precise | [production] | 
            
  | 11:40 | <mark> | Stopped PyBal on amslvs1, failing over traffic to amslvs3 | [production] | 
            
  | 11:32 | <mark> | amslvs2 is back up and serving traffic | [production] | 
            
  | 10:47 | <mark> | Reinstalling amslvs2 with Ubuntu Precise | [production] | 
            
  | 10:36 | <mark> | Stopped PyBal on amslvs2, failing over traffic to amslvs4 | [production] | 
            
  | 09:17 | <hashar> | deployment-prep sync upload6 dirs again. root@deployment-nfs-memc:$ rsync -a --progress --inplace /mnt/export/upload6 /data/project/upload6 | [production] | 
            
  | 07:30 | <Tim> | testing envvars/apache2.conf change on srv193 | [production] | 
            
  | 07:09 | <Tim> | restarted apache on srv193 | [production] | 
            
  | 02:58 | <Tim> | graceful restart of all apaches | [production] | 
            
  | 02:52 | <reedy> | synchronized wmf-config/CommonSettings.php | [production] | 
            
  | 02:47 | <LocalisationUpdate> | completed (1.20wmf6) at Tue Jul 17 02:47:06 UTC 2012 | [production] |