| 2012-01-18
      
      § | 
    
  | 23:59 | <asher> | synchronized wmf-config/db.php  'returning db32 at a low weight' | [production] | 
            
  | 23:50 | <binasher> | rebooting db32 for mysql/kernel upgrades | [production] | 
            
  | 23:49 | <asher> | synchronized wmf-config/db.php  'pulling db32 from s1 for mysql/kernel upgrades' | [production] | 
            
  | 23:44 | <awjrichards> | synchronized php/extensions/CongressLookup/SpecialCongressLookup.php  '[[rev:109457|r109457]]' | [production] | 
            
  | 23:02 | <maplebed> | increased the size of db11's logical volume for /a from 500G to 800G. | [production] | 
            
  | 22:27 | <binasher> | enwiki master changed to db36 - MASTER_LOG_FILE='db36-bin.000599', MASTER_LOG_POS=15773827 | [production] | 
            
  | 22:26 | <asher> | synchronized wmf-config/db.php  'done swapping s1 master to db36' | [production] | 
            
  | 22:25 | <binasher> | swapping s1 master to db36 | [production] | 
            
  | 22:24 | <asher> | synchronized wmf-config/db.php  'starting swap of s1 master to db36, s1 in read-only' | [production] | 
            
  | 22:13 | <asher> | synchronized wmf-config/db.php  'returning db36 to normal weight' | [production] | 
            
  | 22:07 | <asher> | synchronized wmf-config/db.php  'returning db36 at a low weight' | [production] | 
            
  | 21:59 | <awjrichards> | synchronized php/extensions/CongressLookup/SpecialCongressLookup.php  '[[rev:109440|r109440]]' | [production] | 
            
  | 21:58 | <binasher> | rebooting db36, upgrading kernel + mysql | [production] | 
            
  | 21:56 | <asher> | synchronized wmf-config/db.php  'pulling db36 from s1 for mysql/kernel upgrades' | [production] | 
            
  | 21:54 | <Ryan_Lane> | installing python-wurfl on stat1 | [production] | 
            
  | 21:35 | <Ryan_Lane> | installing geoip-bin geoip-database libgeoip1 python-geoip on stat1 | [production] | 
            
  | 21:13 | <asher> | synchronized wmf-config/db.php  'returning db38 at prior weight' | [production] | 
            
  | 21:05 | <Reedy> | Run patch-ug_group-length-increase.sql on all wikis | [production] | 
            
  | 21:04 | <Reedy> | Run patch-uploadstash_chunk.sql on all wikis | [production] | 
            
  | 21:03 | <Reedy> | Run patch-jobs-add-timestamp.sql on all wikis | [production] | 
            
  | 20:55 | <awjr> | update cl_zip5 table for CongressLookup to data in r\t109408 | [production] | 
            
  | 20:43 | <Reedy> | Manually running cleanupUploadStash.php against commonswiki | [production] | 
            
  | 20:42 | <Reedy> | Manually ran cleanupUploadStash.php against enwiki | [production] | 
            
  | 20:31 | <binasher> | db38 in service at a low weight with new lucid kernel and current mysql build | [production] | 
            
  | 20:30 | <RobH> | shutting down db17, confirmed not in db rotation and has no mysql instance active | [production] | 
            
  | 20:30 | <asher> | synchronized wmf-config/db.php  'returning db38 at a lower weight' | [production] | 
            
  | 20:28 | <asher> | synchronized wmf-config/db.php  'pulling db38 again' | [production] | 
            
  | 20:26 | <asher> | synchronized wmf-config/db.php  'returning db38 to service' | [production] | 
            
  | 20:17 | <LeslieCarr> | rebooting spence as it's once again gone crazy | [production] | 
            
  | 20:11 | <binasher> | pulled db38, rebooting for kernel and mysql upgrades | [production] | 
            
  | 20:11 | <asher> | synchronized wmf-config/db.php  'pulling db38 from s1 for upgrade' | [production] | 
            
  | 20:04 | <RobH> | mw1102 coming down for mainboard replacement | [production] | 
            
  | 20:03 | <LeslieCarr> | killing puppet processes on spence | [production] | 
            
  | 19:28 | <Reedy> | Run patch-jobs-add-timestamp.sql on enwiki (jobs table is empty!) | [production] | 
            
  | 19:01 | <mutante> | mw1108 - OS installed, added to puppet, finished catalog run, free for use | [production] | 
            
  | 18:37 | <mutante> | pxe booting mw1108, OS install | [production] | 
            
  | 18:36 | <mutante> | fixed DHCP config for mw1108 on brewster, had the string "Failed to connect to 10.65.1.108." where the MAC address should have been. | [production] | 
            
  | 18:27 | <RobH> | searchidx1001 memory replaced per rt 2208 | [production] | 
            
  | 18:20 | <mutante> | tried to PXE boot mw1108 but no DHCP offers received | [production] | 
            
  | 18:15 | <RobH> | searchidx1001 memory being replaced | [production] | 
            
  | 18:14 | <LeslieCarr> | re-preffing tele2 routes | [production] | 
            
  | 18:11 | <RobH> | db1004 hard disk replaced per rt#2140, rebuilding | [production] | 
            
  | 17:40 | <LeslieCarr> | Draining HE to perform maintenance on the physical port | [production] | 
            
  | 16:57 | <reedy> | synchronized php-1.18/extensions/CongressLookup/SpecialCongressLookup.php  '[[rev:109395|r109395]]' | [production] | 
            
  | 14:45 | <mark> | Changed service IP addresses of lists.wikimedia.org in DNS to US prefixes | [production] |