| 2012-01-18
      
      ยง | 
    
  | 21:05 | <Reedy> | Run patch-ug_group-length-increase.sql on all wikis | [production] | 
            
  | 21:04 | <Reedy> | Run patch-uploadstash_chunk.sql on all wikis | [production] | 
            
  | 21:03 | <Reedy> | Run patch-jobs-add-timestamp.sql on all wikis | [production] | 
            
  | 20:55 | <awjr> | update cl_zip5 table for CongressLookup to data in r\t109408 | [production] | 
            
  | 20:43 | <Reedy> | Manually running cleanupUploadStash.php against commonswiki | [production] | 
            
  | 20:42 | <Reedy> | Manually ran cleanupUploadStash.php against enwiki | [production] | 
            
  | 20:31 | <binasher> | db38 in service at a low weight with new lucid kernel and current mysql build | [production] | 
            
  | 20:30 | <RobH> | shutting down db17, confirmed not in db rotation and has no mysql instance active | [production] | 
            
  | 20:30 | <asher> | synchronized wmf-config/db.php  'returning db38 at a lower weight' | [production] | 
            
  | 20:28 | <asher> | synchronized wmf-config/db.php  'pulling db38 again' | [production] | 
            
  | 20:26 | <asher> | synchronized wmf-config/db.php  'returning db38 to service' | [production] | 
            
  | 20:17 | <LeslieCarr> | rebooting spence as it's once again gone crazy | [production] | 
            
  | 20:11 | <binasher> | pulled db38, rebooting for kernel and mysql upgrades | [production] | 
            
  | 20:11 | <asher> | synchronized wmf-config/db.php  'pulling db38 from s1 for upgrade' | [production] | 
            
  | 20:04 | <RobH> | mw1102 coming down for mainboard replacement | [production] | 
            
  | 20:03 | <LeslieCarr> | killing puppet processes on spence | [production] | 
            
  | 19:28 | <Reedy> | Run patch-jobs-add-timestamp.sql on enwiki (jobs table is empty!) | [production] | 
            
  | 19:01 | <mutante> | mw1108 - OS installed, added to puppet, finished catalog run, free for use | [production] | 
            
  | 18:37 | <mutante> | pxe booting mw1108, OS install | [production] | 
            
  | 18:36 | <mutante> | fixed DHCP config for mw1108 on brewster, had the string "Failed to connect to 10.65.1.108." where the MAC address should have been. | [production] | 
            
  | 18:27 | <RobH> | searchidx1001 memory replaced per rt 2208 | [production] | 
            
  | 18:20 | <mutante> | tried to PXE boot mw1108 but no DHCP offers received | [production] | 
            
  | 18:15 | <RobH> | searchidx1001 memory being replaced | [production] | 
            
  | 18:14 | <LeslieCarr> | re-preffing tele2 routes | [production] | 
            
  | 18:11 | <RobH> | db1004 hard disk replaced per rt#2140, rebuilding | [production] | 
            
  | 17:40 | <LeslieCarr> | Draining HE to perform maintenance on the physical port | [production] | 
            
  | 16:57 | <reedy> | synchronized php-1.18/extensions/CongressLookup/SpecialCongressLookup.php  '[[rev:109395|r109395]]' | [production] | 
            
  | 14:45 | <mark> | Changed service IP addresses of lists.wikimedia.org in DNS to US prefixes | [production] | 
            
  | 14:40 | <mark> | Disabled hold_domains on sodium and lily | [production] | 
            
  | 14:28 | <mark> | Setup lily to route lists.wikimedia.org mails to sodium | [production] | 
            
  | 14:21 | <mark> | rsync complete. Running dpkg-reconfigure mailman on sodium | [production] | 
            
  | 13:43 | <demon> | synchronized php-1.18/extensions/CongressLookup/SpecialCongressLookup.php  '[[rev:109362|r109362]]' | [production] | 
            
  | 13:38 | <mark> | Started rsync of selected mailman directories under /var/lib/mailman from lily to sodium | [production] | 
            
  | 13:37 | <mark> | Removed all test messages on the exim4 queue on sodium | [production] | 
            
  | 13:37 | <mark> | Created /var LVM snapshot on lily | [production] | 
            
  | 13:35 | <mark> | Stopped lighttpd on lily | [production] | 
            
  | 13:34 | <mark> | Stopped mailman on lily and sodium | [production] | 
            
  | 13:30 | <mark> | Set hold_domains = lists.wikimedia.org on lily, to hold new lists mails on the queue | [production] | 
            
  | 13:30 | <mark> | Starting mailman migration | [production] | 
            
  | 13:03 | <mutante> | restarted pdns on ns0 | [production] | 
            
  | 08:49 | <neilk> | synchronizing Wikimedia installation... : | [production] | 
            
  | 07:45 | <awjrichards> | synchronized php/extensions/CongressLookup/SpecialCongressLookup.php  '[[rev:109336|r109336]]' | [production] | 
            
  | 06:28 | <awjrichards> | synchronized php/extensions/CongressLookup/SpecialCongressLookup.php  '[[rev:109324|r109324]]' | [production] | 
            
  | 06:24 | <awjrichards> | synchronized php/extensions/CongressLookup/SpecialCongressLookup.php  '[[rev:109322|r109322]]' | [production] | 
            
  | 06:08 | <awjrichards> | synchronized php/extensions/CongressLookup/SpecialCongressLookup.php  '[[rev:109319|r109319]]' | [production] | 
            
  | 05:32 | <laner> | synchronized wmf-config/InitialiseSettings.php  'Disabling CentralNotice for simplewiki' | [production] | 
            
  | 05:25 | <laner> | synchronized wmf-config/InitialiseSettings.php  'Disabling moodbar on enwiki' | [production] | 
            
  | 04:59 | <laner> | synchronized wmf-config/InitialiseSettings.php  'Disabling all editing for enwiki for SOPA blackout' | [production] | 
            
  | 04:50 | <Ryan_Lane> | queuing up changes for totally disabling edits. DO NOT SCAP! DO NOT SYNC InitialiseSettings! | [production] | 
            
  | 04:47 | <Ryan_Lane> | Editing enwiki's MediaWiki:Robots.txt to disallow BannerController for SOPA blackout | [production] |