| 2010-07-16
      
      § | 
    
  | 21:05 | <mark> | Removed old-style DNS monitoring from Nagios conf.php, now fully Puppet managed | [production] | 
            
  | 21:03 | <mark> | Deployed authoritative DNS on nescio.esams, and moved the service IP | [production] | 
            
  | 19:32 | <mark> | Deployed PowerDNS recursor on nescio, and moved the 91.198.174.6 service ip to it | [production] | 
            
  | 19:16 | <mark> | Installed lucid on new server nescio.esams | [production] | 
            
  | 13:49 | <mark> | Fixed entry in upload.wikimedia.org georecord, sent to text squids for a few ip ranges | [production] | 
            
  | 12:27 | <mark> | Pointed country code 'eu' (127.0.255.1 or 65281) to esams in geodns; since geobackend uses a signed short, I had to mask it to 0x7ff / 32513 in the director map | [production] | 
            
  | 10:49 | <mark> | Shutdown fuchsia for decommissioning | [production] | 
            
  | 07:57 | <catrope> | synchronizing Wikimedia installation... Revision: 69416 | [production] | 
            
  | 07:56 | <RoanKattouw> | Running scap to deploy r69416 | [production] | 
            
  | 01:48 | <tstarling> | synchronized php-1.5/includes/diff/DifferenceInterface.php  'r69414' | [production] | 
            
  
    | 2010-07-15
      
      § | 
    
  | 19:23 | <mark> | Replaced all occurrences of 'rr.wikimedia.org' with 'text.wikimedia.org' in DNS | [production] | 
            
  | 19:14 | <mark> | Updated IP of deprecated record rr.esams.wikimedia.org | [production] | 
            
  | 19:10 | <mark> | Started PyBal on amslvs1 with a new config; it automatically picked up the traffic for both text.esams (91.198.174.232) and bits.esams (91.198.174.233) | [production] | 
            
  | 19:07 | <mark> | Stopped PyBal on amslvs1, BGP and OSPF did an automatic failover of bits.esams (91.198.174.233) to amslvs3 | [production] | 
            
  | 18:59 | <mark> | Removed IP 91.198.174.2 (old text squids service ip) from amslvs1. Anyone still using the old IP after weeks will now be unable to reach our sites. | [production] | 
            
  | 18:56 | <mark> | Depooled knsq1-knsq7 in PyBal | [production] | 
            
  | 17:38 | <Fred> | fixed nfs mounts on Bayes. | [production] | 
            
  | 15:35 | <apergos> | chowned  /mnt/upload6/private/ExtensionDistributor/mw-snapshot/trunk/extensions tree to extdist.  ExtensionDistributor apparently working now | [production] | 
            
  | 15:01 | <apergos> | running svn cleanup on /mnt/upload6/private/ExtensionDistributor/mw-snapshot/trunk/extensions as extdist user | [production] | 
            
  | 12:34 | <tstarling> | synchronizing Wikimedia installation... Revision: 69381 | [production] | 
            
  | 12:18 | <Tim> | svn up/scap to r69380 | [production] | 
            
  | 05:13 | <jeluf> | synchronized php-1.5/wmf-config/InitialiseSettings.php  '24321 - ml.wikiquote.org lost its project namespace' | [production] | 
            
  
    | 2010-07-14
      
      § | 
    
  | 23:44 | <Fred> | re-added ccron job to periodically save rrds on our ganglia server. (cron job seems to have vanished for some reason) | [production] | 
            
  | 17:59 | <catrope> | synchronized php-1.5/wmf-config/InitialiseSettings.php  'Favicon for wikimaniateamwiki per Guillaume' | [production] | 
            
  | 16:06 | <Fred> | restarted apache on mobile1 (had begun to return 500) | [production] | 
            
  | 14:07 | <mark> | Fixed memcached on srv110 | [production] | 
            
  | 12:19 | <mark> | Fixed ganglia and puppet on stafford | [production] | 
            
  | 11:54 | <mark> | Migrated DNS monitoring to puppet | [production] | 
            
  | 10:31 | <mark> | Migrated ZFS RAID nagios check to puppet | [production] | 
            
  | 10:14 | <mark> | Migrated monitoring of lucene to puppet | [production] | 
            
  | 09:37 | <mark> | Migrated monitoring of image scalers to puppet | [production] | 
            
  | 08:49 | <Tim> | using stafford for some pbuilder experimentation | [production] |