| 2010-08-02
      
      § | 
    
  | 23:55 | <guillom> | afaict, all emails sent to OTRS are being rejected with the "retry time not reached for any host after a long failure period" message. The issue seems to have started a few hours ago. | [production] | 
            
  | 22:34 | <tomaszf> | upgrading civicrm to 3.1.6 | [production] | 
            
  | 18:20 | <RobH> | pulling knsq6 for reinstallation and such | [production] | 
            
  | 17:47 | <RobH> | sq33-sq40 reinstalled, online, serving requests | [production] | 
            
  | 15:54 | <RobH> | reinstalling sq34-sq40 | [production] | 
            
  | 14:55 | <mark> | Installed gmetad on streber for collecting ganglia information in local RRDs | [production] | 
            
  | 14:34 | <mark> | Upgraded streber to Ubuntu 10.04 | [production] | 
            
  | 14:24 | <RobH> | restarted pdns on linne | [production] | 
            
  | 14:20 | <RobH> | updated dns for tesla host ci | [production] | 
            
  | 12:46 | <mark> | Fixed ganglia package mess on search12-20 | [production] | 
            
  | 12:10 | <mark> | Depooled all text squids from the bits.esams LVS pool | [production] | 
            
  | 12:02 | <mark> | Reinstalled knsq2 and knsq5 with Ubuntu 10.04, set them up as Varnish bits caches, and pooled them in LVS | [production] | 
            
  | 11:50 | <mark> | Fixed ganglia mess on kaulen | [production] | 
            
  | 11:43 | <mark> | Added missing Wikimedia APT repository to kaulen. Why was it not there? Was this host installed in some nonstandard way? | [production] | 
            
  | 11:35 | <mark> | Set up exim::simple-mail-sender classes for kaulen | [production] | 
            
  | 11:35 | <mark> | Fixed puppet for snapshot*, base classes were not included | [production] | 
            
  | 11:19 | <mark> | Turned off knsq3; broken HBA and out of warranty | [production] | 
            
  | 10:55 | <mark> | Started puppetd on spence inside gdb | [production] | 
            
  
    | 2010-08-01
      
      § | 
    
  | 17:21 | <mark> | Reinstalled linne.wikimedia.org with Lucid | [production] | 
            
  | 15:41 | <mark> | Fixed the haproxy for puppetmaster on brewster, was broken by the upgrade | [production] | 
            
  | 15:19 | <mark> | Upgraded brewster to Ubuntu 10.04 Lucid | [production] | 
            
  | 15:18 | <mark> | Removed Wikimedia repository default pinning on brewster, as it's doing more harm than good | [production] | 
            
  | 15:07 | <mark> | Explicitly install and deinstall gmond / ganglia-monitor packages in puppet, depending on the ubuntu version | [production] | 
            
  | 14:53 | <mark> | Removed broken amanda backup client from brewster. Broken package install/dpkg state, no logs, no documentation | [production] | 
            
  
    | 2010-07-28
      
      § | 
    
  | 15:26 | <Rob> | running updates on sockpuppet | [production] | 
            
  | 13:08 | <andrew> | synchronizing Wikimedia installation... Revision: 70064 | [production] | 
            
  | 12:57 | <Andrew> | about to update LiquidThreads alpha to trunk state | [production] | 
            
  | 07:54 | <tstarling> | synchronized php-1.5/wmf-config/CommonSettings.php  'adding 1.16 branch to ExtensionDistributor' | [production] | 
            
  | 05:01 | <apergos> | added bayes to list of clients for exports on dataset1; added /data to fstab on bayes for stats use | [production] | 
            
  | 04:23 | <tstarling> | synchronized php-1.5/includes/api/ApiBase.php | [production] | 
            
  | 04:22 | <tstarling> | synchronized php-1.5/includes/api/ApiMain.php | [production] | 
            
  | 04:22 | <Tim> | deploying r70063 and r70064 to fix API fatals | [production] | 
            
  | 02:09 | <root> | synchronizing Wikimedia installation... Revision: 70061 | [production] | 
            
  | 02:04 | <Tim> | doing svn up/scap to r70061, to get the API cache header fix | [production] |