| 
      
        2010-08-03
      
      §
     | 
  
    
  | 22:58 | 
  <mark> | 
  Changed /etc/default/exim4 to make exim listen on SMTP | 
  [production] | 
            
  | 22:56 | 
  <mark> | 
  s/srv9/grosley/ on /etc/exim4/exim4.conf on grosley | 
  [production] | 
            
  | 21:20 | 
  <RobH> | 
  grosley exim config was overwritten when converted to puppet control (default install uses simple exim setup now).  defined host specifically, adding in ganglia details and removing exim control from puppet | 
  [production] | 
            
  | 20:21 | 
  <catrope> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  'bug 24652: Archive namespace for officewiki' | 
  [production] | 
            
  | 20:10 | 
  <catrope> | 
  synchronized php-1.5/wmf-config/CommonSettings.php  'Bump style version appendix' | 
  [production] | 
            
  | 20:09 | 
  <catrope> | 
  synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js  'r70409' | 
  [production] | 
            
  | 16:00 | 
  <catrope> | 
  synchronized php-1.5/languages/messages/MessagesEo.php  'r70387' | 
  [production] | 
            
  | 14:24 | 
  <RobH> | 
  knsq7 coming down for reinstall | 
  [production] | 
            
  | 14:24 | 
  <RobH> | 
  kndq7 coming down for reinstall | 
  [production] | 
            
  | 04:57 | 
  <Andrew> | 
  scap | 
  [production] | 
            
  | 04:57 | 
  <andrew> | 
  synchronizing Wikimedia installation... Revision: 70064 | 
  [production] | 
            
  | 04:55 | 
  <Andrew> | 
  Preparing to update LiquidThreads alpha to trunk with r70106 and r70100 unmerged. | 
  [production] | 
            
  | 01:30 | 
  <mark> | 
  Restored exim config on williams (OTRS) | 
  [production] | 
            
  | 00:35 | 
  <RobH> | 
  otrs emails still bouncing, working on it. | 
  [production] | 
            
  | 00:31 | 
  <RobH> | 
  exim was sitting idle? on williams for otrs delivery after complaining of a failed database connection.  restarted exim, it appears to be working on the delivery backlog now, will check back on it in 30 minutes or so | 
  [production] | 
            
  
    | 
      
        2010-08-02
      
      §
     | 
  
    
  | 23:55 | 
  <guillom> | 
  afaict, all emails sent to OTRS are being rejected with the "retry time not reached for any host after a long failure period" message. The issue seems to have started a few hours ago. | 
  [production] | 
            
  | 22:34 | 
  <tomaszf> | 
  upgrading civicrm to 3.1.6 | 
  [production] | 
            
  | 18:20 | 
  <RobH> | 
  pulling knsq6 for reinstallation and such | 
  [production] | 
            
  | 17:47 | 
  <RobH> | 
  sq33-sq40 reinstalled, online, serving requests | 
  [production] | 
            
  | 15:54 | 
  <RobH> | 
  reinstalling sq34-sq40 | 
  [production] | 
            
  | 14:55 | 
  <mark> | 
  Installed gmetad on streber for collecting ganglia information in local RRDs | 
  [production] | 
            
  | 14:34 | 
  <mark> | 
  Upgraded streber to Ubuntu 10.04 | 
  [production] | 
            
  | 14:24 | 
  <RobH> | 
  restarted pdns on linne | 
  [production] | 
            
  | 14:20 | 
  <RobH> | 
  updated dns for tesla host ci  | 
  [production] | 
            
  | 12:46 | 
  <mark> | 
  Fixed ganglia package mess on search12-20 | 
  [production] | 
            
  | 12:10 | 
  <mark> | 
  Depooled all text squids from the bits.esams LVS pool | 
  [production] | 
            
  | 12:02 | 
  <mark> | 
  Reinstalled knsq2 and knsq5 with Ubuntu 10.04, set them up as Varnish bits caches, and pooled them in LVS | 
  [production] | 
            
  | 11:50 | 
  <mark> | 
  Fixed ganglia mess on kaulen | 
  [production] | 
            
  | 11:43 | 
  <mark> | 
  Added missing Wikimedia APT repository to kaulen. Why was it not there? Was this host installed in some nonstandard way? | 
  [production] | 
            
  | 11:35 | 
  <mark> | 
  Set up exim::simple-mail-sender classes for kaulen | 
  [production] | 
            
  | 11:35 | 
  <mark> | 
  Fixed puppet for snapshot*, base classes were not included | 
  [production] | 
            
  | 11:19 | 
  <mark> | 
  Turned off knsq3; broken HBA and out of warranty | 
  [production] | 
            
  | 10:55 | 
  <mark> | 
  Started puppetd on spence inside gdb | 
  [production] | 
            
  
    | 
      
        2010-08-01
      
      §
     | 
  
    
  | 17:21 | 
  <mark> | 
  Reinstalled linne.wikimedia.org with Lucid | 
  [production] | 
            
  | 15:41 | 
  <mark> | 
  Fixed the haproxy for puppetmaster on brewster, was broken by the upgrade | 
  [production] | 
            
  | 15:19 | 
  <mark> | 
  Upgraded brewster to Ubuntu 10.04 Lucid | 
  [production] | 
            
  | 15:18 | 
  <mark> | 
  Removed Wikimedia repository default pinning on brewster, as it's doing more harm than good | 
  [production] | 
            
  | 15:07 | 
  <mark> | 
  Explicitly install and deinstall gmond / ganglia-monitor packages in puppet, depending on the ubuntu version | 
  [production] | 
            
  | 14:53 | 
  <mark> | 
  Removed broken amanda backup client from brewster. Broken package install/dpkg state, no logs, no documentation | 
  [production] |