| 2011-04-21
      
      § | 
    
  | 15:05 | <awjrichards> | synchronized php-1.17/wmf-config/CommonSettings.php  'Added section for EmailCapture' | [production] | 
            
  | 14:50 | <awjrichards> | synchronizing Wikimedia installation... Revision: 86464: | [production] | 
            
  | 14:49 | <awjr> | running scap on fenari to dark-deploy EmailCapture extension | [production] | 
            
  | 03:44 | <Ryan_Lane> | uninstalled python-json on wikitech, and installed python-simplejson | [production] | 
            
  | 01:45 | <Ryan_Lane> | testing statusnet library again | [production] | 
            
  | 01:45 | <Ryan_Lane> | testing statusnet library again | [production] | 
            
  | 01:44 | <Ryan_Lane> | testing statusnet library again | [production] | 
            
  | 01:35 | <Ryan_Lane> | testing a statusnet library for morebots that has a sane license | [production] | 
            
  | 01:20 | <Ryan_Lane> | installing ca-certificates on wikitech | [production] | 
            
  | 01:11 | <Ryan_Lane> | installing curl on wikitech | [production] | 
            
  | 01:04 | <Ryan_Lane> | installing python-json on wikitech | [production] | 
            
  | 00:11 | <Ryan_Lane> | added a sweet, special message to morebots | [production] | 
            
  | 00:03 | <notpeter> | moving some searchidx files into puppet's sphere of influnece. | [production] | 
            
  
    | 2011-04-20
      
      § | 
    
  | 23:10 | <rainman-sr> | search indexing is up and running again, might take a couple of days for things to catch up though | [production] | 
            
  | 23:07 | <rainman-sr> | doing a slow restart of the whole search cluster (2 minute intervals between host restarts) | [production] | 
            
  | 22:52 | <rainman-sr> | starting search indexer on searchidx2 | [production] | 
            
  | 16:29 | <mark> | Squids finished rebuilding, removed iptables rules | [production] | 
            
  | 16:21 | <mark> | Removed iptables rules on sq77/78 | [production] | 
            
  | 16:09 | <mark> | Downgraded sq65 | [production] | 
            
  | 16:06 | <mark> | removed iptables rule on sq66 | [production] | 
            
  | 16:05 | <Ryan_Lane> | downgrading squid on knsq27, knsq28, knsq29, knsq30, knsq26, knsq25, knsq24 | [production] | 
            
  | 16:02 | <Ryan_Lane> | downgrading squid on amssq31 | [production] | 
            
  | 16:01 | <Ryan_Lane> | downgrading squid on amssq38 | [production] | 
            
  | 16:01 | <Ryan_Lane> | downgrading squid on amssq39 | [production] | 
            
  | 16:01 | <Ryan_Lane> | downgrading squid on amssq40 | [production] | 
            
  | 16:01 | <Ryan_Lane> | downgrading squid on amssq41 | [production] | 
            
  | 16:00 | <Ryan_Lane> | downgrading squid on amssq37 | [production] | 
            
  | 15:59 | <Ryan_Lane> | downgrading squid on amssq36 | [production] | 
            
  | 15:59 | <Ryan_Lane> | downgrading squid on amssq32 | [production] | 
            
  | 15:59 | <Ryan_Lane> | downgrading squid on amssq33 | [production] | 
            
  | 15:58 | <Ryan_Lane> | downgrading squid on amssq34 | [production] | 
            
  | 15:58 | <Ryan_Lane> | downgrading squid on amssq35 | [production] | 
            
  | 15:57 | <mark> | Downgrading Squid on sq71-78 | [production] | 
            
  | 15:57 | <Ryan_Lane> | downgrading squid on amssq42 | [production] | 
            
  | 15:56 | <Ryan_Lane> | downgrading squid on amssq43 | [production] | 
            
  | 15:56 | <Ryan_Lane> | downgrading squid on amssq45 | [production] | 
            
  | 15:54 | <Ryan_Lane> | downgrading squid onamssq45 | [production] | 
            
  | 15:53 | <Ryan_Lane> | downgrading squid on knsq23 | [production] | 
            
  | 15:33 | <RobH> | lowered maxclients more on singer, its going to slow down secure, but hopefully keep it online longer. | [production] | 
            
  | 15:04 | <Ryan_Lane> | downgraded sq71 to squid-2.7.7 | [production] | 
            
  | 14:51 | <mark> | Pulled squid 2.7.9 packages from the lucid-wikimedia reprepro APT repository, reinstated 2.7.7 | [production] | 
            
  | 14:34 | <mark> | Downgrading squid on sq66 | [production] | 
            
  | 14:15 | <mark> | Stopping squid on sq72 | [production] | 
            
  | 14:14 | <RobH> | reduced singer apache maxclients from 400 to 200, hopefully will reduce singer crashes | [production] | 
            
  | 14:09 | <RobH> | singer died, rebooting. | [production] | 
            
  | 14:07 | <RobH> | secure, blogs on singer, bouncing due to singer cpu max | [production] | 
            
  | 13:53 | <RobH> | singer threw invalid cert warning, cpu spike was occuring, restarting apache and load is normalizing | [production] | 
            
  | 11:34 | <catrope> | ran sync-common-all | [production] | 
            
  | 11:33 | <RoanKattouw> | Running sync-common-all to deploy r86464 | [production] | 
            
  | 11:11 | <catrope> | synchronized php-1.17/includes/Exif.php  'Fix for bug 28615 (fatal on image description page)' | [production] |