| 2011-04-22
      
      § | 
    
  | 23:56 | <domas> | detached gdb from srv193 apache, apparently it was used for something | [production] | 
            
  | 23:14 | <notpeter> | restarting nagios (again)wq | [production] | 
            
  | 22:43 | <notpeter> | restarting nagios | [production] | 
            
  | 19:23 | <apergos> | shot all stopped rsyncs on ms5 (that were copying from ms4 about two weeks ago), changed all perms on the directories they had reached so thumbs can be served/read from them.. oh. not me, someone else must have done it, I'm not here :-P | [production] | 
            
  | 19:02 | <RobH> | ms4 shutting down for memory troubleshooting | [production] | 
            
  | 18:52 | <RobH> | ms4 troubleshooting, disragrd bounces] | [production] | 
            
  | 18:51 | <notpeter> | restarting nagios | [production] | 
            
  | 12:41 | <hcatlin> | Restarting mobile cluster with April code update. | [production] | 
            
  | 00:49 | <notpeter> | restarting nagios. hopefully now with more sms! | [production] | 
            
  
    | 2011-04-21
      
      § | 
    
  | 23:32 | <midom> | synchronized php-1.17/includes/ImagePage.php | [production] | 
            
  | 20:54 | <pdhanda> | synchronized live-1.5/404.php | [production] | 
            
  | 19:02 | <domas> | bumped up maxclients/serverlimit on singer to 350 (up from 150), set maxrequestsperchild to 30 to avoid heap blowup (down from 0), all governed via apache2/conf.d/maxrequests | [production] | 
            
  | 18:25 | <Ryan_Lane> | restarting apache on singer | [production] | 
            
  | 18:19 | <Ryan_Lane> | applying system patches to raskin | [production] | 
            
  | 17:47 | <Ryan_Lane> | restarting apache on singer | [production] | 
            
  | 17:04 | <pdhanda> | synchronized live-1.5/404.php | [production] | 
            
  | 16:52 | <rainman-sr> | copying stuff from /home/ariel/searchidx/ to searchidx2 as it didn't get copied | [production] | 
            
  | 15:32 | <awjrichards> | synchronized php-1.17/wmf-config/CommonSettings.php  'Fixing typo to properly check for wgUseEmailCapture' | [production] | 
            
  | 15:29 | <awjrichards> | ran sync-common-all | [production] | 
            
  | 15:06 | <awjrichards> | synchronized php-1.17/wmf-config/InitialiseSettings.php  'Added section for EmailCapture, enabling on testwiki' | [production] | 
            
  | 15:05 | <awjrichards> | synchronized php-1.17/wmf-config/CommonSettings.php  'Added section for EmailCapture' | [production] | 
            
  | 14:50 | <awjrichards> | synchronizing Wikimedia installation... Revision: 86464: | [production] | 
            
  | 14:49 | <awjr> | running scap on fenari to dark-deploy EmailCapture extension | [production] | 
            
  | 03:44 | <Ryan_Lane> | uninstalled python-json on wikitech, and installed python-simplejson | [production] | 
            
  | 01:45 | <Ryan_Lane> | testing statusnet library again | [production] | 
            
  | 01:45 | <Ryan_Lane> | testing statusnet library again | [production] | 
            
  | 01:44 | <Ryan_Lane> | testing statusnet library again | [production] | 
            
  | 01:35 | <Ryan_Lane> | testing a statusnet library for morebots that has a sane license | [production] | 
            
  | 01:20 | <Ryan_Lane> | installing ca-certificates on wikitech | [production] | 
            
  | 01:11 | <Ryan_Lane> | installing curl on wikitech | [production] | 
            
  | 01:04 | <Ryan_Lane> | installing python-json on wikitech | [production] | 
            
  | 00:11 | <Ryan_Lane> | added a sweet, special message to morebots | [production] | 
            
  | 00:03 | <notpeter> | moving some searchidx files into puppet's sphere of influnece. | [production] | 
            
  
    | 2011-04-20
      
      § | 
    
  | 23:10 | <rainman-sr> | search indexing is up and running again, might take a couple of days for things to catch up though | [production] | 
            
  | 23:07 | <rainman-sr> | doing a slow restart of the whole search cluster (2 minute intervals between host restarts) | [production] | 
            
  | 22:52 | <rainman-sr> | starting search indexer on searchidx2 | [production] | 
            
  | 16:29 | <mark> | Squids finished rebuilding, removed iptables rules | [production] | 
            
  | 16:21 | <mark> | Removed iptables rules on sq77/78 | [production] | 
            
  | 16:09 | <mark> | Downgraded sq65 | [production] | 
            
  | 16:06 | <mark> | removed iptables rule on sq66 | [production] | 
            
  | 16:05 | <Ryan_Lane> | downgrading squid on knsq27, knsq28, knsq29, knsq30, knsq26, knsq25, knsq24 | [production] | 
            
  | 16:02 | <Ryan_Lane> | downgrading squid on amssq31 | [production] | 
            
  | 16:01 | <Ryan_Lane> | downgrading squid on amssq38 | [production] | 
            
  | 16:01 | <Ryan_Lane> | downgrading squid on amssq39 | [production] |