| 
      
        2011-04-22
      
      §
     | 
  
    
  | 23:56 | 
  <domas> | 
  detached gdb from srv193 apache, apparently it was used for something | 
  [production] | 
            
  | 23:14 | 
  <notpeter> | 
  restarting nagios (again)wq | 
  [production] | 
            
  | 22:43 | 
  <notpeter> | 
  restarting nagios | 
  [production] | 
            
  | 19:23 | 
  <apergos> | 
  shot all stopped rsyncs on ms5 (that were copying from ms4 about two weeks ago), changed all perms on the directories they had reached so thumbs can be served/read from them.. oh. not me, someone else must have done it, I'm not here :-P | 
  [production] | 
            
  | 19:02 | 
  <RobH> | 
  ms4 shutting down for memory troubleshooting | 
  [production] | 
            
  | 18:52 | 
  <RobH> | 
  ms4 troubleshooting, disragrd bounces] | 
  [production] | 
            
  | 18:51 | 
  <notpeter> | 
  restarting nagios | 
  [production] | 
            
  | 12:41 | 
  <hcatlin> | 
  Restarting mobile cluster with April code update. | 
  [production] | 
            
  | 00:49 | 
  <notpeter> | 
  restarting nagios. hopefully now with more sms! | 
  [production] | 
            
  
    | 
      
        2011-04-21
      
      §
     | 
  
    
  | 23:32 | 
  <midom> | 
  synchronized php-1.17/includes/ImagePage.php  | 
  [production] | 
            
  | 20:54 | 
  <pdhanda> | 
  synchronized live-1.5/404.php  | 
  [production] | 
            
  | 19:02 | 
  <domas> | 
  bumped up maxclients/serverlimit on singer to 350 (up from 150), set maxrequestsperchild to 30 to avoid heap blowup (down from 0), all governed via apache2/conf.d/maxrequests | 
  [production] | 
            
  | 18:25 | 
  <Ryan_Lane> | 
  restarting apache on singer | 
  [production] | 
            
  | 18:19 | 
  <Ryan_Lane> | 
  applying system patches to raskin | 
  [production] | 
            
  | 17:47 | 
  <Ryan_Lane> | 
  restarting apache on singer | 
  [production] | 
            
  | 17:04 | 
  <pdhanda> | 
  synchronized live-1.5/404.php  | 
  [production] | 
            
  | 16:52 | 
  <rainman-sr> | 
  copying stuff from /home/ariel/searchidx/ to searchidx2 as it didn't get copied | 
  [production] | 
            
  | 15:32 | 
  <awjrichards> | 
  synchronized php-1.17/wmf-config/CommonSettings.php  'Fixing typo to properly check for wgUseEmailCapture' | 
  [production] | 
            
  | 15:29 | 
  <awjrichards> | 
  ran sync-common-all  | 
  [production] | 
            
  | 15:06 | 
  <awjrichards> | 
  synchronized php-1.17/wmf-config/InitialiseSettings.php  'Added section for EmailCapture, enabling on testwiki' | 
  [production] | 
            
  | 15:05 | 
  <awjrichards> | 
  synchronized php-1.17/wmf-config/CommonSettings.php  'Added section for EmailCapture' | 
  [production] | 
            
  | 14:50 | 
  <awjrichards> | 
  synchronizing Wikimedia installation... Revision: 86464:  | 
  [production] | 
            
  | 14:49 | 
  <awjr> | 
  running scap on fenari to dark-deploy EmailCapture extension | 
  [production] | 
            
  | 03:44 | 
  <Ryan_Lane> | 
  uninstalled python-json on wikitech, and installed python-simplejson | 
  [production] | 
            
  | 01:45 | 
  <Ryan_Lane> | 
  testing statusnet library again | 
  [production] | 
            
  | 01:45 | 
  <Ryan_Lane> | 
  testing statusnet library again | 
  [production] | 
            
  | 01:44 | 
  <Ryan_Lane> | 
  testing statusnet library again | 
  [production] | 
            
  | 01:35 | 
  <Ryan_Lane> | 
  testing a statusnet library for morebots that has a sane license | 
  [production] | 
            
  | 01:20 | 
  <Ryan_Lane> | 
  installing ca-certificates on wikitech | 
  [production] | 
            
  | 01:11 | 
  <Ryan_Lane> | 
  installing curl on wikitech | 
  [production] | 
            
  | 01:04 | 
  <Ryan_Lane> | 
  installing python-json on wikitech | 
  [production] | 
            
  | 00:11 | 
  <Ryan_Lane> | 
  added a sweet, special message to morebots | 
  [production] | 
            
  | 00:03 | 
  <notpeter> | 
  moving some searchidx files into puppet's sphere of influnece. | 
  [production] | 
            
  
    | 
      
        2011-04-20
      
      §
     | 
  
    
  | 23:10 | 
  <rainman-sr> | 
  search indexing is up and running again, might take a couple of days for things to catch up though | 
  [production] | 
            
  | 23:07 | 
  <rainman-sr> | 
  doing a slow restart of the whole search cluster (2 minute intervals between host restarts) | 
  [production] | 
            
  | 22:52 | 
  <rainman-sr> | 
  starting search indexer on searchidx2 | 
  [production] | 
            
  | 16:29 | 
  <mark> | 
  Squids finished rebuilding, removed iptables rules | 
  [production] | 
            
  | 16:21 | 
  <mark> | 
  Removed iptables rules on sq77/78 | 
  [production] | 
            
  | 16:09 | 
  <mark> | 
  Downgraded sq65 | 
  [production] | 
            
  | 16:06 | 
  <mark> | 
  removed iptables rule on sq66 | 
  [production] | 
            
  | 16:05 | 
  <Ryan_Lane> | 
  downgrading squid on knsq27, knsq28, knsq29, knsq30, knsq26, knsq25, knsq24 | 
  [production] | 
            
  | 16:02 | 
  <Ryan_Lane> | 
  downgrading squid on amssq31 | 
  [production] | 
            
  | 16:01 | 
  <Ryan_Lane> | 
  downgrading squid on amssq38 | 
  [production] | 
            
  | 16:01 | 
  <Ryan_Lane> | 
  downgrading squid on amssq39 | 
  [production] | 
            
  | 16:01 | 
  <Ryan_Lane> | 
  downgrading squid on amssq40 | 
  [production] | 
            
  | 16:01 | 
  <Ryan_Lane> | 
  downgrading squid on amssq41 | 
  [production] |