| 2010-10-18
      
      § | 
    
  | 20:10 | <Ryan_Lane> | adding ldap management scripts to formey via puppet class ldap::client::sdk | [production] | 
            
  | 15:44 | <JeLuF> | Set default charset for http://noc.wikimedia.org/conf/ to utf-8 | [production] | 
            
  | 14:52 | <JeLuF> | killed stuck runJobs.php tasks on srv{150-170} | [production] | 
            
  | 14:36 | <jeluf> | synchronized php-1.5/wmf-config/db.php  'removed db28 from s5' | [production] | 
            
  | 14:32 | <jeluf> | synchronized php-1.5/wmf-config/db.php  'removed db13 from s2' | [production] | 
            
  | 13:44 | <RoanKattouw> | Removed trailing ?> from mc.php, InitialiseSettings.php, lucene.php, wgConf.php, secure.php, checkers.php | [production] | 
            
  | 06:44 | <jeluf> | synchronized php-1.5/wmf-config/InitialiseSettings.php  '25557 -  Index namespace on Telugu Wikisource' | [production] | 
            
  | 06:41 | <jeluf> | synchronized php-1.5/wmf-config/InitialiseSettings.php  '25554 -  Change the "upload file" link on jawiki' | [production] | 
            
  
    | 2010-10-17
      
      § | 
    
  | 22:06 | <apergos> | edited /etc/apache2/envvars in place and set APACHE_ULIMIT_MAX_FILES  to 2000, hoping that will address the "too many open files" issue | [production] | 
            
  | 21:29 | <hcatlin> | mobile3 is having file limit issues. Turned off apache so that is de-pooled. Issue description sent to Ariel for RT submission. | [production] | 
            
  | 18:17 | <mark> | Restarted all varnish processes on sq67-70 | [production] | 
            
  | 18:10 | <mark> | authdns-scenario normal | [production] | 
            
  | 18:07 | <apergos> | stopped and restarted varnish on knsq1, 4... loads back to normal there now | [production] | 
            
  | 18:05 | <mark> | Powercycled sq69 | [production] | 
            
  | 18:00 | <Ryan_Lane> | restarting varnish on esams bits | [production] | 
            
  | 17:50 | <JeLuF> | Setting scenario to esams-down. | [production] | 
            
  | 17:47 | <JeLuF> | restarted varnish on knsq1, load average was 5149.87 | [production] | 
            
  | 15:39 | <jeluf> | ran sync-common-all  'Bug 25056 - Create new wiki for Persian Wikinews' | [production] | 
            
  | 15:15 | <jeluf> | ran sync-common-all  'Bug 25130 - Create Wikipedia in Komi-Permyak' | [production] | 
            
  | 14:42 | <jeluf> | ran sync-common-all  'Bug 25307 - Create Wikipedia in Banjar' | [production] | 
            
  | 14:03 | <jeluf> | ran sync-common-all  'Bug 25544 - Create a new Wikipedia in Hill Mari' | [production] | 
            
  | 12:59 | <jeluf> | synchronized php-1.5/wmf-config/InitialiseSettings.php  '25345 - outreachwiki import sources' | [production] | 
            
  
    | 2010-10-14
      
      § | 
    
  | 20:36 | <Ryan_Lane> | also added rkattouw to the engineering alias | [production] | 
            
  | 20:34 | <Ryan_Lane> | adding ^demon and myself to the engineering alias | [production] | 
            
  | 20:00 | <Ryan_Lane> | adding ldap::client::wmf-cluster include to formey in puppet. Adding openldap client as well. | [production] | 
            
  | 19:59 | <Ryan_Lane> | enabling opendj replication on nfs1 and nfs2 | [production] | 
            
  | 18:01 | <hcatlin> | mobiles are acting up. keep an eye on them! | [production] | 
            
  | 17:20 | <mark> | Fixed mobile by restarting apache on mobile2 | [production] | 
            
  | 17:15 | <hcatlin> | LVS is down for mobile... not sure why | [production] | 
            
  | 17:05 | <hcatlin> | mobile deploy is complete! | [production] | 
            
  | 15:52 | <hcatlin> | deploying updates to mobile cluster | [production] | 
            
  | 13:13 | <mark> | Deployed logrotate file for squid-frontend | [production] | 
            
  | 12:34 | <mark> | Fixed gmond on sq61 | [production] | 
            
  | 12:31 | <mark> | restarted scaling on srv222 | [production] |