| 
      
        2010-02-15
      
      §
     | 
  
    
  | 22:53 | 
  <logmsgbot_> | 
  midom synchronized php-1.5/wmf-config/checkers.php  'un-disabling UA check on API' | 
  [production] | 
            
  | 22:14 | 
  <RoanKattouw> | 
  Resynced srv114, was returning empty responses | 
  [production] | 
            
  | 22:10 | 
  <logmsgbot_> | 
  mark synchronized php-1.5/wmf-config/CommonSettings.php  'Test with 220px default thumb size on enwiki (bug #21117)' | 
  [production] | 
            
  | 22:10 | 
  <logmsgbot_> | 
  mark synchronized php-1.5/wmf-config/InitialiseSettings.php  'Test with 220px default thumb size on enwiki (bug #21117)' | 
  [production] | 
            
  | 22:02 | 
  <logmsgbot_> | 
  catrope synchronized php-1.5/wmf-config/CommonSettings.php  'Bump style version appendix' | 
  [production] | 
            
  | 21:58 | 
  <logmsgbot_> | 
  catrope synchronized php-1.5/extensions/UsabilityInitiative/Vector/Vector.combined.min.js  'r62553' | 
  [production] | 
            
  | 21:39 | 
  <logmsgbot_> | 
  catrope synchronized php-1.5/includes/api/ApiMain.php  'r62550' | 
  [production] | 
            
  | 21:30 | 
  <logmsgbot_> | 
  mark synchronized php-1.5/wmf-config/InitialiseSettings.php  'raised wgSearchSuggestCacheExpiry from 1200 to 86400' | 
  [production] | 
            
  | 21:27 | 
  <logmsgbot_> | 
  catrope synchronized php-1.5/includes/api/ApiOpenSearch.php  'r62548' | 
  [production] | 
            
  | 21:27 | 
  <logmsgbot_> | 
  catrope synchronized php-1.5/includes/api/ApiMain.php  'r62548' | 
  [production] | 
            
  | 21:27 | 
  <logmsgbot_> | 
  catrope synchronized php-1.5/includes/DefaultSettings.php  'r62548' | 
  [production] | 
            
  | 20:54 | 
  <logmsgbot_> | 
  catrope synchronized php-1.5/includes/api/ApiBase.php  'r62546' | 
  [production] | 
            
  | 20:54 | 
  <logmsgbot_> | 
  catrope synchronized php-1.5/api.php  'r62546' | 
  [production] | 
            
  | 20:53 | 
  <domas> | 
  Roan and me are doing some perf engineering on opensearch & related stuff. \\o/ | 
  [production] | 
            
  | 20:40 | 
  <logmsgbot_> | 
  midom synchronized php-1.5/includes/StubObject.php  'readding stub trace' | 
  [production] | 
            
  | 20:34 | 
  <logmsgbot_> | 
  midom synchronized php-1.5/wmf-config/checkers.php  'perf comment action' | 
  [production] | 
            
  | 20:30 | 
  <logmsgbot_> | 
  midom synchronized php-1.5/includes/StubObject.php  'removing trace hooks' | 
  [production] | 
            
  | 20:28 | 
  <logmsgbot_> | 
  midom synchronized php-1.5/includes/StubObject.php  | 
  [production] | 
            
  | 20:24 | 
  <logmsgbot_> | 
  catrope synchronized php-1.5/includes/api/ApiOpenSearch.php  'r62543' | 
  [production] | 
            
  | 19:32 | 
  <mark> | 
  Allowing API requests to be sent to the regular app servers temporarily | 
  [production] | 
            
  | 19:27 | 
  <mark> | 
  Restarted sq31 and sq31 API backend squids with slightly reduced memory settings | 
  [production] | 
            
  | 19:18 | 
  <logmsgbot_> | 
  mark synchronized php-1.5/wmf-config/InitialiseSettings.php  | 
  [production] | 
            
  | 17:51 | 
  <Rob> | 
  set srv127 to false on lvs3, will re-enabled once in DC to reboot it manually | 
  [production] | 
            
  | 17:49 | 
  <logmsgbot_> | 
  root synchronized php-1.5/wmf-config/mc.php  'srv127 down taking out of mc rotation' | 
  [production] | 
            
  | 16:02 | 
  <RoanKattouw> | 
  Resynced srv134 to fix HTTP 500 | 
  [production] | 
            
  | 15:13 | 
  <RoanKattouw> | 
  Resynced srv232, was serving HTTP 500s | 
  [production] | 
            
  | 02:02 | 
  <mark> | 
  Fixed stale PHP file removal apache cron job in Puppet | 
  [production] | 
            
  
    | 
      
        2010-02-13
      
      §
     | 
  
    
  | 17:44 | 
  <mark> | 
  Made 64 bit java exception for search13 in puppet | 
  [production] | 
            
  | 17:37 | 
  <mark> | 
  Converted srv214-218 and srv86-87 into API servers | 
  [production] | 
            
  | 17:02 | 
  <mark> | 
  Throwing all API requests at the new API app server cluster | 
  [production] | 
            
  | 16:49 | 
  <mark> | 
  Converted srv187-srv189 and srv250-srv253 into API app servers | 
  [production] | 
            
  | 15:46 | 
  <mark> | 
  Created new Ganglia group "API application servers", deploying with Puppet | 
  [production] | 
            
  | 15:31 | 
  <mark> | 
  Created API app servers group in Puppet, deploying on srv254-257 to start with | 
  [production] | 
            
  | 15:24 | 
  <mark> | 
  Created API LVS service on lvs3 and restarted PyBal | 
  [production] | 
            
  | 15:10 | 
  <mark> | 
  Added DNS entry for api.svc.pmpta.wmnet (LVS ip) | 
  [production] | 
            
  | 14:18 | 
  <mark> | 
  Enabled more verbose logging on all squids: debug message level 1 for all message categories | 
  [production] | 
            
  | 14:18 | 
  <mark> | 
  Disabled cache digests on all squids | 
  [production] | 
            
  | 13:58 | 
  <mark> | 
  Removing sq30 and lower from the Text CARP pool | 
  [production] | 
            
  | 13:51 | 
  <mark> | 
  Reducing persistent_connection_timeout and request_timeout from 1 min to 20s on all frontend squids, to reduce FD usage | 
  [production] | 
            
  | 12:50 | 
  <mark> | 
  Restarting knsq* frontend squids with FD limit raised from 32k to 64k | 
  [production] | 
            
  | 12:23 | 
  <mark> | 
  Grown filesystem /dev/data/ES (/a) on ms3 by 90G, leaving another 98G free | 
  [production] | 
            
  | 00:17 | 
  <catrope> | 
  synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php  'Sync CSS style versions' | 
  [production] | 
            
  | 00:16 | 
  <catrope> | 
  synchronized php-1.5/wmf-config/CommonSettings.php  'Bump style version appendix' | 
  [production] | 
            
  | 00:10 | 
  <catrope> | 
  synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js  'r62403' | 
  [production] | 
            
  | 00:02 | 
  <catrope> | 
  synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js  'r62400' | 
  [production] | 
            
  | 00:01 | 
  <catrope> | 
  synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js  'r62400' | 
  [production] |