| 2010-02-15
      
      § | 
    
  | 21:27 | <logmsgbot_> | catrope synchronized php-1.5/includes/DefaultSettings.php  'r62548' | [production] | 
            
  | 20:54 | <logmsgbot_> | catrope synchronized php-1.5/includes/api/ApiBase.php  'r62546' | [production] | 
            
  | 20:54 | <logmsgbot_> | catrope synchronized php-1.5/api.php  'r62546' | [production] | 
            
  | 20:53 | <domas> | Roan and me are doing some perf engineering on opensearch & related stuff. \\o/ | [production] | 
            
  | 20:40 | <logmsgbot_> | midom synchronized php-1.5/includes/StubObject.php  'readding stub trace' | [production] | 
            
  | 20:34 | <logmsgbot_> | midom synchronized php-1.5/wmf-config/checkers.php  'perf comment action' | [production] | 
            
  | 20:30 | <logmsgbot_> | midom synchronized php-1.5/includes/StubObject.php  'removing trace hooks' | [production] | 
            
  | 20:28 | <logmsgbot_> | midom synchronized php-1.5/includes/StubObject.php | [production] | 
            
  | 20:24 | <logmsgbot_> | catrope synchronized php-1.5/includes/api/ApiOpenSearch.php  'r62543' | [production] | 
            
  | 19:32 | <mark> | Allowing API requests to be sent to the regular app servers temporarily | [production] | 
            
  | 19:27 | <mark> | Restarted sq31 and sq31 API backend squids with slightly reduced memory settings | [production] | 
            
  | 19:18 | <logmsgbot_> | mark synchronized php-1.5/wmf-config/InitialiseSettings.php | [production] | 
            
  | 17:51 | <Rob> | set srv127 to false on lvs3, will re-enabled once in DC to reboot it manually | [production] | 
            
  | 17:49 | <logmsgbot_> | root synchronized php-1.5/wmf-config/mc.php  'srv127 down taking out of mc rotation' | [production] | 
            
  | 16:02 | <RoanKattouw> | Resynced srv134 to fix HTTP 500 | [production] | 
            
  | 15:13 | <RoanKattouw> | Resynced srv232, was serving HTTP 500s | [production] | 
            
  | 02:02 | <mark> | Fixed stale PHP file removal apache cron job in Puppet | [production] | 
            
  
    | 2010-02-13
      
      § | 
    
  | 17:44 | <mark> | Made 64 bit java exception for search13 in puppet | [production] | 
            
  | 17:37 | <mark> | Converted srv214-218 and srv86-87 into API servers | [production] | 
            
  | 17:02 | <mark> | Throwing all API requests at the new API app server cluster | [production] | 
            
  | 16:49 | <mark> | Converted srv187-srv189 and srv250-srv253 into API app servers | [production] | 
            
  | 15:46 | <mark> | Created new Ganglia group "API application servers", deploying with Puppet | [production] | 
            
  | 15:31 | <mark> | Created API app servers group in Puppet, deploying on srv254-257 to start with | [production] | 
            
  | 15:24 | <mark> | Created API LVS service on lvs3 and restarted PyBal | [production] | 
            
  | 15:10 | <mark> | Added DNS entry for api.svc.pmpta.wmnet (LVS ip) | [production] | 
            
  | 14:18 | <mark> | Enabled more verbose logging on all squids: debug message level 1 for all message categories | [production] | 
            
  | 14:18 | <mark> | Disabled cache digests on all squids | [production] | 
            
  | 13:58 | <mark> | Removing sq30 and lower from the Text CARP pool | [production] | 
            
  | 13:51 | <mark> | Reducing persistent_connection_timeout and request_timeout from 1 min to 20s on all frontend squids, to reduce FD usage | [production] | 
            
  | 12:50 | <mark> | Restarting knsq* frontend squids with FD limit raised from 32k to 64k | [production] | 
            
  | 12:23 | <mark> | Grown filesystem /dev/data/ES (/a) on ms3 by 90G, leaving another 98G free | [production] | 
            
  | 00:17 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php  'Sync CSS style versions' | [production] | 
            
  | 00:16 | <catrope> | synchronized php-1.5/wmf-config/CommonSettings.php  'Bump style version appendix' | [production] | 
            
  | 00:10 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js  'r62403' | [production] | 
            
  | 00:02 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js  'r62400' | [production] | 
            
  | 00:01 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js  'r62400' | [production] | 
            
  
    | 2010-02-12
      
      § | 
    
  | 23:55 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js  'Try deploying r62397 again' | [production] | 
            
  | 23:46 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative/css/combined.min.css  'r62397' | [production] | 
            
  | 23:46 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js  'r62397' | [production] | 
            
  | 23:22 | <mark> | Deployed sq86 | [production] | 
            
  | 22:31 | <catrope> | synchronized php-1.5/wmf-config/CommonSettings.php  'Bump style version appendix' | [production] | 
            
  | 22:30 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js  'r62388' | [production] | 
            
  | 21:12 | <mark> | Implemented new routing policy for AS43821 | [production] | 
            
  | 21:01 | <catrope> | synchronized php-1.5/wmf-config/CommonSettings.php  'Bump style version appendix' | [production] | 
            
  | 20:59 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js  'r62382' | [production] | 
            
  | 19:15 | <Rob> | sq67 has both disks installed now (had to get one replaced) and has no OS installation yet.  holding off on install, not sure if mark wants this up as a squid or for varnish development | [production] |