| 2010-02-02
      
      § | 
    
  | 23:04 | <Tim> | pmtpa text squids down | [production] | 
            
  | 22:33 | <mark> | Pooled sq59-68 in LVS (Text) | [production] | 
            
  | 22:27 | <apergos> | restarted apache on srv170 (corrupted apc cache) | [production] | 
            
  | 22:25 | <mark> | Increased CARP weight of the new Upload squids sq51-58 from 10 to 30 | [production] | 
            
  | 22:17 | <mark> | Added sq59-66 to the Text pool of Squids, with low CARP weight (10) to seed the caches | [production] | 
            
  | 21:44 | <andrew> | synchronized php-1.5/wmf-config/liquidthreads.php | [production] | 
            
  | 21:42 | <Andrew> | temporarily disabling $wgLqtEnotif to mitigate mail server DoS impact on LiquidThreads post/reply speed. | [production] | 
            
  | 21:42 | <andrew> | synchronized php-1.5/extensions/LiquidThreads_alpha/classes/NewMessagesController.php  'Deploy r61879' | [production] | 
            
  | 21:42 | <andrew> | synchronized php-1.5/extensions/LiquidThreads/classes/NewMessagesController.php  'Deploy r61879' | [production] | 
            
  | 21:04 | <mark> | Removed sq2-10 from the LVS pool | [production] | 
            
  | 20:46 | <catrope> | synchronized php-1.5/wmf-config/CommonSettings.php  'Bump style version addition for usability alpha' | [production] | 
            
  | 20:45 | <catrope> | synchronized php-1.5/skins/common/jquery.min.js  'Deploing r61700' | [production] | 
            
  | 20:38 | <mark> | Added sq52-58 to the backend Squid pool with low CARP weight, to seed the caches | [production] | 
            
  | 20:37 | <catrope> | synchronized php-1.5/wmf-config/InitialiseSettings.php  'bug 22351: Enable Vector and toolbar on srwikinews' | [production] | 
            
  | 20:36 | <mark> | Increased Exim load average threshold for delivery on mchenry | [production] | 
            
  | 19:09 | <Fred> | rebooted singer (went down for unknown- at this time- reasons). | [production] | 
            
  | 18:25 | <Fred> | OS-installed sq52-ssq58 | [production] | 
            
  | 18:14 | <mark> | OS-installed sq59-sq66 | [production] | 
            
  | 17:23 | <mark> | Thrown sq66 backend squid in the Squid pool, CARP weight 30 | [production] | 
            
  | 06:01 | <Tim> | deploying r61846 via scap | [production] | 
            
  | 04:25 | <Andrew> | Schema change successfully applied. | [production] | 
            
  | 04:24 | <Andrew> | Applying schema change thread_signature.sql to mediawikiwiki, strategywiki, testwiki | [production] | 
            
  | 04:22 | <Andrew> | Applying schema change thread_signature.sql to liquidthreads_labswikimedia | [production] | 
            
  | 01:31 | <catrope> | synchronized php-1.5/wmf-config/CommonSettings.php  'Fix for UsabilityInitiative_alpha config' | [production] | 
            
  | 01:18 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.combined.min.js  'r61842' | [production] | 
            
  | 01:13 | <catrope> | synchronized php-1.5/wmf-config/CommonSettings.php  'Append to $wgStyleVersion when running usability alpha' | [production] | 
            
  | 01:10 | <catrope> | synchronized php-1.5/includes/OutputPage.php  'r61841 for real this time' | [production] | 
            
  | 01:09 | <catrope> | ran sync-common-all | [production] | 
            
  | 01:07 | <catrope> | synchronized php-1.5/includes/OutputPage.php  'r61841' | [production] | 
            
  | 01:05 | <catrope> | synchronized php-1.5/includes/OutputPage.php  'r61839' | [production] | 
            
  | 00:51 | <RoanKattouw> | Running scap | [production] | 
            
  | 00:48 | <catrope> | synchronized php-1.5/includes/OutputPage.php  'r61837' | [production] | 
            
  | 00:40 | <RoanKattouw> | Synced extensions/UsabilityInitiative_alpha by hand | [production] | 
            
  | 00:07 | <catrope> | synchronized php-1.5/skins/common/jquery.min.js | [production] | 
            
  | 00:07 | <RoanKattouw> | Syncing some CSS and JS files for usability deployment so bits.wikimedia.org will pick them up | [production] | 
            
  | 00:06 | <catrope> | synchronized php-1.5/skins/common/wikibits.js  'Sync changes to wikibits.js in r61558' | [production] | 
            
  
    | 2010-02-01
      
      § | 
    
  | 23:30 | <RoanKattouw> | Testing new usability code on test | [production] | 
            
  | 23:27 | <mark> | Pooled sq51 frontend squid in the upload LVS cluster, weight 30 | [production] | 
            
  | 23:03 | <mark> | Thrown sq51 backend squid in the Squid pool, CARP weight 30 | [production] | 
            
  | 20:50 | <Rob> | srv217 shutdown until checked | [production] | 
            
  | 20:50 | <Rob> | srv217 not responsive to network, but LOM works and system is jut not working online, possible bad cable, will check in DC tomorrow | [production] | 
            
  | 20:43 | <mark> | synchronized php-1.5/wmf-config/CommonSettings.php  'Added 36 new Squid servers, removed bart' | [production] | 
            
  | 20:39 | <mark> | Deployed new wikimedia-base package that no longer manages sysctl.conf post Hardy, letting puppet handle it | [production] | 
            
  | 20:10 | <atglenn> | restarted webserver on ms7 to read new conf (fix bug 22321) | [production] | 
            
  | 20:09 | <mark> | Granted cary access to the OTRS db from fenari | [production] | 
            
  | 19:27 | <RoanKattouw> | srv9 reported down, pinging yields Host Unreachable. srv9 is not in Nagios but seems to be in use | [production] | 
            
  | 18:51 | <Fred> | swapped memcache node from srv145 to srv196 | [production] |