| 2010-02-10
      
      § | 
    
  | 21:02 | <Andrew> | Switching strategywiki to opt-out LiquidThreads, rather than opt-in | [production] | 
            
  | 20:46 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js  'Deploy r62262' | [production] | 
            
  | 20:46 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js  'Deploy r62262' | [production] | 
            
  | 19:22 | <catrope> | synchronized php-1.5/wmf-config/CommonSettings.php  'Bump style version appendix' | [production] | 
            
  | 19:21 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js  'Deploying r62257' | [production] | 
            
  | 19:05 | <Rob> | storage1 disks replaced, except for the newly dead one, rma placed. | [production] | 
            
  | 18:39 | <Rob> | shutting down storage1 to replace its bad OS mirror disk | [production] | 
            
  | 16:52 | <mark> | Fixed MySQL permissions on srv186 as well | [production] | 
            
  | 16:15 | <mark> | Rigged puppet to deploy a PHP mail.ini file to the apaches, which sets it to call sendmail with -f <> (sending mail with empty envelope sender, like bounces) | [production] | 
            
  
    | 2010-02-09
      
      § | 
    
  | 22:06 | <Andrew> | LiquidThreads production software updates completed successfully. | [production] | 
            
  | 21:58 | <andrew> | synchronized php-1.5/extensions/LiquidThreads/classes/Dispatch.php | [production] | 
            
  | 21:57 | <mark> | Firewalled one ip on singer which was overloading Apache, looked like a Wordpress attack | [production] | 
            
  | 21:50 | <mark> | Lowered MaxClients setting to 250 on singer | [production] | 
            
  | 21:47 | <Andrew> | Running scap to deploy LiquidThreads updates | [production] | 
            
  | 21:45 | <Andrew> | Updating LiquidThreads production deployments to the alpha version in the next few minutes. | [production] | 
            
  | 21:44 | <mark> | Power cycled singer | [production] | 
            
  | 21:19 | <mark> | Fixed MySQL system user & permissions on srv151-srv185 | [production] | 
            
  | 21:17 | <mark> | Fixed MySQL instance on srv183 | [production] | 
            
  | 20:47 | <mark> | Fixed MySQL instance on srv156 | [production] | 
            
  | 19:39 | <catrope> | synchronized php-1.5/wmf-config/CommonSettings.php  'Bump style version appendix' | [production] | 
            
  | 19:29 | <catrope> | synchronized php-1.5/skins/common/edit.js  'Deploy r62190. Only for test for now, thank God for style versions' | [production] | 
            
  | 17:01 | <catrope> | synchronized php-1.5/wmf-config/CommonSettings.php  'Bump style version appendix' | [production] | 
            
  | 17:01 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js  'Deploying r62187' | [production] | 
            
  | 16:52 | <catrope> | synchronized php-1.5/wmf-config/CommonSettings.php  'Bump style version appendix' | [production] | 
            
  | 16:51 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js  'Deploying r62184' | [production] | 
            
  | 14:19 | <catrope> | synchronized php-1.5/wmf-config/CommonSettings.php  'Bump style version appendix' | [production] | 
            
  | 14:19 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative/js/plugins.combined.min.js  'Deploying r62181' | [production] | 
            
  | 02:14 | <andrew> | synchronized php-1.5/extensions/LiquidThreads_alpha/lqt.js  'Merge r62158' | [production] | 
            
  
    | 2010-02-08
      
      § | 
    
  | 23:25 | <catrope> | synchronized php-1.5/wmf-config/CommonSettings.php  'Bump style version appendix for r62041. Varnish needs this or Tampa will serve stale JS off bits' | [production] | 
            
  | 23:06 | <mark> | Deploying Exim on all application servers as well | [production] | 
            
  | 22:06 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative/OptIn/SpecialOptIn.php  'Deploy r62141' | [production] | 
            
  | 21:51 | <mark> | Added search3 and search9 to search_pool_1 on lvs3 | [production] | 
            
  | 21:48 | <mark> | Deploying Exim as a simple mail sending-only MTA (with queuing enabled) on squids, image scalers, search servers and core databases | [production] | 
            
  | 21:38 | <Andrew> | LiquidThreads wikinews rollout complete, though there was some buggage related to replication that had to be resolved. | [production] | 
            
  | 21:29 | <andrew> | synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Dispatch.php | [production] | 
            
  | 21:29 | <andrew> | synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Dispatch.php | [production] | 
            
  | 21:27 | <andrew> | synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Dispatch.php | [production] | 
            
  | 21:24 | <andrew> | synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Dispatch.php | [production] | 
            
  | 21:22 | <andrew> | synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Dispatch.php | [production] | 
            
  | 21:21 | <andrew> | synchronized php-1.5/extensions/LiquidThreads_alpha/classes/Dispatch.php | [production] | 
            
  | 21:18 | <andrew> | synchronized php-1.5/wmf-config/InitialiseSettings.php | [production] | 
            
  | 21:18 | <andrew> | synchronized php-1.5/wmf-config/liquidthreads.php | [production] | 
            
  | 21:13 | <Andrew> | Starting LiquidThreads deployment on enwikinews. | [production] | 
            
  | 20:00 | <catrope> | synchronized php-1.5/wmf-config/CommonSettings.php  'Track NTOC and dialogs as well in PrefStats' | [production] | 
            
  | 13:04 | <Tim> | running fixBug20757.php on enwiki, in a screen on hume | [production] | 
            
  | 12:51 | <Tim> | locke went into swap half an hour ago and is now dead. Trying reboot. | [production] | 
            
  | 07:38 | <Tim> | stopped recompressTracked.php for now due to bug 20757 | [production] | 
            
  | 07:04 | <tstarling> | synchronized php-1.5/includes/Revision.php  'r62120' | [production] | 
            
  | 06:18 | <Tim> | cleared out db9 root partition (which had 370MB free) by moving 4GB of SQL files from /root and /home/tfinc to /a/backup/junk | [production] | 
            
  | 06:01 | <tstarling> | synchronized php-1.5/wmf-config/db.php  're-added ms2 as an ES slave for rc1,cluster22 and the ex-fedora clusters, was depooled for ~6 months' | [production] |