| 2010-02-04
      
      § | 
    
  | 22:20 | <tstarling> | synchronized php-1.5/includes/api/ApiUpload.php  'disabled chunk upload' | [production] | 
            
  | 22:09 | <catrope> | synchronized php-1.5/wmf-config/CommonSettings.php  'Bump UsabilityInitiative_alpha style version appendix' | [production] | 
            
  | 22:07 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.hooks.php | [production] | 
            
  | 22:06 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative_alpha/UsabilityInitiative.hooks.php | [production] | 
            
  | 22:06 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative_alpha/css/combined.min.css | [production] | 
            
  | 22:06 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.combined.min.js | [production] | 
            
  | 22:06 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins.combined.min.js | [production] | 
            
  | 22:05 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative_alpha/images/wikiEditor/dialogs/insert-link-external-rtl.png | [production] | 
            
  | 22:04 | <RoanKattouw> | Deploying r61993 with individual sync-files | [production] | 
            
  | 21:15 | <atglenn> | turned off hourly snaps on ms7, we'll have daily plus the standard replication | [production] | 
            
  | 18:49 | <atglenn> | cleared /tmp on srv175 blah blah | [production] | 
            
  | 18:36 | <atglenn> | cleared out php* files from /tmp on srv163 (need cron job) | [production] | 
            
  | 10:35 | <mark> | Removed sq11-15 from the upload LVS pool | [production] | 
            
  | 10:30 | <mark> | Removed sq11-15 from the upload CARP pool | [production] | 
            
  | 03:28 | <apergos> | added /usr/local/apache/common/php to include path of php on fenari | [production] | 
            
  
    | 2010-02-03
      
      § | 
    
  | 23:56 | <catrope> | synchronized php-1.5/wmf-config/CommonSettings.php  'Bump UsabilityInitiative_alpha style version appendix' | [production] | 
            
  | 23:55 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.hooks.php | [production] | 
            
  | 23:55 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins/jquery.wikiEditor.html | [production] | 
            
  | 23:55 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins.combined.min.js | [production] | 
            
  | 23:55 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.combined.min.js | [production] | 
            
  | 23:55 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative_alpha/css/combined.min.css | [production] | 
            
  | 23:54 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative_alpha/UsabilityInitiative.hooks.php | [production] | 
            
  | 23:54 | <RoanKattouw> | Deploying r61959 using individual sync-files | [production] | 
            
  | 22:36 | <catrope> | synchronized php-1.5/wmf-config/CommonSettings.php  'Explicitly enable EditWarning by default here. Previously done in extension file' | [production] | 
            
  | 17:32 | <mark> | Set root password on all servers using Puppet | [production] | 
            
  | 17:26 | <Rob> | loaded sq68 full with 6 ssd, had to 'borrow' drives from the unracked squids in pmtpa awaiting the rail issue resolution. | [production] | 
            
  | 17:26 | <Rob> | connected sq67 and sq68 to secondary network ports as requested | [production] | 
            
  | 16:43 | <Rob> | had to update gmetad and add sq14 and sq17 into puppet as ganglia aggregator hosts.  Currently not showing in ganglia, but should when puppet updates their files shortly. | [production] | 
            
  | 16:37 | <Rob> | updated gmetad_pmtpa in NFS store, as well as on zwinger and spence. | [production] | 
            
  | 16:20 | <Rob> | !log cleaned sq1-sq10 certificates off sockpuppet | [production] | 
            
  | 16:10 | <mark> | Rebooting sockpuppet for security upgrades | [production] | 
            
  | 16:06 | <Rob> | updating DNS to remove lvs3.wikimedia.org, as its technically lvs3.pmtpa.wmnet | [production] | 
            
  | 16:03 | <Rob> | removed sq1-sq10 in dsh groups, server roles, pybal (already done, just doublechecked), and now starting wipe and removing their network connections. | [production] | 
            
  | 15:37 | <Rob> | updated video plugin on techblog, had to comment out sumotv support line due to errors. | [production] | 
            
  | 13:30 | <mark> | Removed sq1-10 from the upload backend Squid pool, preparing for decommissioning | [production] | 
            
  | 12:27 | <mark> | Removed eiximenis from the text backend pool | [production] | 
            
  | 12:21 | <mark> | Restarted sq51 with hyperthreading disabled | [production] | 
            
  | 11:31 | <Tim-away> | exim was flooding its mainlog with errors like "User 'exim' has exceeded the 'max_user_connections' resource (current value: 150)". OTRS is apparently broken as a result. Set the limit to infinity to fix it. | [production] | 
            
  | 11:10 | <Tim-away> | debugging mysql connection errors from mchenry to db9 | [production] | 
            
  | 11:02 | <Tim-away> | restarting exim4 on mchenry | [production] | 
            
  | 10:57 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative_alpha/Vector/Vector.hooks.php  'r61922' | [production] | 
            
  | 10:47 | <RoanKattouw> | Mail delivery reported broken again; last mail delivery to mediawiki-cvs was 3h22m ago | [production] | 
            
  | 10:14 | <catrope> | synchronized php-1.5/wmf-config/CommonSettings.php  'Bump UsabilityInitiative_alpha style version' | [production] | 
            
  | 10:13 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins/jquery.wikiEditor.html | [production] | 
            
  | 10:13 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative_alpha/UsabilityInitiative.hooks.php | [production] | 
            
  | 10:12 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins.combined.min.js | [production] | 
            
  | 10:12 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative_alpha/Vector/Vector.hooks.php | [production] | 
            
  | 10:12 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative_alpha/Vector/Vector.combined.min.js | [production] | 
            
  | 10:12 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.hooks.php | [production] | 
            
  | 10:12 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.combined.min.js | [production] |