| 2009-12-02
      
      § | 
    
  | 15:02 | <mark> | Shutting down mint for installation of a wifi card | [production] | 
            
  | 14:35 | <Andrew> | Scapping to update LiquidThreads alpha | [production] | 
            
  | 14:34 | <Andrew> | Updates of LiquidThreads alpha to trunk state in progress | [production] | 
            
  | 13:46 | <RoanKattouw> | Remove all LU cache files and rebuild from scratch to mitigate problems with root-owned cache files | [production] | 
            
  | 13:41 | <RoanKattouw> | Purging LocalisationUpdate hashes and running the update script to ensure we didn't miss anything in the Nov 19 - Dec 1 blackout | [production] | 
            
  | 05:28 | <Tim> | edited authz on mediawiki svn to deny write access to root | [production] | 
            
  | 01:17 | <tfinc> | synchronized php-1.5/wmf-config/CommonSettings.php  'new banner for cn' | [production] | 
            
  | 01:04 | <Tim> | restarted apache on srv211 and srv127 | [production] | 
            
  
    | 2009-11-30
      
      § | 
    
  | 22:54 | <Fred> | rebooting pascal since it once again went down. | [production] | 
            
  | 21:21 | <RoanKattouw> | Fred moved l10nupdate cronjob on hume to /etc/cron.d/l10nupdate , now runs as brion instead of root | [production] | 
            
  | 21:21 | <RoanKattouw> | Unlocked /h/w/l10n/trunk/extensions , had been locked since Nov 19th blocking L10nUpdates to extensions | [production] | 
            
  | 20:06 | <rainman-sr> | set search limit to 51 per Roan | [production] | 
            
  | 20:06 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative/EditToolbar/EditToolbar.php  'Raise style version for r59598, r59599' | [production] | 
            
  | 20:05 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative/EditToolbar/EditToolbar.js  'r59599 for real this time' | [production] | 
            
  | 20:04 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative/EditToolbar/EditToolbar.js  'r59599' | [production] | 
            
  | 20:00 | <catrope> | synchronized php-1.5/extensions/UsabilityInitiative/EditToolbar/EditToolbar.js  'r59598' | [production] | 
            
  | 19:55 | <rainman-sr> | converting searchidx1 back to indexer, things seem to be back to normal now | [production] | 
            
  | 19:46 | <atglenn> | enabled ExternalPages on test.wikipedia (fundraising extension) after adding it by hand to extension-list and ExtensionMessages.php | [production] | 
            
  | 19:30 | <rainman-sr> | doing a delayed restart of all searchers | [production] | 
            
  | 19:10 | <mark> | Setup puppet to enforce desired JVM alternatives on the search servers | [production] | 
            
  | 17:00 | <rainman-sr> | all search servers (except search11) should be switched back from 64 bit java (java-6-openjdk) to 32 bit java (ia32-java-6-sun) | [production] | 
            
  | 16:33 | <rainman-sr> | temporarely stop indexing and put searchidx1 in en.wp rotation, will back out when peak times are over | [production] | 
            
  | 15:28 | <mark> | Puppet removed syslog-ng on nfs1/2 as well. Restored. | [production] | 
            
  | 14:49 | <mark> | Replacing stock syslogd with rsyslog globally across the cluster (with puppet). rsyslog is the default in Ubuntu 9.10 onwards | [production] | 
            
  | 10:35 | <mark> | Using streber for karmic install testing | [production] | 
            
  | 10:35 | <mark> | Using brewster for karmic install testing | [production] | 
            
  | 10:19 | <mark> | Created karmic-wikimedia APT repository | [production] | 
            
  | 02:27 | <rainman-sr> | added search11 to en.wp search rotation.. we're getting peak-level traffic on 2am on sunday?! | [production] | 
            
  
    | 2009-11-29
      
      § | 
    
  | 21:32 | <mark> | Unmounted /home on ms5 | [production] | 
            
  | 20:58 | <mark> | Disabled NIS on ms5 | [production] | 
            
  | 20:54 | <mark> | Carefully starting puppet management of ms5 (Solaris)... just ssh host key exchanging for now | [production] | 
            
  | 20:42 | <mark> | Installed Blastwave puppet package (and deps) on ms5 | [production] | 
            
  | 20:39 | <mark> | Installed Blastwave pkgutil on ms5 | [production] | 
            
  | 20:37 | <mark> | Uninstalled coolstack CSKruntime and CSKruby packages on ms5 | [production] | 
            
  | 20:16 | <mark> | Installed coolstack CSKruntime and CSKruby packages on ms5 | [production] | 
            
  | 19:32 | <midom> | synchronized php-1.5/wmf-config/db.php | [production] | 
            
  | 19:25 | <domas> | s3 replication switched from db11-bin.215:36889084 to db17-bin.001:79 | [production] | 
            
  | 19:22 | <midom> | synchronized php-1.5/wmf-config/db.php | [production] | 
            
  | 19:22 | <midom> | synchronized php-1.5/wmf-config/db.php | [production] | 
            
  | 17:40 | <mark> | Fixed puppet config for searchidx1 | [production] | 
            
  | 14:32 | <midom> | synchronized php-1.5/wmf-config/db.php  'adding db1 and db7 to s6' | [production] | 
            
  | 14:16 | <catrope> | synchronized php-1.5/wmf-config/CommonSettings.php  'Per Domas, do bug 21510 a little bit more performance-friendly' | [production] | 
            
  | 14:12 | <catrope> | synchronized php-1.5/wmf-config/CommonSettings.php  'bug 21510: Second attempt' | [production] | 
            
  | 14:04 | <catrope> | synchronized php-1.5/wmf-config/CommonSettings.php  'backing out last change, breaks' | [production] | 
            
  | 14:03 | <catrope> | synchronized php-1.5/wmf-config/CommonSettings.php  'bug 21510: Temp raise acct create throttle on eswiki for UEM IP for wiki week' | [production] |