| 2009-05-01
      
      § | 
    
  | 13:14 | <mark> | Installed server sockpuppet | [production] | 
            
  | 12:59 | <andrew> | synchronized php-1.5/includes/ChangeTags.php  'Live-merged r50104 -- escaping for classes applied to change tags' | [production] | 
            
  | 05:47 | <tstarling> | synchronized php-1.5/includes/UserMailer.php  'merged r49682' | [production] | 
            
  | 05:36 | <andrew> | synchronized php-1.5/InitialiseSettings.php  'Re-activated tag filtering with the live patch to ChangeTags.php' | [production] | 
            
  | 05:32 | <andrew> | synchronized php-1.5/includes/ChangeTags.php  'Index has been renamed since it was created on Wikimedia' | [production] | 
            
  | 05:13 | <tstarling> | synchronized php-1.5/InitialiseSettings.php | [production] | 
            
  | 05:10 | <Tim> | $wgUseTagFilter=true experimentally | [production] | 
            
  | 05:10 | <tstarling> | synchronized php-1.5/InitialiseSettings.php | [production] | 
            
  | 04:54 | <tstarling> | synchronized php-1.5/includes/ChangeTags.php | [production] | 
            
  | 04:53 | <tstarling> | synchronized php-1.5/includes/ChangeTags.php | [production] | 
            
  | 04:50 | <Tim> | merging r49068 and r49086 | [production] | 
            
  
    | 2009-04-30
      
      § | 
    
  | 17:10 | <Rob> | srv143 back online | [production] | 
            
  | 17:07 | <Rob> | all memcached back online | [production] | 
            
  | 17:07 | <robh> | synchronized php-1.5/mc-pmtpa.php  'swapped out srv142' | [production] | 
            
  | 17:06 | <Rob> | srv143 locked up, restarting | [production] | 
            
  | 17:05 | <Rob> | srv142 reinstalling | [production] | 
            
  | 16:52 | <Rob> | srv31 setup and good to go back to tomasz | [production] | 
            
  | 16:48 | <Rob> | srv31 reinstalled, installing wikimedia-task-appserver package but NOT pooling. | [production] | 
            
  | 16:40 | <Rob> | srv81 back online | [production] | 
            
  | 16:25 | <Rob> | upgrading srv31 to ubuntu | [production] | 
            
  | 16:10 | <Rob> | reinstalling srv81 | [production] | 
            
  | 16:08 | <Rob> | srv130 back online | [production] | 
            
  | 15:57 | <domas> | db30 has drive failure, needs replacement | [production] | 
            
  | 15:41 | <Rob> | upgrading srv124 to ubuntu | [production] | 
            
  | 15:30 | <Rob> | srv127 was readonly, restarted, fsck, back online | [production] | 
            
  | 15:25 | <Rob> | upgrading srv137 to ubuntu | [production] | 
            
  | 13:29 | <river> | upgraded ms4/ms6 to solaris 10 update 7 | [production] | 
            
  | 02:34 | <Tim> | reset slave on db3 | [production] | 
            
  | 02:28 | <Tim> | updated /root/.ssh/authorized_keys on all machines identified with a pingscan that allowed a login with nagios's key. Revoked access for nagios, jeronim and kyle. | [production] | 
            
  
    | 2009-04-29
      
      § | 
    
  | 21:32 | <brion> | synchronized php-1.5/includes/specials/SpecialExport.php  'merging r50054 fix for recursive depth export' | [production] | 
            
  | 21:23 | <Rob> | ran namespaceDupes script against mtwiki once the new portal namespaces were created. | [production] | 
            
  | 21:22 | <robh> | synchronized php-1.5/InitialiseSettings.php  'Bug 18498, adding portal and portal talk namespaces' | [production] | 
            
  | 21:13 | <robh> | synchronized php-1.5/InitialiseSettings.php  'Bug 18498, adding metanamespace_talk for mtwiki' | [production] | 
            
  | 21:12 | <brion> | set up system administrators global group with export depth override right so Trevor can test the batch export | [production] | 
            
  | 20:49 | <robh> | synchronized php-1.5/InitialiseSettings.php  'Bug 18237 enable autopatrolling and improve patrolling user rights on itwiktionary' | [production] | 
            
  | 19:05 | <Rob> | DHCP services stopped on zwinger and started on khaldun.  Khaldun is now the dhcp server as well as the installation server. | [production] | 
            
  | 14:53 | <Rob> | restarted wikitech and manually ran morebots upon reboot. | [production] | 
            
  | 04:07 | <Tim> | doing some network scanning to make sure our host lists are up to date | [production] | 
            
  | 02:36 | <Tim> | removed all remaining obsolete by_ssh* checks from the nagios configuration | [production] | 
            
  | 02:27 | <Tim> | installed NRPE on amane and adjusted nagios configurator | [production] | 
            
  | 01:54 | <tomaszf> | testing commons upload of top level storage directory on zwinger to offsite backup. | [production] | 
            
  | 01:38 | <Tim> | fixed the mediawiki installation on amane: installed wikimedia-task-appserver, disabled apache, ran sync-common, added to ganglia | [production] | 
            
  
    | 2009-04-28
      
      § | 
    
  | 18:02 | <Rob> | futzing around with moving dhcp, taking srv209 as my guineapig. | [production] | 
            
  | 10:58 | <Tim> | re-added srv31 to mediawiki-installation node group, backup task was rogue and generating "missing cluster" exceptions | [production] | 
            
  | 10:21 | <tstarling> | synchronized php-1.5/includes/ExternalStoreDB.php | [production] | 
            
  | 10:19 | <Tim> | re-added srv57 to mediawiki-installation, was rogue and causing "unknown cluster" errors | [production] | 
            
  | 07:59 | <tstarling> | synchronized php-1.5/db.php  'set the new cluster22 to be the sole ES write destination' | [production] | 
            
  | 07:57 | <Tim> | pdns on bayle is broken, stuck in futex, restarting | [production] | 
            
  | 07:52 | <tstarling> | synchronized php-1.5/db.php | [production] | 
            
  | 07:49 | <tstarling> | synchronized php-1.5/db.php  'introducing cluster22 (ms3/ms2)' | [production] |