| 2009-03-30
      
      § | 
    
  | 23:50 | <domas> | interrupted crash recovery for db2, will do crime scene investigation afterwards :) | [production] | 
            
  | 23:37 | <midom> | synchronized php-1.5/CommonSettings.php  'enwiki rw with new master' | [production] | 
            
  | 23:36 | <midom> | synchronized php-1.5/db.php  'welcome new enwiki master, db12 - db12-bin.016 79' | [production] | 
            
  | 23:27 | <domas> | too many open transactions (?) on enwiki/db2 caused it to go OOM or so... | [production] | 
            
  | 23:25 | <midom> | synchronized php-1.5/CommonSettings.php | [production] | 
            
  | 23:23 | <river> | Out of Memory: Killed process 5374 (mysqld). | [production] | 
            
  | 23:21 | <Andrew> | mysqld on db2 crashed. | [production] | 
            
  | 15:58 | <aZaFred> | restarted Apache on srv126 and srv103 | [production] | 
            
  | 08:22 | <domas> | memcached on srv60 segfaulted.. | [production] | 
            
  | 05:55 | <andrew> | synchronized php-1.5/includes/specials/SpecialUserrights.php  'Live-merging r48993, fix for global group membership form (regression)' | [production] | 
            
  | 05:55 | <andrew> | synchronized php-1.5/extensions/CentralAuth/SpecialGlobalGroupMembership.php  'Live-merging r48993, fix for global group membership form (regression)' | [production] | 
            
  
    | 2009-03-29
      
      § | 
    
  | 14:20 | <midom> | synchronized php-1.5/includes/specials/SpecialWatchlist.php  'sync up watchlist fixes, r49002' | [production] | 
            
  | 11:20 | <midom> | synchronized php-1.5/includes/specials/SpecialRecentchanges.php  ''''temporary''' increasing internal RC limit to 5000' | [production] | 
            
  | 11:15 | <midom> | synchronized php-1.5/includes/specials/SpecialRecentchanges.php  'some more efficient joining' | [production] | 
            
  | 11:09 | <midom> | synchronized php-1.5/includes/ChangesList.php  'livemerging up to 48990' | [production] | 
            
  | 11:09 | <midom> | synchronized php-1.5/includes/specials/SpecialRecentchanges.php  'livemerging up to 48990' | [production] | 
            
  | 10:35 | <andrew> | synchronized php-1.5/CommonSettings.php  'Bug 18223, let sysops edit abuse filters on dewiki' | [production] | 
            
  | 08:52 | <midom> | synchronized php-1.5/CommonSettings.php  'move away the mainpage delete protection to getUserPermissionsErrorsExpensive' | [production] | 
            
  | 08:41 | <midom> | synchronized php-1.5/includes/Title.php  'merging in c48983' | [production] | 
            
  | 07:37 | <andrew> | synchronized php-1.5/InitialiseSettings.php  'Turning on AbuseFilter on enwikiquote' | [production] | 
            
  
    | 2009-03-28
      
      § | 
    
  | 00:56 | <Tim> | fixed permissions on .ssh directories on singer. Converted jforrester's authorized_keys file from RFC to OpenSSH format. | [production] | 
            
  | 00:08 | <andrew> | synchronized php-1.5/includes/specials/SpecialUserrights.php  'Syncing revert of apiuserrights r48909' | [production] | 
            
  | 00:07 | <andrew> | synchronized php-1.5/includes/User.php  'Syncing revert of apiuserrights r48909' | [production] | 
            
  | 00:06 | <andrew> | synchronized php-1.5/includes/AutoLoader.php  'Syncing revert of apiuserrights r48909' | [production] | 
            
  | 00:06 | <andrew> | synchronized php-1.5/includes/api/ApiQueryUsers.php  'Syncing revert of apiuserrights r48909' | [production] | 
            
  | 00:05 | <andrew> | synchronized php-1.5/includes/api/ApiQueryRecentChanges.php  'Syncing revert of apiuserrights r48909' | [production] | 
            
  | 00:04 | <andrew> | synchronized php-1.5/includes/api/ApiMain.php  'Syncing revert of apiuserrights r48909' | [production] | 
            
  | 00:03 | <andrew> | synchronized php-1.5/extensions/CentralAuth/SpecialGlobalGroupMembership.php  'Reverting apiuserrights (r48910)' | [production] | 
            
  
    | 2009-03-27
      
      § | 
    
  | 23:29 | <aZaFred> | rebooting srv217 since it is wedged. | [production] | 
            
  | 22:10 | <aZaFred> | restarted apache on srv203 | [production] | 
            
  | 21:39 | <brion> | synchronized php-1.5/InitialiseSettings.php  'disabling TIFF->JPEG thumbnailing, doesn't work at present with our setup' | [production] | 
            
  | 21:38 | <brion> | synchronized php-1.5/InitialiseSettings.php  'Enabling TIFF->JPEG thumbnailing experimentally' | [production] | 
            
  | 17:49 | <brion> | enabled upload-by-url for all users on testwiki for wider testing, upped $wgMaxUploadSize to 500MB from default 100 | [production] | 
            
  | 17:44 | <brion> | synchronized php-1.5/includes/specials/SpecialUpload.php | [production] | 
            
  | 17:43 | <brion> | synchronized php-1.5/includes/DefaultSettings.php | [production] | 
            
  | 17:43 | <brion> | live-merging r48923 to make CURL timeout for upload-by-url configurable | [production] | 
            
  | 17:27 | <brion> | enabled xml-rpc publishing on techblog so I can administer from WordPress iPhone app | [production] | 
            
  | 10:56 | <mark> | Moved server nehalem to vlan 101 | [production] | 
            
  | 02:28 | <Tim> | db29 was full of relay logs, ran RESET SLAVE. | [production] | 
            
  
    | 2009-03-26
      
      § | 
    
  | 22:45 | <brion> | synchronized php-1.5/includes/specials/SpecialBlockip.php  'to r48899 - fixes for hiding' | [production] | 
            
  | 22:42 | <Danny_B> | wikibugs-l stopped to send mails to wikibugs-irc mailbox due to excessive bounces. reenabling sending again | [production] | 
            
  | 22:05 | <midom> | synchronized php-1.5/db.php  'db1 coming back as frwiki/jawiki slave' | [production] | 
            
  | 21:49 | <brion> | synchronized php-1.5/extensions/ProofreadPage/ProofreadPage.php | [production] | 
            
  | 18:36 | <brion> | synchronized php-1.5/languages/LanguageConverter.php | [production] | 
            
  | 18:36 | <brion> | applying r48836 language converter fix live | [production] | 
            
  | 18:22 | <aZaFred> | srv224 and srv225 have been kickstarted, deployed and put in rotation. | [production] | 
            
  | 15:56 | <Rob> | srv224 and srv225 have temp power from A4 until new cables can be made. | [production] | 
            
  | 15:56 | <Rob> | morebots was dead! | [production] |