| 
      
        2009-03-30
      
      §
     | 
  
    
  | 23:50 | 
  <domas> | 
  interrupted crash recovery for db2, will do crime scene investigation afterwards :) | 
  [production] | 
            
  | 23:37 | 
  <midom> | 
  synchronized php-1.5/CommonSettings.php  'enwiki rw with new master' | 
  [production] | 
            
  | 23:36 | 
  <midom> | 
  synchronized php-1.5/db.php  'welcome new enwiki master, db12 - db12-bin.016 79' | 
  [production] | 
            
  | 23:27 | 
  <domas> | 
  too many open transactions (?) on enwiki/db2 caused it to go OOM or so...  | 
  [production] | 
            
  | 23:25 | 
  <midom> | 
  synchronized php-1.5/CommonSettings.php  | 
  [production] | 
            
  | 23:23 | 
  <river> | 
  Out of Memory: Killed process 5374 (mysqld). | 
  [production] | 
            
  | 23:21 | 
  <Andrew> | 
  mysqld on db2 crashed. | 
  [production] | 
            
  | 15:58 | 
  <aZaFred> | 
  restarted Apache on srv126 and srv103 | 
  [production] | 
            
  | 08:22 | 
  <domas> | 
  memcached on srv60 segfaulted.. | 
  [production] | 
            
  | 05:55 | 
  <andrew> | 
  synchronized php-1.5/includes/specials/SpecialUserrights.php  'Live-merging r48993, fix for global group membership form (regression)' | 
  [production] | 
            
  | 05:55 | 
  <andrew> | 
  synchronized php-1.5/extensions/CentralAuth/SpecialGlobalGroupMembership.php  'Live-merging r48993, fix for global group membership form (regression)' | 
  [production] | 
            
  
    | 
      
        2009-03-29
      
      §
     | 
  
    
  | 14:20 | 
  <midom> | 
  synchronized php-1.5/includes/specials/SpecialWatchlist.php  'sync up watchlist fixes, r49002' | 
  [production] | 
            
  | 11:20 | 
  <midom> | 
  synchronized php-1.5/includes/specials/SpecialRecentchanges.php  ''''temporary''' increasing internal RC limit to 5000' | 
  [production] | 
            
  | 11:15 | 
  <midom> | 
  synchronized php-1.5/includes/specials/SpecialRecentchanges.php  'some more efficient joining' | 
  [production] | 
            
  | 11:09 | 
  <midom> | 
  synchronized php-1.5/includes/ChangesList.php  'livemerging up to 48990' | 
  [production] | 
            
  | 11:09 | 
  <midom> | 
  synchronized php-1.5/includes/specials/SpecialRecentchanges.php  'livemerging up to 48990' | 
  [production] | 
            
  | 10:35 | 
  <andrew> | 
  synchronized php-1.5/CommonSettings.php  'Bug 18223, let sysops edit abuse filters on dewiki' | 
  [production] | 
            
  | 08:52 | 
  <midom> | 
  synchronized php-1.5/CommonSettings.php  'move away the mainpage delete protection to getUserPermissionsErrorsExpensive' | 
  [production] | 
            
  | 08:41 | 
  <midom> | 
  synchronized php-1.5/includes/Title.php  'merging in c48983' | 
  [production] | 
            
  | 07:37 | 
  <andrew> | 
  synchronized php-1.5/InitialiseSettings.php  'Turning on AbuseFilter on enwikiquote' | 
  [production] | 
            
  
    | 
      
        2009-03-28
      
      §
     | 
  
    
  | 00:56 | 
  <Tim> | 
  fixed permissions on .ssh directories on singer. Converted jforrester's authorized_keys file from RFC to OpenSSH format. | 
  [production] | 
            
  | 00:08 | 
  <andrew> | 
  synchronized php-1.5/includes/specials/SpecialUserrights.php  'Syncing revert of apiuserrights r48909' | 
  [production] | 
            
  | 00:07 | 
  <andrew> | 
  synchronized php-1.5/includes/User.php  'Syncing revert of apiuserrights r48909' | 
  [production] | 
            
  | 00:06 | 
  <andrew> | 
  synchronized php-1.5/includes/AutoLoader.php  'Syncing revert of apiuserrights r48909' | 
  [production] | 
            
  | 00:06 | 
  <andrew> | 
  synchronized php-1.5/includes/api/ApiQueryUsers.php  'Syncing revert of apiuserrights r48909' | 
  [production] | 
            
  | 00:05 | 
  <andrew> | 
  synchronized php-1.5/includes/api/ApiQueryRecentChanges.php  'Syncing revert of apiuserrights r48909' | 
  [production] | 
            
  | 00:04 | 
  <andrew> | 
  synchronized php-1.5/includes/api/ApiMain.php  'Syncing revert of apiuserrights r48909' | 
  [production] | 
            
  | 00:03 | 
  <andrew> | 
  synchronized php-1.5/extensions/CentralAuth/SpecialGlobalGroupMembership.php  'Reverting apiuserrights (r48910)' | 
  [production] | 
            
  
    | 
      
        2009-03-27
      
      §
     | 
  
    
  | 23:29 | 
  <aZaFred> | 
  rebooting srv217 since it is wedged.  | 
  [production] | 
            
  | 22:10 | 
  <aZaFred> | 
  restarted apache on srv203 | 
  [production] | 
            
  | 21:39 | 
  <brion> | 
  synchronized php-1.5/InitialiseSettings.php  'disabling TIFF->JPEG thumbnailing, doesn't work at present with our setup' | 
  [production] | 
            
  | 21:38 | 
  <brion> | 
  synchronized php-1.5/InitialiseSettings.php  'Enabling TIFF->JPEG thumbnailing experimentally' | 
  [production] | 
            
  | 17:49 | 
  <brion> | 
  enabled upload-by-url for all users on testwiki for wider testing, upped $wgMaxUploadSize to 500MB from default 100 | 
  [production] | 
            
  | 17:44 | 
  <brion> | 
  synchronized php-1.5/includes/specials/SpecialUpload.php  | 
  [production] | 
            
  | 17:43 | 
  <brion> | 
  synchronized php-1.5/includes/DefaultSettings.php  | 
  [production] | 
            
  | 17:43 | 
  <brion> | 
  live-merging r48923 to make CURL timeout for upload-by-url configurable | 
  [production] | 
            
  | 17:27 | 
  <brion> | 
  enabled xml-rpc publishing on techblog so I can administer from WordPress iPhone app | 
  [production] | 
            
  | 10:56 | 
  <mark> | 
  Moved server nehalem to vlan 101 | 
  [production] | 
            
  | 02:28 | 
  <Tim> | 
  db29 was full of relay logs, ran RESET SLAVE. | 
  [production] | 
            
  
    | 
      
        2009-03-26
      
      §
     | 
  
    
  | 22:45 | 
  <brion> | 
  synchronized php-1.5/includes/specials/SpecialBlockip.php  'to r48899 - fixes for hiding' | 
  [production] | 
            
  | 22:42 | 
  <Danny_B> | 
  wikibugs-l stopped to send mails to wikibugs-irc mailbox due to excessive bounces. reenabling sending again | 
  [production] | 
            
  | 22:05 | 
  <midom> | 
  synchronized php-1.5/db.php  'db1 coming back as frwiki/jawiki slave' | 
  [production] | 
            
  | 21:49 | 
  <brion> | 
  synchronized php-1.5/extensions/ProofreadPage/ProofreadPage.php  | 
  [production] | 
            
  | 18:36 | 
  <brion> | 
  synchronized php-1.5/languages/LanguageConverter.php  | 
  [production] | 
            
  | 18:36 | 
  <brion> | 
  applying r48836 language converter fix live | 
  [production] | 
            
  | 18:22 | 
  <aZaFred> | 
  srv224 and srv225 have been kickstarted, deployed and put in rotation. | 
  [production] | 
            
  | 15:56 | 
  <Rob> | 
  srv224 and srv225 have temp power from A4 until new cables can be made. | 
  [production] | 
            
  | 15:56 | 
  <Rob> | 
  morebots was dead! | 
  [production] |