| 2009-03-25
      
      § | 
    
  | 14:08 | <andrew> | synchronized php-1.5/extensions/ProofreadPage/ProofreadPage.php | [production] | 
            
  | 14:01 | <andrew> | synchronized php-1.5/includes/api/ApiQueryImageInfo.php | [production] | 
            
  | 14:01 | <andrew> | synchronized php-1.5/extensions/ProofreadPage/ProofreadPage.php  'Yet another fatal' | [production] | 
            
  | 13:54 | <andrew> | synchronized php-1.5/extensions/ProofreadPage/ProofreadPage.php  'Fatal errors' | [production] | 
            
  | 13:49 | <andrew> | synchronized php-1.5/includes/api/ApiQueryCategories.php  'Fatal for invalid titles' | [production] | 
            
  | 13:49 | <andrew> | synchronized php-1.5/includes/specials/SpecialUserrights.php  'Fatal for cross-wiki user rights' | [production] | 
            
  | 13:47 | <andrew> | synchronized php-1.5/extensions/FlaggedRevs/FlaggedRevs.hooks.php  'Returning a value from a hook, causing exceptions.' | [production] | 
            
  | 13:41 | <andrew> | synchronized php-1.5/extensions/ProofreadPage/ProofreadPage.php  'Fixing a fatal' | [production] | 
            
  | 13:37 | <brion> | ProofreadPage is borked | [production] | 
            
  | 13:34 | <brion> | scap complete! | [production] | 
            
  | 13:23 | <brion> | starting general scap to r48811 -- yay! | [production] | 
            
  | 13:16 | <Andrew> | Reverted some live hacks for AbuseFilter that were in there because of dependencies on core. | [production] | 
            
  | 12:48 | <brion> | svn up'd test to r48811 ... last one? | [production] | 
            
  | 12:31 | <brion> | svn up'ing test to r48810 | [production] | 
            
  | 12:22 | <brion> | disabling Configure extension on testwiki for now; we'll poke at it more later | [production] | 
            
  | 12:22 | <brion> | synchronized php-1.5/InitialiseSettings.php | [production] | 
            
  | 12:20 | <brion> | applying DB schema tweaks for flaggedrevs, codereview | [production] | 
            
  | 12:16 | <andrew> | synchronized php-1.5/extensions/AbuseFilter/AbuseFilter.parser.php  'Trying to get rid of a warning' | [production] | 
            
  | 11:55 | <brion> | scap time cometh! svn up'ing testwiki for shakedown... | [production] | 
            
  | 10:26 | <brion> | svn up'ing CodeReview | [production] | 
            
  | 01:29 | <Tim> | running trackBlobs.php again on hume | [production] | 
            
  | 01:23 | <Tim> | db18 back up, replicating, 14285s lag | [production] | 
            
  | 01:18 | <tstarling> | synchronized php-1.5/db.php | [production] | 
            
  | 01:14 | <Tim> | rebooted ILOM on db18, was refusing to reboot the machine | [production] | 
            
  | 00:58 | <Tim> | db18 was locked up in kswapd, attempting reboot | [production] | 
            
  | 00:30 | <Tim> | installed NRPE etc. on ms2 and ms3 | [production] | 
            
  | 00:15 | <tstarling> | synchronized php-1.5/db.php | [production] | 
            
  | 00:11 | <tstarling> | synchronized php-1.5/db.php | [production] | 
            
  | 00:09 | <tstarling> | synchronized php-1.5/db.php | [production] | 
            
  | 00:04 | <Prodego> | domas is logging | [production] | 
            
  
    | 2009-03-24
      
      § | 
    
  | 23:57 | <Tim> | changing master for rc1 to ms3. Omitting srv183, which will be removed from the group. | [production] | 
            
  | 23:46 | <azafred_> | udpated motd on srv32 to reflect its puppeteer status. | [production] | 
            
  | 23:35 | <domas> | experimented with PG vs MySQL5.0 performance on db28 :) | [production] | 
            
  | 23:34 | <robh> | synchronized php-1.5/InitialiseSettings.php  'Ack, added to foundationwiki, not fishbowl, reverted, and added to fishbowl.' | [production] | 
            
  | 23:07 | <Rob> | lowered the memory limit on usability project server | [production] | 
            
  | 22:52 | <robh> | synchronized php-1.5/InitialiseSettings.php  'Adding an inactive group to foundationwiki' | [production] | 
            
  | 22:43 | <brion> | usability.wikimedia.org timing out | [production] | 
            
  | 22:23 | <azafred_> | fixed spamassassin rules compilation on mchenry to speed up the process | [production] | 
            
  | 22:22 | <azafred_> | Updated spamassassin rules to include more spam definitions on mchenry. | [production] | 
            
  | 21:36 | <brion> | updating usability.wikimedia.org to current, installing AntiSpoof, AbuseFilter to help clean up vandalism problems | [production] | 
            
  | 18:59 | <azafred_> | bounced apache on srv201 | [production] | 
            
  | 18:58 | <azafred_> | bounced apache on srv188 | [production] | 
            
  | 18:30 | <Rob> | pushing apache changes for wikizdroje.cz for cs.wikisource.org | [production] | 
            
  | 17:40 | <azafred_> | restarted apache on srv190 | [production] | 
            
  | 15:30 | <robh> | synchronized php-1.5/InitialiseSettings.php  'Bug 18023 Enable Collection extension on svwiki' | [production] | 
            
  | 15:22 | <robh> | synchronized php-1.5/InitialiseSettings.php  '//For bug 18061 Remove obsolete settings from cluster config files' | [production] | 
            
  | 10:45 | <midom> | synchronized php-1.5/includes/specials/SpecialRecentchanges.php  'RC cost became too high to support 5000 limit, decreased to 500' | [production] | 
            
  | 06:56 | <Tim> | stopped mysql on srv171 and srv183 for copy to ms2 and ms3. Depooled. | [production] | 
            
  | 06:55 | <tstarling> | synchronized php-1.5/db.php | [production] | 
            
  | 06:53 | <Tim> | installed ganglia on ms2 and ms3, put them in the mysql cluster | [production] |