| 2008-11-06
      
      § | 
    
  | 21:03 | <Tim> | switched GIFs to use Bitmap_ClientOnly (client-side scaling) | [production] | 
            
  | 17:23 | <brion> | restarting apache on srv47, seem smysteriously stuck | [production] | 
            
  | 17:15 | <brion> | setting $wgMaxAnimatedGifArea to 1 to prevent animated thumbnailing of GIFs for now, see if that helps | [production] | 
            
  | 17:10 | <brion> | river complaining of image scaler issues -- load spikes, depooling? | [production] | 
            
  | 01:42 | <Tim> | restarting squids | [production] | 
            
  | 01:42 | <mark> | Setup lvs4 as temp LVS support for lvs2, balancing the load | [production] | 
            
  | 01:07 | <brion> | updated ContributionReporting to add paging links to ContributionHistory (might be a little funky w/ caching, we'll work it out :) | [production] | 
            
  | 00:45 | <Tim> | progressively clearing /a on the remaining image scalers | [production] | 
            
  | 00:37 | <Tim> | wiping /a on srv44 | [production] | 
            
  | 00:00 | <brion> | scapping for update to ContributionReporting | [production] | 
            
  
    | 2008-11-05
      
      § | 
    
  | 23:38 | <brion> | set yongle to restart apache every hour since it still seems to bork up and get stuck sometimes | [production] | 
            
  | 22:01 | <RobH> | srv100 rebooted, was down. | [production] | 
            
  | 18:28 | <mark> | tech team is procrastinating | [production] | 
            
  | 18:16 | <atglenn> | added dhelps to office@wikimedia.org alias, redirected office@wikipedia.org to him also | [production] | 
            
  | 18:14 | <brion> | disabling centralnotice on private wikis, we don't need to be told to donate to ourselves ;) | [production] | 
            
  | 18:03 | <brion> | poking sitenotices off wikibooks, on *.wikipedia | [production] | 
            
  | 18:03 | <brion> | set up ariel on mchenry for mail admin | [production] | 
            
  | 05:38 | <brion_> | opera users may rejoice ;) | [production] | 
            
  | 05:38 | <brion_> | tweaked storage1 lighttpd config so centralnotice.js is served with utf-8 charset | [production] | 
            
  | 05:17 | <brion_> | for reference -- load spikes are page rendering on enwiki and dewiki mostly :) | [production] | 
            
  | 05:17 | <brion_> | bumping enwiki notice to 100% | [production] | 
            
  | 05:07 | <Tim> | killed various mysqld_safe processes which were using 100% CPU on ES servers | [production] | 
            
  | 04:50 | <brion_> | fixed morebots -- bots now allowed to edit again at wikitech | [production] | 
            
  | 04:50 | <brion_> | enabling enwiki notice at about 10% sampling | [production] | 
            
  | 03:27 | <brion_> | squids are... i think.... looking better :D | [production] | 
            
  | 02:47 | <brion_> | seems happier after restart of front-end squids | [production] | 
            
  | 02:43 | <brion_> | tim's doing hard restarts of more squids, we're kinda offline briefly | [production] | 
            
  | 02:34 | <brion_> | disabling centralnotices on remaining sites just for good measure while we debug | [production] | 
            
  | 02:29 | <brion_> | current status: the squids which borked are still kind of borked, but perhaps slightly better. mark is examining squid memory reports | [production] |