| 2008-12-08
      
      § | 
    
  | 23:30 | <RobH> | updated enwiktionary group settings per https://bugzilla.wikimedia.org/show_bug.cgi?id=16248 | [production] | 
            
  | 23:24 | <brion> | updating Oversight for bug 16065 | [production] | 
            
  | 22:44 | <RobH> | no.wikimedia.org is now functioning per https://bugzilla.wikimedia.org/show_bug.cgi?id=15383 | [production] | 
            
  | 22:35 | <RobH> | made changes to InitialiseSettings.php for cswikisource per https://bugzilla.wikimedia.org/show_bug.cgi?id=16277 | [production] | 
            
  | 21:38 | <RobH> | authdns-update for no.wikimedia.org | [production] | 
            
  | 21:20 | <RobH> | running sync-common-all for wikimedia norge (found the php error) | [production] | 
            
  | 21:01 | <RobH> | its all back up now. | [production] | 
            
  | 20:59 | <RobH> | I stupidly crashed the site with a php typo, rolling back my changes since i was ignorant and did not php -l  ;_; | [production] | 
            
  | 20:58 | <RobH> | setup wikimedia norge wiki per https://bugzilla.wikimedia.org/show_bug.cgi?id=15383 | [production] | 
            
  | 19:23 | <brion> | updating OggHandler for fix for bug 15920 (chopped oggs) | [production] | 
            
  | 15:57 | <mark> | Set up mirroring of traffic of e7/2 to e7/14 for testing the fiber patch loop/optics | [production] | 
            
  
    | 2008-12-05
      
      § | 
    
  | 22:42 | <domas> | srv47 is running scaler usr.sbin.apache2 aa profile in learning mode | [production] | 
            
  | 22:33 | <RobH> | sq50 reinstalled and back in rotation | [production] | 
            
  | 22:26 | <RobH> | finished setup on srv146, back in apache pool | [production] | 
            
  | 21:32 | <RobH> | setting up packages on srv146 | [production] | 
            
  | 21:32 | <RobH> | reinstalling sq50 | [production] | 
            
  | 21:27 | <brion> | pointing SiteMatrix at local copy, not NFS master, of langlist file | [production] | 
            
  | 19:19 | <RobH> | added sq48, and sq49 back into pool.  sq50 pending reinstallation. | [production] | 
            
  | 18:58 | <mark> | depooled broken squids sq1 and sq3 | [production] | 
            
  | 18:26 | <RobH> | depooled sq48-sq50 for relocation | [production] | 
            
  | 18:17 | <RobH> | added sq44-sq47 back into pybal, relocation complete. | [production] | 
            
  | 17:45 | <brion> | sync-common-all to add w/test-headers.php | [production] | 
            
  | 17:28 | <RobH> | shutting down sq44-sq47 for relocation. | [production] | 
            
  | 17:27 | <RobH> | sq41 - sq43 back online. | [production] | 
            
  | 17:17 | <RobH> | sq40 oddness, but its back up now | [production] | 
            
  | 16:44 | <RobH> | accidentally pulled power for sq38, opps! | [production] | 
            
  | 15:36 | <RobH> | removed sq41 - sq43 from pybal to relocate from pmtpa to sdtpa | [production] | 
            
  | 15:34 | <domas> | srv178 running usr.sbin.apache2 aa profile in complain mode | [production] | 
            
  | 15:34 | <RobH> | removed sq40 from pybal to relocate from pmtpa to sdtpa | [production] | 
            
  
    | 2008-12-04
      
      § | 
    
  | 22:50 | <domas> | job runners are no longer blue on ganglia CPU graphs :((((((( | [production] | 
            
  | 22:45 | <domas> | fc4 maintenance, reniced job runners to 20 (10 behind apaches), installed apc3.0.19 (APC3.0.13 seams to have hit severe lock contention/busylooping at overloads) | [production] | 
            
  | 22:04 | <RobH> | re-enabled sq38 in pybal.  all is well | [production] | 
            
  | 22:02 | <RobH> | fired sq37-sq39 back up | [production] | 
            
  | 21:58 | <RobH> | shutdown sq37-sq39, cuz I need to balance the power distribution a bit better. | [production] | 
            
  | 21:40 | <RobH> | sq38 is trying to break my spirit, so i reinstalled it to show it who is boss (me!) | [production] | 
            
  | 21:02 | <RobH> | setup asw-a4-sdtpa and asw-a5-sdtpa on scs-a1-sdtpa | [production] | 
            
  | 20:52 | <mark> | Increased TCP buffers on srv88 (a Fedora), matching the Ubuntus - Fedora Apaches appear to get stuck/deadlocked on writes to Squids | [production] | 
            
  | 19:39 | <RobH> | pulled sq38 back out, as it is giving me issues.  need to fix the msw-a3-sdtpa before i can fix sq38. | [production] | 
            
  | 19:35 | <RobH> | added sq38, sq39 back into pybal | [production] | 
            
  | 19:25 | <RobH> | added sq36, sq37 back into pybal | [production] | 
            
  | 18:14 | <RobH> | I need to stop forgetting about lunch and stop working through it, oh well. | [production] |