| 2008-12-11
      
      § | 
    
  | 23:10 | <brion> | restarted apache on 134, it's scary/corrupt | [production] | 
            
  | 22:55 | <brion> | manually syncing updated skin files to upload.wm.o ... | [production] | 
            
  | 22:53 | <brion> | scapping to r44474 | [production] | 
            
  | 21:31 | <brion> | don't sync yet; RC regression in r44033 being worked on | [production] | 
            
  | 19:41 | <brion-codereview> | removed conflicting live profiling hack from AutoLoader.php. Put this stuff in SVN, huh guys? | [production] | 
            
  | 19:39 | <brion-codereview> | applying flaggedrevs schema updates | [production] | 
            
  | 19:38 | <brion-codereview> | starting svn up for testwiki | [production] | 
            
  | 13:41 | <mark> | configured asw-a4-sdtpa and asw-a5-sdtpa, but no link | [production] | 
            
  | 10:41 | <mark> | bart out of disk space, removed some old cruft (mailman) | [production] | 
            
  
    | 2008-12-10
      
      § | 
    
  | 23:50 | <RobH> | pulled srv76 due to two dead fans  (yay for da bot) | [production] | 
            
  | 23:35 | <RobH> | srv78 reinstalled and in apache pool | [production] | 
            
  | 22:57 | <RobH> | srv78 kernel panic, old FC install, pulled for reinstall | [production] | 
            
  | 22:49 | <RobH> | sq1. sq3, sq6 cache cleaned and back online serving requests. | [production] | 
            
  | 22:35 | <RobH> | sq1, sq3, sq6 all unresponsive to console, flashing leds on kvm.  rebooted. | [production] | 
            
  | 20:40 | <RobH> | srv118 installation completed. | [production] | 
            
  | 20:00 | <RobH> | reinstalled srv118 after replacing dead parts.  installing packages now. | [production] | 
            
  | 19:48 | <RobH> | started rebuild of storage1 /c1/p0 into array | [production] | 
            
  | 19:47 | <RobH> | replaced disk /c1/p0 in storage1.  /c1/p13 is now bad as well, placing rma for it. | [production] | 
            
  | 19:14 | <RobH> | db13-db16 responsive to ssh. | [production] | 
            
  | 19:13 | <RobH> | db15 rebooted. | [production] | 
            
  | 18:05 | <RobH> | temp probes installed in a3-sdtpa | [production] | 
            
  
    | 2008-12-08
      
      § | 
    
  | 23:30 | <RobH> | updated enwiktionary group settings per https://bugzilla.wikimedia.org/show_bug.cgi?id=16248 | [production] | 
            
  | 23:24 | <brion> | updating Oversight for bug 16065 | [production] | 
            
  | 22:44 | <RobH> | no.wikimedia.org is now functioning per https://bugzilla.wikimedia.org/show_bug.cgi?id=15383 | [production] | 
            
  | 22:35 | <RobH> | made changes to InitialiseSettings.php for cswikisource per https://bugzilla.wikimedia.org/show_bug.cgi?id=16277 | [production] | 
            
  | 21:38 | <RobH> | authdns-update for no.wikimedia.org | [production] | 
            
  | 21:20 | <RobH> | running sync-common-all for wikimedia norge (found the php error) | [production] | 
            
  | 21:01 | <RobH> | its all back up now. | [production] | 
            
  | 20:59 | <RobH> | I stupidly crashed the site with a php typo, rolling back my changes since i was ignorant and did not php -l  ;_; | [production] | 
            
  | 20:58 | <RobH> | setup wikimedia norge wiki per https://bugzilla.wikimedia.org/show_bug.cgi?id=15383 | [production] | 
            
  | 19:23 | <brion> | updating OggHandler for fix for bug 15920 (chopped oggs) | [production] | 
            
  | 15:57 | <mark> | Set up mirroring of traffic of e7/2 to e7/14 for testing the fiber patch loop/optics | [production] | 
            
  
    | 2008-12-05
      
      § | 
    
  | 22:42 | <domas> | srv47 is running scaler usr.sbin.apache2 aa profile in learning mode | [production] | 
            
  | 22:33 | <RobH> | sq50 reinstalled and back in rotation | [production] | 
            
  | 22:26 | <RobH> | finished setup on srv146, back in apache pool | [production] | 
            
  | 21:32 | <RobH> | setting up packages on srv146 | [production] | 
            
  | 21:32 | <RobH> | reinstalling sq50 | [production] | 
            
  | 21:27 | <brion> | pointing SiteMatrix at local copy, not NFS master, of langlist file | [production] | 
            
  | 19:19 | <RobH> | added sq48, and sq49 back into pool.  sq50 pending reinstallation. | [production] | 
            
  | 18:58 | <mark> | depooled broken squids sq1 and sq3 | [production] | 
            
  | 18:26 | <RobH> | depooled sq48-sq50 for relocation | [production] | 
            
  | 18:17 | <RobH> | added sq44-sq47 back into pybal, relocation complete. | [production] | 
            
  | 17:45 | <brion> | sync-common-all to add w/test-headers.php | [production] | 
            
  | 17:28 | <RobH> | shutting down sq44-sq47 for relocation. | [production] | 
            
  | 17:27 | <RobH> | sq41 - sq43 back online. | [production] |