| 2009-01-10
      
      § | 
    
  | 16:08 | <domas> | rotated 300g sampled-1000.log ;-) | [production] | 
            
  | 07:09 | <river> | applied current OS patches to ms2 and rebooted | [production] | 
            
  | 01:21 | <Tim> | restarted apache on srv95,srv114,srv37,srv49 | [production] | 
            
  | 01:19 | <Tim> | cleaned up disk space on db1. Still looks suspiciously like the master... | [production] | 
            
  | 00:33 | <brion> | redirecting old bylaws.pdf to wiki page bylaws on wikimediafoundation.org (foundation.conf update) | [production] | 
            
  | 00:13 | <brion> | reconfigured exim on wikitech to hopefully actually send mail out. whether it reaches anything, we'll see | [production] | 
            
  | 00:12 | <tomaszf> | turned off fundraising banners | [production] | 
            
  | 00:08 | <brion> | installed a mail server on wikitech server, hopefully | [production] | 
            
  
    | 2009-01-08
      
      § | 
    
  | 22:08 | <brion> | putting db12 back in service, caught up | [production] | 
            
  | 21:42 | <RobH> | changed the ip address for the management interfaces on sq31-sq50 | [production] | 
            
  | 21:30 | <RobH> | updated dns with the squids and srv mangement info for pmtpa | [production] | 
            
  | 21:16 | <brion> | taking load off db12 while it updates | [production] | 
            
  | 21:15 | <brion> | killing stuck query threads on db12 (lagged 13k seconds) | [production] | 
            
  | 20:23 | <RobH> | updated dns removing a large number of decommissioned servers from records. | [production] | 
            
  | 20:08 | <RobH> | pushed updates to dns for mangement ip allocations, changed mangement ips of search8-search12 | [production] | 
            
  | 19:43 | <RobH> | changed the mangement ip addresses of db5-db10 to fit into current ip scheme | [production] | 
            
  | 18:20 | <RobH> | updated dns for the management name resolution of db11-db30 | [production] | 
            
  | 18:11 | <RobH> | ms5 has lom access enabled and is ready for testing.  (Only one ethernet connection in lieu of the typical 3 on the thumper/thors) | [production] | 
            
  | 15:50 | <RobH> | srv118 reinstalled | [production] | 
            
  | 15:46 | <RobH> | srv136 is borked.  Even after reinstall, it will run for a few minutes, then lock hard.  Going to RMA it. | [production] | 
            
  | 15:38 | <RobH> | reinstalled srv136 and srv118 cuz they were pissing me off (a valid reinstallation reason if there ever was one.) | [production] | 
            
  | 15:09 | <RobH> | and srv118 back down, thing is borked. | [production] | 
            
  | 15:06 | <RobH> | srv118 back online and serving requests. | [production] | 
            
  | 15:01 | <RobH> | pushed db13 back into cluster, same with db14, from yesterdays work | [production] | 
            
  | 14:26 | <RobH> | srv101 back online and in lvs | [production] | 
            
  | 14:15 | <RobH> | reinstalled srv101, installing wikimedia-task-app packages now | [production] | 
            
  | 06:37 | <JeLuF> | rebooted db18. Mysqld was stuck but couldn't be killed. | [production] | 
            
  | 04:08 | <Tim> | migrated all locked wikis from $wgReadOnly(File) to permissions-based locking, so that stewards can edit the alternate project links, and so that various MediaWiki components don't break on page view | [production] | 
            
  | 03:57 | <river> | set up ms3/ms4 with solaris 10 update 6 | [production] | 
            
  
    | 2009-01-07
      
      § | 
    
  | 22:50 | <RobH> | db13 and db14 are replicating but not in the cluster (not sure if they are caught up) | [production] | 
            
  | 22:35 | <RobH> | updated power strip information for ps1-a1-sdtpa and balanced load | [production] | 
            
  | 22:35 | <RobH> | reseated mrj cable for csw1-sdtpa_1/13 | [production] | 
            
  | 21:36 | <RobH> | started up db13 and db14 | [production] | 
            
  | 21:19 | <RobH> | updating firmware on db13-db14 | [production] | 
            
  | 21:15 | <RobH> | shutdown db13 and db14 to fix lom lockup issue. | [production] | 
            
  | 20:52 | <RobH> | depooled db13 and db14 in db.php to reboot them and fix the SP lockup issue. | [production] | 
            
  | 20:49 | <RobH> | updating firmware on db16. | [production] | 
            
  | 20:43 | <RobH> | started mysql back up on db15 | [production] | 
            
  | 20:42 | <RobH> | cold reset of db16 to resolve lom issue.  will update firmware upon boot. | [production] | 
            
  | 20:39 | <RobH> | swappned hostnames on ms3 and ms4, updated racktables and dns to reflect change | [production] | 
            
  | 20:24 | <brion> | disabled wikidiff2 since it's not installed, and this apparaently is nicely broken | [production] | 
            
  | 20:21 | <RobH> | db15 now responsive to lom and ready to be re-integrated into the cluster | [production] |