| 2009-01-13
      
      § | 
    
  | 20:42 | <brion> | setting all pdf generators to use the new server | [production] | 
            
  | 20:40 | <brion> | testing pdf gen on erzurumi on testwiki | [production] | 
            
  | 20:35 | <RobH> | setup erzurumi for dev testing | [production] | 
            
  | 20:35 | <RobH> | some random updates on [[server roles]] to clean it up | [production] | 
            
  | 19:37 | <mark> | Restored normal situation, with 14907 -> 43821 traffic downpreffed to HGTN to avoid peering network congestion | [production] | 
            
  | 18:40 | <mark> | Retracted outbound announcement to all AMS-IX peers, 16265 and 13030 to force inbound via 1299 | [production] | 
            
  | 18:25 | <mark> | Undid any routing changes as they were not having the desired effect | [production] | 
            
  | 18:14 | <mark> | Prepended 43821 twice on outgoing announcements to 16265 to make pmtpa-esams path via nycx less attractive | [production] | 
            
  | 11:38 | <Tim> | reducing innodb_buffer_pool_size on db19, db21, db22, db29 | [production] | 
            
  | 09:15 | <Tim> | restarting mysqld on db23 again | [production] | 
            
  | 09:09 | <Tim> | restarting mysqld on db18 again | [production] | 
            
  | 07:08 | <Tim> | removed db23 from rotation, since I'm bringing it up soon and it will be lagged | [production] | 
            
  | 07:02 | <Tim> | shutting down mysqld on db18 for further mem usage tweak | [production] | 
            
  | 06:53 | <Tim> | fixed broken /etc/fstab on db23 via serial console | [production] | 
            
  | 06:42 | <Tim> | restarting db23 | [production] | 
            
  | 00:08 | <Tim> | repooling db18, has caught up | [production] | 
            
  
    | 2009-01-12
      
      § | 
    
  | 21:50 | <brion> | testing a scap after touching MessagesWuu.php to see if that clears borked serialized btis | [production] | 
            
  | 21:22 | <RobH> | erzurumi installed | [production] | 
            
  | 17:55 | <brion> | temporarily stopped apache on srv78, srv118 | [production] | 
            
  | 17:54 | <brion> | srv78 doesn't have upload5 mounted | [production] | 
            
  | 17:54 | <brion> | srv118 doesn't have upload5 mounted | [production] | 
            
  | 17:46 | <RobH> | fixed some settings for flaggedrevs in https://bugzilla.wikimedia.org/show_bug.cgi?id=14648 | [production] | 
            
  | 17:31 | <RobH> | per brion commented out db18 in db.php cuz its making other crap lag too much | [production] | 
            
  | 17:26 | <RobH> | updated flaggedrevs.php for https://bugzilla.wikimedia.org/show_bug.cgi?id=16365 | [production] | 
            
  | 17:23 | <RobH> | updated apache config on yongle for wap => mobile forwarding oversight per https://bugzilla.wikimedia.org/show_bug.cgi?id=16692 | [production] | 
            
  | 17:05 | <brion> | db18 is backlogged 191k seconds. depooling it if it's still in; complaints of hella lag | [production] | 
            
  | 15:32 | <Tim> | restarted mysqld on db18 with reduced memory usage, repooled | [production] | 
            
  | 14:12 | <Tim> | rebooting db18 | [production] | 
            
  | 13:20 | <Tim> | depooled db18 (is down) | [production] | 
            
  
    | 2009-01-10
      
      § | 
    
  | 16:08 | <domas> | rotated 300g sampled-1000.log ;-) | [production] | 
            
  | 07:09 | <river> | applied current OS patches to ms2 and rebooted | [production] | 
            
  | 01:21 | <Tim> | restarted apache on srv95,srv114,srv37,srv49 | [production] | 
            
  | 01:19 | <Tim> | cleaned up disk space on db1. Still looks suspiciously like the master... | [production] | 
            
  | 00:33 | <brion> | redirecting old bylaws.pdf to wiki page bylaws on wikimediafoundation.org (foundation.conf update) | [production] | 
            
  | 00:13 | <brion> | reconfigured exim on wikitech to hopefully actually send mail out. whether it reaches anything, we'll see | [production] | 
            
  | 00:12 | <tomaszf> | turned off fundraising banners | [production] | 
            
  | 00:08 | <brion> | installed a mail server on wikitech server, hopefully | [production] |