| 2009-02-20
      
      § | 
    
  | 16:19 | <NicDumZ> | nagios definitely needs a nagia | [production] | 
            
  | 16:17 | <domas> | plenty of fedoras had futex deadlocks | [production] | 
            
  | 16:16 | <Rob> | upgraded kernel and squid on sq14 and sq15 | [production] | 
            
  | 15:49 | <Rob> | updated squid and kernel on sq13, rebooted, back online | [production] | 
            
  | 14:59 | <Rob> | upgraded squid and kernel on sq5, sq6, sq7, sq8 | [production] | 
            
  | 14:51 | <Rob> | upgraded squid and kernel on sq2-sq4 | [production] | 
            
  | 14:50 | <Tim> | updated ContactPage extension, will deploy it on nlwiki shortly | [production] | 
            
  | 10:00 | <mark> | Started backend squid on sq26, it was gone | [production] | 
            
  
    | 2009-02-19
      
      § | 
    
  | 23:54 | <brion> | updating AbuseFilter to r47523 :P | [production] | 
            
  | 23:51 | <brion> | updating AbuseFilter to r47522 | [production] | 
            
  | 23:40 | <brion> | updating FlaggedRevs to r47522 | [production] | 
            
  | 23:39 | <Andrew> | Enabled Abuse Filter on MediaWiki.org | [production] | 
            
  | 23:17 | <mark> | Stopped experimental varnish on sq1, please keep Squid off as well | [production] | 
            
  | 22:52 | <Andrew> | Allowed bureaucrats to remove 'sysop' right on testwiki. | [production] | 
            
  | 22:42 | <brion> | updating includes/api to r47522 to fix a couple regressions | [production] | 
            
  | 22:15 | <mark> | Started an experimental varnish instance on sq1 port 80 | [production] | 
            
  | 21:23 | <mark> | Stopped Squids on sq1 | [production] | 
            
  | 14:23 | <Tim> | removing memcached from srv154,srv155,srv157,srv158,srv169,srv170 | [production] | 
            
  | 14:18 | <Tim> | started memcached on srv190-199 | [production] | 
            
  | 14:06 | <mark> | Added "vport=80" to the http_host directive on all backend squids, to force Squid to use the default HTTP port, 80 | [production] | 
            
  | 10:53 | <domas> | livemerged r47483 (backlinks cache read explicit order, :( ) | [production] | 
            
  
    | 2009-02-18
      
      § | 
    
  | 23:30 | <river> | test | [production] | 
            
  | 23:12 | <domas> | did anyone notice that our GFDL licensed log messages are syndicated as CC-BY? | [production] | 
            
  | 23:09 | <domas> | @mark, go fix our servers! :) | [production] | 
            
  | 23:09 | <mark> | it's good to know that some other @mark will be responsible now | [production] | 
            
  | 23:06 | <domas> | #tag #clouds #are #awesome | [production] | 
            
  | 23:04 | <domas> | @brion, will we have to log in #web20 formats from now on? #roundedcorners and such? | [production] | 
            
  | 22:59 | <brion> | poke poke | [production] | 
            
  | 22:44 | <paperweight> | changed root password on all servers to 'web20rox' | [production] | 
            
  | 22:43 | <idiotsinthehouse> | hey all identi.ca idiots! | [production] | 
            
  | 22:41 | <Andrew> | morebots back up, now logs to identi.ca with the name wikimediatech | [production] | 
            
  | 22:30 | <Andrew> | test | [production] | 
            
  | 22:28 | <Andrew> | test | [production] | 
            
  | 22:27 | <Andrew> | test | [production] | 
            
  | 22:20 | <Andrew> | This is a test; a test is this | [production] | 
            
  | 21:59 | <Rob> | PDF creation moved to pdf1 | [production] | 
            
  | 21:58 | <Rob> | changed pdf generation from eruzumi to pdf1, testing. | [production] | 
            
  | 19:21 | <Rob> | srv255 changed to pdf1 and moved, drac setup along with dns resolution | [production] | 
            
  | 19:19 | <brion> | scapping | [production] | 
            
  | 19:18 | <brion> | svn up'ing test to r47457 | [production] | 
            
  | 18:37 | <Rob> | reinstalling srv209 due to dhcp misconfiguration making it think it was srv208 | [production] | 
            
  | 15:13 | <mark> | Restarted all upload frontend squids to get rid of the memleaking | [production] | 
            
  | 14:20 | <mark> | Blocked all non-GET/HEAD HTTP methods in requests to upload frontend squids | [production] | 
            
  | 12:46 | <Tim> | put r47447 live for temporary proposed fix of bug 17552 | [production] | 
            
  | 08:38 | <Tim> | svn up r47434 to fix Special:BrokenRedirects | [production] | 
            
  | 08:04 | <Tim> | cleaned up binlogs on db2 | [production] | 
            
  | 06:33 | <brion> | note there's a live hack in api categorymembers query which may be breaking lookups | [production] | 
            
  | 05:54 | <Tim> | set up bugzilla attachment_base, pointing to the new domain http://bug-attachment.wikimedia.org/, and set allow_attachment_display=on | [production] | 
            
  | 05:51 | <brion> | disabling $wgTorTagChanges in CommonSettings after the ext gets loaded (needs fix for testwiki) | [production] | 
            
  | 05:46 | <brion> | syncing reverted expr.php w/o bc stuff | [production] |