| 
      
        2009-03-20
      
      §
     | 
  
    
  | 23:58 | 
  <Tim> | 
  repooled srv126, part of cluster12, appears to be up and working | 
  [production] | 
            
  | 23:58 | 
  <tstarling> | 
  synchronized php-1.5/db.php  | 
  [production] | 
            
  | 23:54 | 
  <Tim> | 
  depooled srv125 from ES, has been down for 12 days. cluster12 is now down to 1 server | 
  [production] | 
            
  | 23:54 | 
  <tstarling> | 
  synchronized php-1.5/db.php  | 
  [production] | 
            
  | 23:52 | 
  <Tim> | 
  install ganglia-metrics on db24 | 
  [production] | 
            
  | 23:13 | 
  <Tim> | 
  set up logrotate on locke, using the same script that we use for MW debug logs | 
  [production] | 
            
  | 22:56 | 
  <domas> | 
  kickbanned security threat from #wikimedia-tech, was trying to install keylogger and steal our passwords | 
  [production] | 
            
  | 22:54 | 
  <Tim> | 
  removed old squid log stream going to iris. Set up a log stream going from all the squids to locke. | 
  [production] | 
            
  | 22:37 | 
  <Tim> | 
  depooled adler, it's down | 
  [production] | 
            
  | 22:36 | 
  <tstarling> | 
  synchronized php-1.5/db.php  'depooled adler' | 
  [production] | 
            
  | 21:01 | 
  <mark> | 
  removed vlan 5 on csw5-pmtpa that was accidently created/left behind by Tim | 
  [production] | 
            
  | 20:47 | 
  <domas> | 
  installed snaprotate on db26, enabled snapshots with 2x8h schedule, updated [[Database snapshots]] | 
  [production] | 
            
  | 20:44 | 
  <midom> | 
  synchronized php-1.5/includes/api/ApiQueryRevisions.php  'livemerging r48642' | 
  [production] | 
            
  | 20:43 | 
  <midom> | 
  synchronized php-1.5/includes/filerepo/ArchivedFile.php  'livemerging r48644' | 
  [production] | 
            
  | 20:27 | 
  <domas> | 
  someone who wrote ArchivedFile::load, needs some pain and torture applied (query doesn't use index... ;-) | 
  [production] | 
            
  | 20:25 | 
  <midom> | 
  synchronized php-1.5/db.php  'reduced load for db26 from 200 to 100, as it has reduced amount of RAM and increased amount of other work' | 
  [production] | 
            
  | 20:09 | 
  <domas> | 
  added db26 to s1 pool | 
  [production] | 
            
  | 20:08 | 
  <midom> | 
  synchronized php-1.5/db.php  | 
  [production] | 
            
  | 18:47 | 
  <mark> | 
  JeLuF> |log \\o/ | 
  [production] | 
            
  | 18:46 | 
  <domas> | 
  pooled db22 back in, as it caught up on replication! | 
  [production] | 
            
  | 18:46 | 
  <mark> | 
  raised karma of /h/w/b/reset-mysql-slave | 
  [production] | 
            
  | 18:46 | 
  <midom> | 
  synchronized php-1.5/db.php  | 
  [production] | 
            
  | 18:45 | 
  <domas> | 
  ran /h/w/b/reset-mysql/slave on db26, after copy from db22! | 
  [production] | 
            
  | 18:45 | 
  <domas> | 
  wrote /h/w/b/reset-mysql-slave to reset mysql slaves!  | 
  [production] | 
            
  | 18:32 | 
  <jeluf> | 
  synchronized php-1.5/InitialiseSettings.php  '15880 Pseudo-Namespace on Korean Wikipedia' | 
  [production] | 
            
  | 18:25 | 
  <jeluf> | 
  synchronized php-1.5/InitialiseSettings.php  '17699 Create Appendix namespace on Spanish Wiktionary' | 
  [production] | 
            
  | 18:19 | 
  <jeluf> | 
  synchronized php-1.5/InitialiseSettings.php  '17987 Set $wgBlockAllowsUTEdit = true for zh.wikipedia' | 
  [production] | 
            
  | 18:16 | 
  <jeluf> | 
  synchronized php-1.5/InitialiseSettings.php  '16446 tlwikibooks namespaces to be searched by default' | 
  [production] | 
            
  | 18:14 | 
  <jeluf> | 
  synchronized php-1.5/InitialiseSettings.php  '16446 Add Pagluluto: namespace to tlwikibooks' | 
  [production] | 
            
  | 18:12 | 
  <domas> | 
  db26 bootstrap problem was inconsistent ibdata specification, probably first copy would've been enough :) | 
  [production] | 
            
  | 17:55 | 
  <jeluf> | 
  synchronized php-1.5/InitialiseSettings.php  '17818 Create "Wikijunior" namespace on Polish Wikibooks (fix)' | 
  [production] | 
            
  | 17:53 | 
  <jeluf> | 
  synchronized php-1.5/InitialiseSettings.php  '17818 Create "Wikijunior" namespace on Polish Wikibooks' | 
  [production] | 
            
  | 17:41 | 
  <domas> | 
  I'm out of luck, copy from db22 to db26 has failed again | 
  [production] | 
            
  | 17:32 | 
  <JeLuF> | 
  changed sync-common-file to automatically log messages to [[Server admin log]] | 
  [production] | 
            
  | 17:27 | 
  <jeluf> | 
  synchronized php-1.5/InitialiseSettings.php "17739 nowiktionary logo" | 
  [production] | 
            
  | 17:18 | 
  <JeLuF> | 
  changed slwikibooks, slwikisource logos to $stdlogo | 
  [production] | 
            
  | 17:17 | 
  <domas> | 
  tried to poke Werdna about AbuseFilter regexp warnings, but he didn't listen (check remotelogtail @ db20 ) | 
  [production] | 
            
  | 17:16 | 
  <domas> | 
  restarted srv170 - was throwing occasional segfaults | 
  [production] | 
            
  | 17:15 | 
  <domas> | 
  'RESET SLAVE" on db22, to clean the relay log congestion | 
  [production] | 
            
  | 17:14 | 
  <JeLuF> | 
  changed mgwiki logo to $stdlogo | 
  [production] |