| 
      
        2009-08-21
      
      §
     | 
  
    
  | 22:16 | 
  <atglenn> | 
  starting copy of image data to ms5 (consider this timing tests, not the real thing): zfs send running in screen session | 
  [production] | 
            
  | 20:51 | 
  <aZaFred_> | 
  restarted memcached on srv212 | 
  [production] | 
            
  | 20:36 | 
  <fvassard> | 
  synchronized php-1.5/wmf-config/db.php  'commenting out the rc2 cluster since it doesnt seem to be used anymore...' | 
  [production] | 
            
  | 20:01 | 
  <fvassard> | 
  synchronized php-1.5/wmf-config/mc.php  'Removed srv52 and added srv219 in its place since srv52 is not behaving...' | 
  [production] | 
            
  | 17:48 | 
  <aZaFred_> | 
  fixed nrpe config on adler as it was set not to accept any connection not coming form 127.0.0.1 | 
  [production] | 
            
  | 17:37 | 
  <aZaFred_> | 
  switching Nagios from bart to spence (its new home) and decomissioning Nagios on Bart. | 
  [production] | 
            
  | 16:34 | 
  <aZaFred_> | 
  changed settings for sq43 since /dev/sdd disappeared. | 
  [production] | 
            
  | 16:15 | 
  <aZaFred_> | 
  restarted memcached on srv89 | 
  [production] | 
            
  | 15:09 | 
  <hcatlin> | 
  Mobile1 just had af, az, bn, ca, bg, and drb languages deployed. | 
  [production] | 
            
  
    | 
      
        2009-08-20
      
      §
     | 
  
    
  | 23:02 | 
  <domas> | 
  opensearch is causing most of 'revision' SQL load (probably text too).  search overall accounts for more than 25% of database::query time :-) | 
  [production] | 
            
  | 19:48 | 
  <midom> | 
  synchronized php-1.5/includes/Revision.php  'removing profiling hooks' | 
  [production] | 
            
  | 19:07 | 
  <midom> | 
  synchronized php-1.5/includes/Revision.php  'profiling hooks' | 
  [production] | 
            
  | 18:42 | 
  <brion> | 
  running featured article imports on readerfeedback & flaggedrevs test wikis | 
  [production] | 
            
  | 18:25 | 
  <brion> | 
  pulling enwiki featured articles to copy into readerfeedback & flaggrevs test wikis | 
  [production] | 
            
  | 18:18 | 
  <domas> | 
  Just logging, that SUL and job queue was broken because of bogus entry in dblist | 
  [production] | 
            
  | 18:18 | 
  <domas> | 
  DNS was unhappy! | 
  [production] | 
            
  | 18:09 | 
  <brion> | 
  nuked bogus 'en_flaggedrevs_labswikimedia' entry from all.dblist; per domas it broke job queue | 
  [production] | 
            
  | 18:00 | 
  <brion> | 
  synchronized php-1.5/wmf-config/flaggedrevs.php  'adding flaggedrevs_wikimedia config' | 
  [production] | 
            
  | 17:41 | 
  <fvassard> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  'Added default languages for flaggedrevs.labs and readerfeedback.labs' | 
  [production] | 
            
  | 17:22 | 
  <Fred> | 
  synched db-list file for readerfeedback/flaggedrevs test wikis | 
  [production] | 
            
  | 17:22 | 
  <Fred> | 
  removed srv52 from dsh nodelist as it was causing sync* to fail (amongst other things) | 
  [production] | 
            
  | 16:58 | 
  <brion> | 
  poking at readerfeedback/flaggedrevs test wikis setup with fred | 
  [production] | 
            
  | 16:42 | 
  <jeluf> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  '20300 - Fix arwikisource namespaces' | 
  [production] | 
            
  | 12:25 | 
  <andrew> | 
  synchronized php-1.5/wmf-config/abusefilter.php  'Remove abusefilter-private right from abuse filter users on nlwiki, it leaks IP addresses to unidentified users' | 
  [production] | 
            
  
    | 
      
        2009-08-19
      
      §
     | 
  
    
  | 21:00 | 
  <robh> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  'labs settings' | 
  [production] | 
            
  | 20:53 | 
  <Rob> | 
  added dns for readerfeedback and flaggedrevs lab sites | 
  [production] | 
            
  | 20:47 | 
  <robh> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  'flaggedrevslabswiki' | 
  [production] | 
            
  | 18:57 | 
  <Rob> | 
  and bug 19019 | 
  [production] | 
            
  | 18:57 | 
  <robh> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  'setting wgVariantArticlePath for bug 8532' | 
  [production] | 
            
  | 17:29 | 
  <robh> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  | 
  [production] | 
            
  | 17:14 | 
  <robh> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  'rob undoing changes he made' | 
  [production] | 
            
  | 17:08 | 
  <robh> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  | 
  [production] | 
            
  | 17:06 | 
  <robh> | 
  synchronized wmf-deployment/wmf-config/InitialiseSettings.php  | 
  [production] | 
            
  | 15:39 | 
  <mark> | 
  Shutting down MySQL on ms2, rebooting the machine | 
  [production] | 
            
  | 15:13 | 
  <midom> | 
  synchronized php-1.5/wmf-config/CommonSettings.php  'db/mc file renames' | 
  [production] | 
            
  | 15:12 | 
  <midom> | 
  synchronized php-1.5/wmf-config/db.php  | 
  [production] | 
            
  | 15:12 | 
  <midom> | 
  synchronized php-1.5/wmf-config/mc.php  | 
  [production] | 
            
  | 15:07 | 
  <midom> | 
  synchronized php-1.5/wmf-config/db-pmtpa.php  | 
  [production] | 
            
  | 14:27 | 
  <apergos> | 
  snapshots enabled on ms1 about ten hours ago after a small disaster with image deletion affecting wps: ch, cdo, ce, ceb (restored from ms6 and elsewhere, thumbs not impacted)  | 
  [production] | 
            
  | 12:11 | 
  <mark> | 
  Repeated cache dir changes for SSD testing on sq49... they disappeared last week, how come? Did someone overwrite the upload-settings squid conf file? | 
  [production] | 
            
  | 10:40 | 
  <rainman-sr> | 
  updated to latest mwdumper on searchidx1 to fix en.wp index update breakage | 
  [production] | 
            
  | 04:38 | 
  <aaron> | 
  synchronized php-1.5/wmf-config/flaggedrevs.php  'Give unreviewedpages right to Coders' | 
  [production] | 
            
  
    | 
      
        2009-08-18
      
      §
     | 
  
    
  | 23:16 | 
  <brion> | 
  synchronized php-1.5/languages/messages/MessagesBg.php  'touching for cache test' | 
  [production] | 
            
  | 22:24 | 
  <brion> | 
  configured wikitech-l and mediawiki-l to accept jpeg, png, and gif attachments | 
  [production] | 
            
  | 20:14 | 
  <jeluf> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  '#20300 arwikisource namespaces' | 
  [production] | 
            
  | 19:34 | 
  <brion> | 
  synchronized php-1.5/wmf-config/CommonSettings.php  'bump style version' | 
  [production] | 
            
  | 18:31 | 
  <jeluf> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  '#20301 bnwiki sitename' | 
  [production] | 
            
  | 12:12 | 
  <domas> | 
  ruby processes were running out of FDs because of new logging code - fds weren't closed, leading to 1024 limit hit in ~16 hours :) | 
  [production] | 
            
  | 11:19 | 
  <domas> | 
  mobile1 was serving 502's. mobile1 is a fucked up mess too. restarted merb via /etc/init.d/thin start, seems to have brought mobile1 up | 
  [production] |