| 
      
        2010-02-07
      
      §
     | 
  
    
  | 23:36 | 
  <Andrew> | 
  Clarification: Somebody should fix it, not I think it's fixed now. | 
  [production] | 
            
  | 23:35 | 
  <Andrew> | 
  The issue of centralnotice pulling notices from the wrong place should be fixed now, though | 
  [production] | 
            
  | 23:34 | 
  <Andrew> | 
  Moved /mnt/upload5/centralnotice/centralnotice.js and /mnt/upload5/centralnotice/wikipedia/centralnotice.js to centralnotice-old.js, should stop fundraising banners from appearing. | 
  [production] | 
            
  | 23:33 | 
  <Andrew> | 
  Issue with fundraising banners appearing on random pages caused by clients with the URL http://upload.wikimedia.org/centralnotice/wikipedia//centralnotice.js?257z6 | 
  [production] | 
            
  | 22:11 | 
  <Andrew> | 
  Reports that fundraising banners are appearing on random pages for some users. | 
  [production] | 
            
  | 21:32 | 
  <domas> | 
  we're running on single ES copy. | 
  [production] | 
            
  | 21:32 | 
  <midom> | 
  synchronized php-1.5/wmf-config/db.php  'disabling broken storage1' | 
  [production] | 
            
  | 20:05 | 
  <domas> | 
  removing all fixes I've been doing to recover mail. WMF staff will handle this tomorrow, or the problems will be gone by then. | 
  [production] | 
            
  | 19:01 | 
  <domas> | 
  blackholed few thousand IPs on mchenry via OUTPUT chain (using -j REJECT action) | 
  [production] | 
            
  
    | 
      
        2010-02-05
      
      §
     | 
  
    
  | 06:00 | 
  <andrew> | 
  synchronized php-1.5/wmf-config/liquidthreads.php  'Re-activate LiquidThreads email notifications, DoS has passed' | 
  [production] | 
            
  | 05:22 | 
  <Tim> | 
  freed up a little space on ms3 by reducing expire_logs_days from 14 to 7, and running flush logs | 
  [production] | 
            
  | 02:45 | 
  <apergos> | 
  cleared out /tmp on srv169... | 
  [production] | 
            
  | 00:23 | 
  <catrope> | 
  synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php  'Bump style version for jQuery UI stylesheet' | 
  [production] | 
            
  | 00:06 | 
  <catrope> | 
  synchronized php-1.5/extensions/UsabilityInitiative/UsabilityInitiative.hooks.php  'Bump style version for combined.min.css' | 
  [production] | 
            
  | 00:03 | 
  <catrope> | 
  synchronized php-1.5/extensions/UsabilityInitiative/css/combined.min.css  'Resync this, doesn't seem to have been picked up right' | 
  [production] | 
            
  
    | 
      
        2010-02-04
      
      §
     | 
  
    
  | 23:55 | 
  <catrope> | 
  synchronized php-1.5/wmf-config/CommonSettings.php  'Bump style version appendix' | 
  [production] | 
            
  | 23:55 | 
  <catrope> | 
  synchronized php-1.5/extensions/UsabilityInitiative/WikiEditor/WikiEditor.combined.min.js  'r62002' | 
  [production] | 
            
  | 23:50 | 
  <catrope> | 
  synchronized php-1.5/wmf-config/CommonSettings.php  'Bump style version appendix' | 
  [production] | 
            
  | 23:37 | 
  <RoanKattouw> | 
  Running scap to deploy new UsabilityInitiative code | 
  [production] | 
            
  | 22:55 | 
  <catrope> | 
  synchronized php-1.5/wmf-config/CommonSettings.php  'Bump UsabilityInitiative_alpha style version appendix' | 
  [production] | 
            
  | 22:54 | 
  <catrope> | 
  synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins.combined.min.js  'r61995' | 
  [production] | 
            
  | 22:28 | 
  <catrope> | 
  synchronized php-1.5/wmf-config/InitialiseSettings.php  'Enable UsabilityInitiative_alpha on usabilitywiki' | 
  [production] | 
            
  | 22:20 | 
  <tstarling> | 
  synchronized php-1.5/includes/api/ApiUpload.php  'disabled chunk upload' | 
  [production] | 
            
  | 22:09 | 
  <catrope> | 
  synchronized php-1.5/wmf-config/CommonSettings.php  'Bump UsabilityInitiative_alpha style version appendix' | 
  [production] | 
            
  | 22:07 | 
  <catrope> | 
  synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.hooks.php  | 
  [production] | 
            
  | 22:06 | 
  <catrope> | 
  synchronized php-1.5/extensions/UsabilityInitiative_alpha/UsabilityInitiative.hooks.php  | 
  [production] | 
            
  | 22:06 | 
  <catrope> | 
  synchronized php-1.5/extensions/UsabilityInitiative_alpha/css/combined.min.css  | 
  [production] | 
            
  | 22:06 | 
  <catrope> | 
  synchronized php-1.5/extensions/UsabilityInitiative_alpha/WikiEditor/WikiEditor.combined.min.js  | 
  [production] | 
            
  | 22:06 | 
  <catrope> | 
  synchronized php-1.5/extensions/UsabilityInitiative_alpha/js/plugins.combined.min.js  | 
  [production] | 
            
  | 22:05 | 
  <catrope> | 
  synchronized php-1.5/extensions/UsabilityInitiative_alpha/images/wikiEditor/dialogs/insert-link-external-rtl.png  | 
  [production] | 
            
  | 22:04 | 
  <RoanKattouw> | 
  Deploying r61993 with individual sync-files | 
  [production] | 
            
  | 21:15 | 
  <atglenn> | 
  turned off hourly snaps on ms7, we'll have daily plus the standard replication | 
  [production] | 
            
  | 18:49 | 
  <atglenn> | 
  cleared /tmp on srv175 blah blah  | 
  [production] | 
            
  | 18:36 | 
  <atglenn> | 
  cleared out php* files from /tmp on srv163 (need cron job) | 
  [production] | 
            
  | 10:35 | 
  <mark> | 
  Removed sq11-15 from the upload LVS pool | 
  [production] | 
            
  | 10:30 | 
  <mark> | 
  Removed sq11-15 from the upload CARP pool | 
  [production] | 
            
  | 03:28 | 
  <apergos> | 
  added /usr/local/apache/common/php to include path of php on fenari  | 
  [production] |