| 
      
        2011-03-08
      
      ยง
     | 
  
    
  | 20:35 | 
  <catrope> | 
  synchronized php-1.17/includes/Title.php  | 
  [production] | 
            
  | 20:34 | 
  <catrope> | 
  synchronized php-1.17/includes/LinksUpdate.php  | 
  [production] | 
            
  | 20:34 | 
  <catrope> | 
  synchronized php-1.17/includes/CategoryPage.php  | 
  [production] | 
            
  | 20:34 | 
  <catrope> | 
  synchronized php-1.17/extensions/intersection/DynamicPageList.php  | 
  [production] | 
            
  | 20:34 | 
  <catrope> | 
  synchronized php-1.17/extensions/FlaggedRevs/specialpages/UnreviewedPages_body.php  | 
  [production] | 
            
  | 20:34 | 
  <catrope> | 
  synchronized php-1.17/extensions/Collection/Collection.body.php  | 
  [production] | 
            
  | 20:33 | 
  <catrope> | 
  synchronized php-1.17/extensions/CategoryTree/CategoryTreeFunctions.php  | 
  [production] | 
            
  | 20:32 | 
  <RoanKattouw> | 
  Re-syncing a bunch of files without changes, didn't seem to have been picked up right by scap | 
  [production] | 
            
  | 19:38 | 
  <catrope> | 
  synchronized php-1.17/includes/api/ApiQueryCategoryMembers.php  'Sync without changes, maybe this didn't get picked up right' | 
  [production] | 
            
  | 19:35 | 
  <catrope> | 
  synchronized php-1.17/includes/CategoryPage.php  'Use array_merge instead of + for paranoia' | 
  [production] | 
            
  | 17:46 | 
  <RoanKattouw> | 
  Running updateCollation.php on all wikis | 
  [production] | 
            
  | 17:37 | 
  <RoanKattouw> | 
  Running updateCollation.php on cswiktionary | 
  [production] | 
            
  | 17:36 | 
  <robh> | 
  synchronizing Wikimedia installation... Revision: 83531:  | 
  [production] | 
            
  | 17:24 | 
  <catrope> | 
  synchronizing Wikimedia installation... Revision: 83531:  | 
  [production] | 
            
  | 17:23 | 
  <RoanKattouw> | 
  Deploying category collation code changes (r83254, r83531) sitewide using scap | 
  [production] | 
            
  | 15:16 | 
  <RoanKattouw> | 
  Updating MediaWiki on fenari from SVN to reintroduce category collation changes. For now, this is only for testing, so don't run sync-common-all or scap just yet | 
  [production] | 
            
  | 15:03 | 
  <RoanKattouw> | 
  Looks fine | 
  [production] | 
            
  | 15:02 | 
  <RoanKattouw> | 
  upgrade-1.17wmf1-final.php seems to have finished, running it again to verify it succeeded | 
  [production] | 
            
  | 13:28 | 
  <catrope> | 
  synchronized php-1.17/wmf-config/InitialiseSettings.php  'NamespaceWithSubpages settings for Czech wikis, per Danny B' | 
  [production] | 
            
  | 12:44 | 
  <mark> | 
  Changed 'wrr' scheduler to 'sh' on payments and owas LVS services | 
  [production] | 
            
  | 12:16 | 
  <midom> | 
  synchronized php-1.17/extensions/FlaggedRevs/specialpages/PendingChanges_body.php  'syncing http://p.defau.lt/?9RQEZQiZ_WeMT7_f7TpX6A live too' | 
  [production] | 
            
  | 12:09 | 
  <midom> | 
  synchronized php-1.17/extensions/FlaggedRevs/FlaggedRevs.hooks.php  'syncing livehack http://p.defau.lt/?BQwaKa6fH17X_CLt74khzA' | 
  [production] | 
            
  | 10:55 | 
  <Tim> | 
  running upgrade-1.17wmf1-final.php to upgrade old masters | 
  [production] | 
            
  | 10:53 | 
  <Tim> | 
  master positions: db36-bin.000001:106, db13-bin.000001:106, db31-bin.000001:106, db23-bin.001:79 | 
  [production] | 
            
  | 10:53 | 
  <tstarling> | 
  synchronized php-1.17/wmf-config/db.php  | 
  [production] | 
            
  | 10:52 | 
  <tstarling> | 
  synchronized php-1.17/wmf-config/db.php  | 
  [production] | 
            
  | 10:52 | 
  <Tim> | 
  switching master for s5: db14 -> db23 | 
  [production] | 
            
  | 10:52 | 
  <tstarling> | 
  synchronized php-1.17/wmf-config/db.php  | 
  [production] | 
            
  | 10:52 | 
  <tstarling> | 
  synchronized php-1.17/wmf-config/db.php  | 
  [production] | 
            
  | 10:51 | 
  <Tim> | 
  switching master for s4: db22 -> db31 | 
  [production] | 
            
  | 10:51 | 
  <tstarling> | 
  synchronized php-1.17/wmf-config/db.php  | 
  [production] | 
            
  | 10:49 | 
  <tstarling> | 
  synchronized php-1.17/wmf-config/db.php  | 
  [production] | 
            
  | 10:48 | 
  <Tim> | 
  switching master for s2: db30 -> db13 | 
  [production] | 
            
  | 10:48 | 
  <tstarling> | 
  synchronized php-1.17/wmf-config/db.php  | 
  [production] | 
            
  | 10:48 | 
  <tstarling> | 
  synchronized php-1.17/wmf-config/db.php  | 
  [production] | 
            
  | 10:48 | 
  <Tim> | 
  switching master for s1: db32 -> db36 | 
  [production] | 
            
  | 10:08 | 
  <midom> | 
  synchronized php-1.17/wmf-config/db.php  'adding db33 back with snapshotting disabled - checking if that plus temptables were causing the melt' | 
  [production] | 
            
  | 08:53 | 
  <jeluf> | 
  synchronized php-1.17/wmf-config/db.php  'removed db33 from s4' | 
  [production] | 
            
  | 08:51 | 
  <JeLuF> | 
  db33 down, commons unavailable | 
  [production] | 
            
  | 07:31 | 
  <tstarling> | 
  synchronized php-1.17/wmf-config/CommonSettings.php  're-enabled squid purging' | 
  [production] | 
            
  | 07:24 | 
  <tstarling> | 
  synchronized php-1.17/maintenance/jobs-loop.sh  'temp hack: no priority for htmlCacheUpdate' | 
  [production] | 
            
  | 07:22 | 
  <Tim> | 
  it appears more likely that the overload was due to htmlCacheUpdate jobs updating page_touched, in a flood after jobs-loop.sh fixes | 
  [production] | 
            
  | 07:18 | 
  <Tim> | 
  apaches overloaded after I fixed job queue squid purging. Disabled all squid purging and stopped job queue. | 
  [production] | 
            
  | 07:15 | 
  <tstarling> | 
  synchronized php-1.17/wmf-config/CommonSettings.php  | 
  [production] | 
            
  | 06:47 | 
  <tstarling> | 
  synchronized php-1.17/maintenance/jobs-loop.sh  | 
  [production] | 
            
  | 06:40 | 
  <tstarling> | 
  synchronized php-1.17/includes/SquidUpdate.php  | 
  [production] | 
            
  | 06:40 | 
  <tstarling> | 
  synchronized php-1.17/includes/Title.php  | 
  [production] | 
            
  | 06:39 | 
  <tstarling> | 
  synchronized php-1.17/includes/DefaultSettings.php  | 
  [production] | 
            
  | 06:06 | 
  <tstarling> | 
  synchronized php-1.17/maintenance/jobs-loop.sh  | 
  [production] | 
            
  | 05:51 | 
  <Tim> | 
  deployed new jobs-loop.sh script which prioritises fast important jobs like email notification. Thousands of emails are currently being sent. | 
  [production] |