| 
      
        2011-11-30
      
      §
     | 
  
    
  | 16:03 | 
  <reedy> | 
  synchronizing Wikimedia installation... : Pushing ApiSandbox out to the cluster, not enabled anywhere yet | 
  [production] | 
            
  | 15:56 | 
  <reedy> | 
  synchronized wmf-config/InitialiseSettings.php  'Bug 32500 - Defining transwiki sources for sa.wikipedia.org' | 
  [production] | 
            
  | 15:49 | 
  <catrope> | 
  synchronized wmf-config/liquidthreads.php  'Fix notice about wmgLQTUserControlNamespaces being undefined' | 
  [production] | 
            
  | 15:46 | 
  <reedy> | 
  synchronized wmf-config/InitialiseSettings.php  'Group tweak for [[bugzilla:32637|bug 32637]]' | 
  [production] | 
            
  | 15:37 | 
  <reedy> | 
  synchronized wmf-config/CommonSettings.php  'Adding wmgBabelUseUserLanguage and making wmgBabelDefaultLevel from InitialiseSettings be respected' | 
  [production] | 
            
  | 15:35 | 
  <reedy> | 
  synchronized wmf-config/InitialiseSettings.php  'Creating wmgBabelUseUserLanguage' | 
  [production] | 
            
  | 14:24 | 
  <reedy> | 
  synchronized php-1.18/extensions/VipsScaler/modules/ext.vipsScaler/ext.vipsScaler.js  '[[rev:104663|r104663]]' | 
  [production] | 
            
  | 14:15 | 
  <cmjohnson1> | 
  shutting down ms3 | 
  [production] | 
            
  | 13:48 | 
  <RobH> | 
  software raid rebuilding on singer | 
  [production] | 
            
  | 13:45 | 
  <RobH> | 
  singer back up, racktables, outreachcivi and other services back online | 
  [production] | 
            
  | 13:36 | 
  <cmjohnson1> | 
  shutdown singer to replace hard drive | 
  [production] | 
            
  | 13:33 | 
  <mark> | 
  Fixed puppet on the karmic search servers (search13..20) | 
  [production] | 
            
  | 13:27 | 
  <mark> | 
  Imported puppet 2.7.6+2.7.7rc2-1wm1~karmic1 packages into the karmic-wikimedia APT repository | 
  [production] | 
            
  | 13:26 | 
  <RobH> | 
  singer coming down for hard disk replacement.  racktables, outreachcivi, and other services will be offline for a short time. | 
  [production] | 
            
  | 12:37 | 
  <mutante> | 
  magnesium - rebooted to finish upgrade, started swift | 
  [production] | 
            
  | 12:22 | 
  <mutante> | 
  torrus threw 500 errors - fixed it by following "deadlock problem" instructions on wikitech | 
  [production] | 
            
  | 11:21 | 
  <mutante> | 
  copper - rebooted to finish upgraded, started swift | 
  [production] | 
            
  | 11:17 | 
  <mark> | 
  Imported puppet 2.7.6+2.7.7rc2-1wm1~hardy1 packages into the hardy-wikimedia APT repository | 
  [production] | 
            
  | 11:16 | 
  <mutante> | 
  copper - installing new kernel and apt,apache2,bind9-host,.. upgrades | 
  [production] | 
            
  | 10:47 | 
  <mutante> | 
  zinc - back up, running swift-init all start | 
  [production] | 
            
  | 10:45 | 
  <mutante> | 
  zinc - installed new kernel,rebooting.. | 
  [production] | 
            
  | 10:02 | 
  <mutante> | 
  done, bugzilla back up | 
  [production] | 
            
  | 09:56 | 
  <mutante> | 
  installing apache2,php-wikidiff2,kernel,.. upgrades on kaulen | 
  [production] | 
            
  | 09:50 | 
  <tstarling> | 
  synchronized wmf-config/InitialiseSettings.php  | 
  [production] | 
            
  | 09:43 | 
  <tstarling> | 
  synchronized wmf-config/InitialiseSettings.php  | 
  [production] | 
            
  | 02:07 | 
  <tstarling> | 
  ran sync-common-all  | 
  [production] | 
            
  | 01:57 | 
  <LocalisationUpdate> | 
  failed | 
  [production] | 
            
  | 01:21 | 
  <binasher> | 
  added prod eqiad bits caches to dsh varnish group | 
  [production] | 
            
  | 01:17 | 
  <Tim> | 
  running sync-common on srv260, srv280, srv284 | 
  [production] | 
            
  | 01:16 | 
  <Tim> | 
  rebooting srv284, ssh not working | 
  [production] | 
            
  | 01:13 | 
  <binasher> | 
  deployed varnish changes for bits+mobile - serve slightly stale content if backend down + retry backend 503's 4 times | 
  [production] | 
            
  | 01:05 | 
  <Tim> | 
  rebooting srv162, ssh not working | 
  [production] | 
            
  | 01:01 | 
  <Tim> | 
  rebooting srv280, no working ssh | 
  [production] | 
            
  | 00:58 | 
  <Tim> | 
  rebooting srv260, hangs on ssh connections | 
  [production] | 
            
  | 00:43 | 
  <tstarling> | 
  synchronized php-1.18/extensions/VipsScaler/SpecialVipsTest.php  | 
  [production] | 
            
  | 00:42 | 
  <tstarling> | 
  synchronized php-1.18/extensions/VipsScaler/SpecialVipsTest.php  | 
  [production] | 
            
  | 00:39 | 
  <tstarling> | 
  synchronized php-1.18/extensions/VipsScaler/VipsTest.php  | 
  [production] | 
            
  | 00:28 | 
  <tstarling> | 
  synchronized php-1.18/includes/StreamFile.php  'debugging hack' | 
  [production] | 
            
  | 00:21 | 
  <tstarling> | 
  synchronized php-1.18/extensions/VipsScaler/SpecialVipsTest.php  | 
  [production] | 
            
  | 00:19 | 
  <tstarling> | 
  synchronized php-1.18/extensions/VipsScaler/VipsScaler_body.php  | 
  [production] | 
            
  | 00:10 | 
  <tstarling> | 
  synchronized php-1.18/extensions/VipsScaler/SpecialVipsTest.php  | 
  [production] | 
            
  | 00:09 | 
  <tstarling> | 
  synchronized php-1.18/extensions/VipsScaler/VipsTest.php  | 
  [production] |