| 
      
        2013-09-02
      
      §
     | 
  
    
  | 16:00 | 
  <reedy> | 
  synchronized wmf-config/InitialiseSettings.php | 
  [production] | 
            
  | 14:35 | 
  <hashar> | 
  Jenkins: restoring mediawiki-core jobs to previous state | 
  [production] | 
            
  | 14:31 | 
  <hashar> | 
  broken Jenkins whenever a huge repository is being used (i.e. mediawiki/core or operations/puppet ) | 
  [production] | 
            
  | 13:41 | 
  <akosiaris> | 
  aptitude hold puppet puppet-common puppetmaster puppetmaster-common puppetmaster-passenger on stafford till we solve the problem with CPU spiking present in puppet 2.7.11-1ubuntu2.4 | 
  [production] | 
            
  | 13:35 | 
  <springle> | 
  synchronized wmf-config/db-eqiad.php  'reutrn db1027 to the pool' | 
  [production] | 
            
  | 06:20 | 
  <springle> | 
  granted process/show/usage to watchdog on db masters | 
  [production] | 
            
  | 04:33 | 
  <springle> | 
  granted process/show/usage to watchdog on db masters | 
  [production] | 
            
  | 02:25 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at Mon Sep  2 02:25:02 UTC 2013 | 
  [production] | 
            
  | 02:13 | 
  <LocalisationUpdate> | 
  completed (1.22wmf15) at Mon Sep  2 02:13:50 UTC 2013 | 
  [production] | 
            
  | 02:07 | 
  <LocalisationUpdate> | 
  completed (1.22wmf14) at Mon Sep  2 02:07:46 UTC 2013 | 
  [production] | 
            
  | 00:49 | 
  <faidon> | 
  synchronized wmf-config/InitialiseSettings.php  'new wikivoyage logo' | 
  [production] | 
            
  | 00:48 | 
  <faidon> | 
  synchronized docroot/bits/favicon/wikivoyage.ico  'new wikivoyage favicon' | 
  [production] | 
            
  
    | 
      
        2013-09-01
      
      §
     | 
  
    
  | 23:57 | 
  <springle> | 
  running pt-table-sync over db1027 wikis | 
  [production] | 
            
  | 21:29 | 
  <springle> | 
  restarted db1027 mysqld. leaving port 3306 blocked while replication recovers | 
  [production] | 
            
  | 20:51 | 
  <paravoid> | 
  powercycled db1027, BUG: soft lockup - CPU#14 stuck for 22s! [xfsaild/dm-0:933] | 
  [production] | 
            
  | 20:29 | 
  <paravoid> | 
  restarting gerrit | 
  [production] | 
            
  | 17:56 | 
  <paravoid> | 
  killed snapshot1004 workers, flooding dberror with wikiadmin auth failures + WikiExporter::dumpFrom slow queries (10' ago) | 
  [production] | 
            
  | 17:52 | 
  <faidon> | 
  cleared profiling data | 
  [production] | 
            
  | 17:04 | 
  <maxsem> | 
  synchronized /php-1.22wmf14/includes/api/ApiQueryImageInfo.php  'Not the cause, restoring back' | 
  [production] | 
            
  | 16:53 | 
  <maxsem> | 
  synchronized /php-1.22wmf14/includes/api/ApiQueryImageInfo.php  'Trying to disable this...' | 
  [production] | 
            
  | 02:07 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at Sun Sep  1 02:07:18 UTC 2013 | 
  [production] | 
            
  | 02:02 | 
  <LocalisationUpdate> | 
  completed (1.22wmf15) at Sun Sep  1 02:02:30 UTC 2013 | 
  [production] | 
            
  | 02:01 | 
  <LocalisationUpdate> | 
  completed (1.22wmf14) at Sun Sep  1 02:01:46 UTC 2013 | 
  [production] | 
            
  
    | 
      
        2013-08-30
      
      §
     | 
  
    
  | 22:20 | 
  <mutante> | 
  after re-pooling search1015 a while ago, now temp. disable search1016 while indices are rebuilt for new wikis | 
  [production] | 
            
  | 21:48 | 
  <mutante> | 
  on sodium:  Class[Backup::Host] is already defined ... | 
  [production] | 
            
  | 21:45 | 
  <mutante> | 
  temp. disabling search1015 in pybal  | 
  [production] | 
            
  | 21:27 | 
  <mutante> | 
  dumping/importing viwikivoyage and tyvwiki lucene search indices | 
  [production] | 
            
  | 21:23 | 
  <mutante> | 
  restarting lucene search indexers on searchidx2 and searchidx1001 | 
  [production] | 
            
  | 20:22 | 
  <csteipp> | 
  synchronized php-1.22wmf15/extensions/LiquidThreads  'bug53320' | 
  [production] | 
            
  | 18:00 | 
  <mutante> | 
  powercycling sq36 | 
  [production] | 
            
  | 15:25 | 
  <hashar> | 
  Jenkins: removing labels hasSlaveScripts hasContintPackages productionSlaves from lanthanum slaves, it caught some jobs | 
  [production] | 
            
  | 14:37 | 
  <hashar> | 
  jenkins: lanthanum slave now has 8 executors and labels hasSlaveScripts hasContintPackages productionSlaves | 
  [production] | 
            
  | 10:47 | 
  <mark> | 
  reverted | 
  [production] | 
            
  | 10:34 | 
  <mark> | 
  Moving esams text squid backend traffic from pmtpa to eqiad | 
  [production] | 
            
  | 08:46 | 
  <hashar> | 
  restarting Jenkins for plugins upgrade | 
  [production] | 
            
  | 05:03 | 
  <olivneh> | 
  synchronized wmf-config/CommonSettings.php  'Make Flickr API URL protocol-relative for compatibility with HTTPS' | 
  [production] | 
            
  | 02:53 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at Fri Aug 30 02:53:46 UTC 2013 | 
  [production] | 
            
  | 02:38 | 
  <LocalisationUpdate> | 
  completed (1.22wmf15) at Fri Aug 30 02:38:00 UTC 2013 | 
  [production] | 
            
  | 02:18 | 
  <LocalisationUpdate> | 
  completed (1.22wmf14) at Fri Aug 30 02:18:27 UTC 2013 | 
  [production] | 
            
  | 00:18 | 
  <Ryan_Lane> | 
  rebooting sq36 | 
  [production] | 
            
  | 00:17 | 
  <Ryan_Lane> | 
  switching esams from using pmtpa to using eqiad for carp | 
  [production] |