| 2013-09-01
      
      § | 
    
  | 23:57 | <springle> | running pt-table-sync over db1027 wikis | [production] | 
            
  | 21:29 | <springle> | restarted db1027 mysqld. leaving port 3306 blocked while replication recovers | [production] | 
            
  | 20:51 | <paravoid> | powercycled db1027, BUG: soft lockup - CPU#14 stuck for 22s! [xfsaild/dm-0:933] | [production] | 
            
  | 20:29 | <paravoid> | restarting gerrit | [production] | 
            
  | 17:56 | <paravoid> | killed snapshot1004 workers, flooding dberror with wikiadmin auth failures + WikiExporter::dumpFrom slow queries (10' ago) | [production] | 
            
  | 17:52 | <faidon> | cleared profiling data | [production] | 
            
  | 17:04 | <maxsem> | synchronized /php-1.22wmf14/includes/api/ApiQueryImageInfo.php  'Not the cause, restoring back' | [production] | 
            
  | 16:53 | <maxsem> | synchronized /php-1.22wmf14/includes/api/ApiQueryImageInfo.php  'Trying to disable this...' | [production] | 
            
  | 02:07 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Sun Sep  1 02:07:18 UTC 2013 | [production] | 
            
  | 02:02 | <LocalisationUpdate> | completed (1.22wmf15) at Sun Sep  1 02:02:30 UTC 2013 | [production] | 
            
  | 02:01 | <LocalisationUpdate> | completed (1.22wmf14) at Sun Sep  1 02:01:46 UTC 2013 | [production] | 
            
  
    | 2013-08-30
      
      § | 
    
  | 22:20 | <mutante> | after re-pooling search1015 a while ago, now temp. disable search1016 while indices are rebuilt for new wikis | [production] | 
            
  | 21:48 | <mutante> | on sodium:  Class[Backup::Host] is already defined ... | [production] | 
            
  | 21:45 | <mutante> | temp. disabling search1015 in pybal | [production] | 
            
  | 21:27 | <mutante> | dumping/importing viwikivoyage and tyvwiki lucene search indices | [production] | 
            
  | 21:23 | <mutante> | restarting lucene search indexers on searchidx2 and searchidx1001 | [production] | 
            
  | 20:22 | <csteipp> | synchronized php-1.22wmf15/extensions/LiquidThreads  'bug53320' | [production] | 
            
  | 18:00 | <mutante> | powercycling sq36 | [production] | 
            
  | 15:25 | <hashar> | Jenkins: removing labels hasSlaveScripts hasContintPackages productionSlaves from lanthanum slaves, it caught some jobs | [production] | 
            
  | 14:37 | <hashar> | jenkins: lanthanum slave now has 8 executors and labels hasSlaveScripts hasContintPackages productionSlaves | [production] | 
            
  | 10:47 | <mark> | reverted | [production] | 
            
  | 10:34 | <mark> | Moving esams text squid backend traffic from pmtpa to eqiad | [production] | 
            
  | 08:46 | <hashar> | restarting Jenkins for plugins upgrade | [production] | 
            
  | 05:03 | <olivneh> | synchronized wmf-config/CommonSettings.php  'Make Flickr API URL protocol-relative for compatibility with HTTPS' | [production] | 
            
  | 02:53 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Fri Aug 30 02:53:46 UTC 2013 | [production] | 
            
  | 02:38 | <LocalisationUpdate> | completed (1.22wmf15) at Fri Aug 30 02:38:00 UTC 2013 | [production] | 
            
  | 02:18 | <LocalisationUpdate> | completed (1.22wmf14) at Fri Aug 30 02:18:27 UTC 2013 | [production] | 
            
  | 00:18 | <Ryan_Lane> | rebooting sq36 | [production] | 
            
  | 00:17 | <Ryan_Lane> | switching esams from using pmtpa to using eqiad for carp | [production] | 
            
  
    | 2013-08-29
      
      § | 
    
  | 23:50 | <gwicke> | updated Parsoid to b26a506 | [production] | 
            
  | 23:36 | <Ryan_Lane> | removing sq33 from squid configuration, it was decommissioned a while ago | [production] | 
            
  | 21:55 | <mutante> | upgrading apparmor packages on iron | [production] | 
            
  | 21:31 | <mutante> | powercycled sq36 which was frozen but probably all unrelated to API outage | [production] | 
            
  | 20:57 | <olivneh> | synchronized php-1.22wmf15/extensions/NavigationTiming  'Updating NavigationTiming to master (2/2)' | [production] | 
            
  | 20:57 | <olivneh> | synchronized php-1.22wmf14/extensions/NavigationTiming  'Updating NavigationTiming to master (1/2)' | [production] | 
            
  | 20:55 | <mflaschen> | synchronized php-1.22wmf14/extensions/GuidedTour/  'Deploy GuidedTour to 1.22wmf14' | [production] | 
            
  | 20:54 | <olivneh> | synchronized php-1.22wmf15/includes/RecentChange.php  'I4570473ae' | [production] | 
            
  | 20:51 | <reedy> | synchronized php-1.22wmf14/extensions/Wikibase | [production] | 
            
  | 20:50 | <reedy> | synchronized php-1.22wmf15/extensions/Wikibase | [production] | 
            
  | 20:31 | <olivneh> | synchronized php-1.22wmf15/extensions/NavigationTiming  'Updating NavigationTiming to master (2/2)' | [production] | 
            
  | 20:31 | <olivneh> | synchronized php-1.22wmf14/extensions/NavigationTiming  'Updating NavigationTiming to master (1/2)' | [production] | 
            
  | 19:52 | <Aaron|home> | Running PurgeDeletedFiles everywhere to bring ceph/swift more in sync with each other and the DB; copy scripts already finished | [production] |