| 2014-01-05
      
      § | 
    
  | 23:59 | <reedy> | synchronized wmf-config/InitialiseSettings.php | [production] | 
            
  | 20:18 | <aaron> | finished scap: timing test (beta) (duration: 02m 49s) | [production] | 
            
  | 20:15 | <aaron> | started scap: timing test (beta) | [production] | 
            
  | 20:13 | <aaron> | finished scap: timing test (duration: 06m 51s) | [production] | 
            
  | 20:11 | <paravoid> | rebooting cp3011, kmem_allock deadlock | [production] | 
            
  | 20:06 | <aaron> | started scap: timing test | [production] | 
            
  | 19:26 | <bblack> | varnish mobile caches updated to -wm27 package (crash fix, http/0.9 patch removal to test effect on logging anomalies) | [production] | 
            
  | 08:21 | <mark> | Power cycled ms-be1012 | [production] | 
            
  | 02:08 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Sun Jan  5 02:08:47 UTC 2014 | [production] | 
            
  | 02:03 | <LocalisationUpdate> | completed (1.23wmf9) at Sun Jan  5 02:03:44 UTC 2014 | [production] | 
            
  | 02:02 | <LocalisationUpdate> | completed (1.23wmf8) at Sun Jan  5 02:01:57 UTC 2014 | [production] | 
            
  | 00:55 | <reedy> | updated /a/common to {{Gerrit|I3543a34aa}}: db1040 back to full steam | [production] | 
            
  
    | 2014-01-04
      
      § | 
    
  | 21:06 | <hashar> | jenkins: purged php5-mysqld package from lanthanum and gallium. It was no more installed but had some conf file left in /etc/php5/conf.d/ | [production] | 
            
  | 14:26 | <springle> | synchronized wmf-config/db-eqiad.php  'db1040 back to full steam' | [production] | 
            
  | 08:54 | <springle> | synchronized wmf-config/db-eqiad.php  'reduce db1026 load while reindexing wikidatawiki.wb_terms' | [production] | 
            
  | 07:42 | <springle> | synchronized wmf-config/db-eqiad.php  'reduce db1040 load while adding indexes' | [production] | 
            
  | 04:13 | <springle> | schema change, ad-hoc, additional indexes on recentchanges & wb_terms for recent slow queries | [production] | 
            
  | 03:45 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Sat Jan  4 03:45:39 UTC 2014 | [production] | 
            
  | 03:13 | <LocalisationUpdate> | completed (1.23wmf9) at Sat Jan  4 03:13:45 UTC 2014 | [production] | 
            
  | 02:38 | <LocalisationUpdate> | completed (1.23wmf8) at Sat Jan  4 02:38:29 UTC 2014 | [production] | 
            
  | 01:33 | <hashar> | jenkins fixed up mwext-VisualEditor-qunit job, its configuration got reverted to some old/incorrect version when I downgraded the git plugin.  Retriggered all changes | [production] | 
            
  | 00:02 | <ori> | synchronized php-1.23wmf9/skins/common  'Fix SVG MIME-type detection by reverting a9b855eea52ba3 (part 2/2)' | [production] | 
            
  | 00:01 | <ori> | synchronized php-1.23wmf9/skins/vector/images  'Fix SVG MIME-type detection by reverting a9b855eea52ba3' | [production] | 
            
  
    | 2014-01-03
      
      § | 
    
  | 22:07 | <jeremyb> | del | [production] | 
            
  | 22:03 | <awight> | tools updated from b54854fcdd7a272ca2af8475affa033517456d07 to 0df36fd9bc55b24ffad49c0dbd0db67d67a2b1f7 | [production] | 
            
  | 21:53 | <mwalker> | updating fundraising-tools on al from b54854fcdd7a272ca2af8475affa033517456d07 (for statistics generation in json format) | [production] | 
            
  | 19:33 | <ori> | Some migration pains while moving graphite from professor to tungsten; expect graphite & gdash flakiness | [production] | 
            
  | 19:01 | <hashar> | apparently jenkins is back up and happy. Had to revert the git plugin to a previous version ... | [production] | 
            
  | 18:42 | <aaron> | started scap | [production] | 
            
  | 18:36 | <aaron> | started scap: active Testing timing | [production] | 
            
  | 18:11 | <hashar> | Jenkins downgrading git plugin client to 1.4.6 and restarting jenkins | [production] | 
            
  | 17:52 | <hashar> | Jenkins downgrading git plugin from 2.0 to 1.5  , we might be hit by https://issues.jenkins-ci.org/browse/JENKINS-21057 | [production] | 
            
  | 16:15 | <reedy> | synchronized php-1.23wmf8/extensions/ | [production] | 
            
  | 16:14 | <reedy> | synchronized php-1.23wmf9/extensions/ | [production] | 
            
  | 14:53 | <hashar> | jenkins restarted | [production] | 
            
  | 14:43 | <akosiaris> | zapped /vol/root export from nas-1001-a | [production] | 
            
  | 14:23 | <hashar> | restarting Jenkins , some git plugin are misbehaving | [production] | 
            
  | 13:35 | <springle> | killed msnbot spike on s2 | [production] | 
            
  | 12:43 | <Krinkle> | Jenkins still unable to use integration-slave01 (restarted the node in labs, and disconnected/re-launched slave agent afterwards, too; no effect) | [production] | 
            
  | 12:14 | <Krinkle> | Jenkins is showing failures for tests executed on integration-slave01 (remote file system failing) | [production] | 
            
  | 09:50 | <apergos> | restarted parsoid on wtp1010 and 1023, several gigs of logs full of "Error: Can't set headers after they are sent." from ServerResponse.OutgoingMessage.setHeader | [production] | 
            
  | 07:37 | <apergos> | streber from mgmt console reports eth0 link down (hence it appears down to icinga and ganglia) | [production] | 
            
  | 07:06 | <mark> | Disabled OSPF3 on csw2-knams:xe-1/1/0.0 | [production] | 
            
  | 04:27 | <Ryan_Lane> | added python-keystone-redis to apt repo | [production] | 
            
  | 03:24 | <springle> | start schema changes for bug 59236, indexing only, ipblocks ipb_parent_block_id | [production] |