| 2014-05-17
      
      § | 
    
  | 03:08 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Sat May 17 03:07:12 UTC 2014 (duration 7m 11s) | [production] | 
            
  | 02:25 | <LocalisationUpdate> | completed (1.24wmf5) at 2014-05-17 02:24:46+00:00 | [production] | 
            
  | 02:15 | <LocalisationUpdate> | completed (1.24wmf4) at 2014-05-17 02:14:13+00:00 | [production] | 
            
  | 01:58 | <mutante> | powercycling labsdb1003 | [production] | 
            
  | 00:23 | <ori> | varnishadm on cp1056 confirms that varnish recognizes mw1151 as "sick" | [production] | 
            
  | 00:20 | <ori> | stopping apache and disabling puppet on mw1151 so that varnish stops forwarding reqs to it | [production] | 
            
  | 00:16 | <Krinkle> | On mw1151, Gadget::loadStructuredList() returns false, memcached has no value for 'enwiki:gadgets-definition:7' and is unable to store it. | [production] | 
            
  | 00:11 | <ori> | timo identified weird RL responses as all originating in mw1151; dmesg shows ata1 disk troubles: "failed command: READ DMA EXT", "sd 0:0:0:0: [sda]  Add. Sense: Unrecovered read error - auto reallocate failed" | [production] | 
            
  | 00:04 | <krinkle> | synchronized php-1.24wmf4/includes/resourceloader/ResourceLoader.php  'I718fcf23d' | [production] | 
            
  
    | 2014-05-16
      
      § | 
    
  | 23:07 | <awight> | updated crm from 0b8aa8aa046935b6cfc67c10ebe10396d5e42745 to 65a433b5564f42c3aa4f310cd4bb938ae70f841d | [production] | 
            
  | 22:56 | <aaron> | synchronized php-1.24wmf4/includes/db/Database.php  '182e42c173b9ab0c2bc5d753879a000b1ff39e77' | [production] | 
            
  | 22:54 | <aaron> | synchronized php-1.24wmf5/includes/db/Database.php  '8829ffc72d3332d348a1a2e58d525e54e126bad5' | [production] | 
            
  | 22:17 | <awight> | tools updated from 85bb7293d83517086e3609f03365aecde9f58c71 to ee31fc94b17c11a48ddac19aabfcdaab69fd2f72 | [production] | 
            
  | 21:37 | <ori> | synchronized php-1.24wmf4/extensions/MultimediaViewer  'Update MultimediaViewer for I0df067a61: Add sampling to unsampled event logging' | [production] | 
            
  | 21:33 | <ori> | synchronized php-1.24wmf5/extensions/MultimediaViewer  'Update MultimediaViewer for I0df067a61: Add sampling to unsampled event logging' | [production] | 
            
  | 21:10 | <mwalker> | updating fundraising smashpig from 2fdf982b20f1cbeaf9f57af64ef21b5b69a36f6e to f64f79f13cf4ab560d0bb5bd69690c827a821629 | [production] | 
            
  | 20:41 | <awight> | update crm from 243641de631b712c4a29ca1f3618771b78dadeae to 0b8aa8aa046935b6cfc67c10ebe10396d5e42745 | [production] | 
            
  | 18:43 | <awight> | update tools from a40c0caa18a0efd93bc5d3f7f68386fbc36bf1fa to 85bb7293d83517086e3609f03365aecde9f58c71 | [production] | 
            
  | 18:12 | <ori> | synchronized wmf-config/InitialiseSettings.php  'I3c453b0949f4e: Tweak MediaViewer sampling settings' | [production] | 
            
  | 18:07 | <ori> | synchronized wmf-config  'Ia43821231: Add sampling control setting for MediaViewer event' | [production] | 
            
  | 18:04 | <ori> | updated /a/common to {{Gerrit|Ia43821231}}: Add sampling control setting for MediaViewer event logging | [production] | 
            
  | 17:47 | <demon> | synchronized wmf-config/CommonSettings.php  'GeoData to Elastic for all wikivoyages' | [production] | 
            
  | 17:11 | <mwalker> | updating staging payments servers as well from 5e24b953dcff5305099e152139e6e93daba8aeec to d99177518b741e7fe18ffda86c83f93c72e164a6 | [production] | 
            
  | 17:10 | <mwalker> | and updated to 1.22.6 | [production] | 
            
  | 17:10 | <mwalker> | moved fundraising wiki from 6a1d4983319038edeb88dc34a1c220ecaec1cbde to d99177518b741e7fe18ffda86c83f93c72e164a6 -- including json i18n changes | [production] | 
            
  | 16:55 | <manybubbles> | "in place" reindexing (for cirrus) all the wikipedias after the deploy train hit them yesterday | [production] | 
            
  | 16:53 | <demon> | synchronized wmf-config/CommonSettings.php  'Removing old WikiEditor settings' | [production] | 
            
  | 16:44 | <RobH> | partial zirconium downtime | [production] | 
            
  | 16:44 | <RobH> | i logged into zirconium, but it had recovered by the time I checked it. | [production] | 
            
  | 16:33 | <mwalker> | updated fundraising civicrm from 7a23465e620211739421cce3ad57c62597eb8cc3 to 75c1a50b8aa7e7b6f218d7c420932a8fc53a0a34 for an exchange rates fix | [production] | 
            
  | 16:26 | <qchris> | updated gerrit's hooks-bugzilla plugin to version 2.8.1.2 to allow talking to bugzilla-4.4.4 | [production] | 
            
  | 13:54 | <springle> | synchronized wmf-config/db-eqiad.php  'repool db1056 in s4, warm up' | [production] | 
            
  | 13:17 | <springle> | synchronized wmf-config/db-eqiad.php  'raise db1070 and db1071 to normal load' | [production] | 
            
  | 10:14 | <springle> | xtrabackup clone db1049 to db1056 | [production] | 
            
  | 10:13 | <springle> | synchronized wmf-config/db-eqiad.php  'reduce db1049 load while cloning' | [production] | 
            
  | 09:40 | <springle> | synchronized wmf-config/db-eqiad.php  'pool db1070 and db1071 in s1, warm up' | [production] | 
            
  | 06:20 | <ori> | synchronized php-1.24wmf4/maintenance/compareParserCache.php  'Ica69a3ef2: Added a script to compare current parser output to cache (no impact on prod; syncing for consistency)' | [production] | 
            
  | 03:55 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Fri May 16 03:54:03 UTC 2014 (duration 54m 2s) | [production] | 
            
  | 03:09 | <LocalisationUpdate> | completed (1.24wmf5) at 2014-05-16 03:08:04+00:00 | [production] | 
            
  | 02:39 | <LocalisationUpdate> | completed (1.24wmf4) at 2014-05-16 02:38:41+00:00 | [production] | 
            
  | 02:07 | <springle> | xtrabackup db1070 to db1071 | [production] | 
            
  | 01:02 | <ori> | synchronized php-1.24wmf5/includes/parser  'I12a60b5cc: Revert "Declare visibility on class properties of includes/parser/"' | [production] | 
            
  | 00:54 | <hoo> | rebuildItemsPerSite finished running for Wikidata (after about 30h). | [production] | 
            
  | 00:32 | <hoo> | manually ran rebuildEntityPerPage for Wikidata to fix 2 broken records | [production] | 
            
  | 00:07 | <maxsem> | synchronized wmf-config  'https://gerrit.wikimedia.org/r/#/c/131762/' | [production] | 
            
  | 00:00 | <ori> | synchronized wmf-config/squid.php  'Id188979c1: Use whole subnets in squid.php list for XFF acceptance' | [production] |