| 
      
        2014-08-15
      
      §
     | 
  
    
  | 20:59 | 
  <kaldari> | 
  Synchronized php-1.24wmf16/extensions/MobileFrontend/less: fixing iOS search bug (duration: 00m 05s) | 
  [production] | 
            
  | 17:58 | 
  <aude> | 
  Synchronized wmf-config/Wikibase.php: Enable redirects on test.wikidata (duration: 00m 07s) | 
  [production] | 
            
  | 15:53 | 
  <aude> | 
  Synchronized php-1.24wmf17/extensions/Wikidata: Update test.wikidata (duration: 00m 07s) | 
  [production] | 
            
  | 15:50 | 
  <aude> | 
  Synchronized php-1.24wmf17/extensions/Wikidata: Fix database error and snak value display on test wikidata (duration: 00m 09s) | 
  [production] | 
            
  | 15:00 | 
  <ori> | 
  re-enabled puppet on mw1017 | 
  [production] | 
            
  | 13:33 | 
  <ori> | 
  disabling puppet on mw1017 to test rsyslog config | 
  [production] | 
            
  | 03:51 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at Fri Aug 15 03:50:23 UTC 2014 (duration 50m 22s) | 
  [production] | 
            
  | 03:04 | 
  <LocalisationUpdate> | 
  completed (1.24wmf17) at 2014-08-15 03:03:49+00:00 | 
  [production] | 
            
  | 02:34 | 
  <LocalisationUpdate> | 
  completed (1.24wmf16) at 2014-08-15 02:33:21+00:00 | 
  [production] | 
            
  | 00:24 | 
  <ori> | 
  Finished scap: SWAT: cherry picks for TMH and Echo (duration: 14m 38s) | 
  [production] | 
            
  | 00:09 | 
  <ori> | 
  Started scap: SWAT: cherry picks for TMH and Echo | 
  [production] | 
            
  
    | 
      
        2014-08-14
      
      §
     | 
  
    
  | 23:24 | 
  <aude> | 
  Synchronized wmf-config/Wikibase.php: Bump cache epoch and add badges setting on test.wikidata (duration: 00m 32s) | 
  [production] | 
            
  | 23:14 | 
  <aude> | 
  Finished scap: Update branch for test.wikidata (duration: 16m 48s) | 
  [production] | 
            
  | 22:57 | 
  <aude> | 
  Started scap: Update branch for test.wikidata | 
  [production] | 
            
  | 22:26 | 
  <aaron> | 
  Synchronized php-1.24wmf16/includes/DefaultSettings.php: 67bf481ce1644ff194d7565107d9b8ffe11bf4b7 (duration: 00m 07s) | 
  [production] | 
            
  | 22:23 | 
  <aaron> | 
  Synchronized wmf-config/CommonSettings.php: Increased wgParsoidCacheUpdateTitlesPerJob to 12 to lower the backlog (duration: 00m 07s) | 
  [production] | 
            
  | 22:13 | 
  <aude> | 
  Started scap: Update branch for test.wikidata | 
  [production] | 
            
  | 21:49 | 
  <reedy> | 
  Synchronized php-1.24wmf17/includes/context/RequestContext.php: (no message) (duration: 00m 15s) | 
  [production] | 
            
  | 21:11 | 
  <godog> | 
  restarted hhvm on mw1053 | 
  [production] | 
            
  | 20:48 | 
  <_joe|away> | 
  stopping puppet, jobrunner on mw1053; HHVM is eating memory like godzilla  | 
  [production] | 
            
  | 19:29 | 
  <bblack> | 
  puppeting labmon1001, etc | 
  [production] | 
            
  | 18:57 | 
  <reedy> | 
  Synchronized wmf-config/: (no message) (duration: 00m 14s) | 
  [production] | 
            
  | 18:55 | 
  <reedy> | 
  Synchronized database lists: (no message) (duration: 00m 14s) | 
  [production] | 
            
  | 18:26 | 
  <mutante> | 
  stopped ircecho on neon temporarily | 
  [production] | 
            
  | 18:10 | 
  <reedy> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf17 | 
  [production] | 
            
  | 18:05 | 
  <reedy> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.24wmf16 | 
  [production] | 
            
  | 17:46 | 
  <AaronSchulz> | 
  /srv/deployment/jobrunner updated to 795baf3ca4ce8308597dd74e5242aa5bfbbe961d | 
  [production] | 
            
  | 17:39 | 
  <aaron> | 
  Synchronized rpc: 6c0ece687bb6ff3fec0ca7e80a587525ebf18a70 (duration: 00m 08s) | 
  [production] | 
            
  | 16:52 | 
  <_joe_> | 
  uploaded new hhvm package 3.3-dev+20140728+wmf4 | 
  [production] | 
            
  | 16:23 | 
  <reedy> | 
  Synchronized php-1.24wmf17/extensions/CentralAuth/: (no message) (duration: 00m 13s) | 
  [production] | 
            
  | 16:23 | 
  <reedy> | 
  Synchronized php-1.24wmf16/extensions/CentralAuth/: (no message) (duration: 00m 14s) | 
  [production] | 
            
  | 15:49 | 
  <Reedy> | 
  Running sync-common on mw1053 | 
  [production] | 
            
  | 15:49 | 
  <reedy> | 
  Finished scap: testwiki to 1.24wmf17 (duration: 33m 13s) | 
  [production] | 
            
  | 15:47 | 
  <Jeff_Green> | 
  adjust wiki-mail._domainkey DNS record to allow sending from 'wiki*@" addresses, instead of just wiki@ | 
  [production] | 
            
  | 15:23 | 
  <_joe_> | 
  powercycling mw1053, which looks like the victim of hhvm-induced ooms | 
  [production] | 
            
  | 15:15 | 
  <reedy> | 
  Started scap: testwiki to 1.24wmf17 | 
  [production] | 
            
  | 14:01 | 
  <_joe_> | 
  puppet re-enabled on the appserver | 
  [production] | 
            
  | 12:38 | 
  <_joe_> | 
  stopping puppet on appservers while deploying a delicate change. | 
  [production] | 
            
  | 09:30 | 
  <_joe_> | 
  the hhvm jobrunner is back in production, seems healthy, see https://logstash.wikimedia.org/#/dashboard/elasticsearch/hhvm_jobrunner | 
  [production] | 
            
  | 08:09 | 
  <_joe_> | 
  reactivated the jobrunner on mw1053, with promising results. Puppettization pending (in ~ 1 hour) | 
  [production] | 
            
  | 03:12 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at Thu Aug 14 03:11:33 UTC 2014 (duration 11m 32s) | 
  [production] | 
            
  | 02:31 | 
  <LocalisationUpdate> | 
  completed (1.24wmf16) at 2014-08-14 02:29:52+00:00 | 
  [production] | 
            
  | 02:17 | 
  <LocalisationUpdate> | 
  completed (1.24wmf15) at 2014-08-14 02:16:34+00:00 | 
  [production] |