| 
      
        2014-12-01
      
      §
     | 
  
    
  | 04:46 | 
  <tstarling> | 
  Synchronized php-1.25wmf10/includes/parser/MWTidy.php: change previously pulled but scap was apparently not run (duration: 00m 05s) | 
  [production] | 
            
  | 04:44 | 
  <tstarling> | 
  Synchronized php-1.25wmf9/includes/parser/MWTidy.php: change previously pulled but scap was apparently not run (duration: 00m 06s) | 
  [production] | 
            
  | 03:34 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at Mon Dec  1 03:34:57 UTC 2014 (duration 34m 56s) | 
  [production] | 
            
  | 02:17 | 
  <LocalisationUpdate> | 
  completed (1.25wmf10) at 2014-12-01 02:17:56+00:00 | 
  [production] | 
            
  | 02:17 | 
  <l10nupdate> | 
  Synchronized php-1.25wmf10/cache/l10n: (no message) (duration: 00m 03s) | 
  [production] | 
            
  | 02:10 | 
  <LocalisationUpdate> | 
  completed (1.25wmf9) at 2014-12-01 02:10:33+00:00 | 
  [production] | 
            
  | 02:10 | 
  <l10nupdate> | 
  Synchronized php-1.25wmf9/cache/l10n: (no message) (duration: 00m 01s) | 
  [production] | 
            
  | 00:51 | 
  <tstarling> | 
  Synchronized wmf-config/StartProfiler.php: (no message) (duration: 00m 05s) | 
  [production] | 
            
  
    | 
      
        2014-11-30
      
      §
     | 
  
    
  | 23:17 | 
  <qchris> | 
  Updated EventLogging to 19c23698bc03694017d764af33307d6f035fc224 (on 22:51) and restarted it | 
  [production] | 
            
  | 20:51 | 
  <qchris> | 
  restarted eventlogging mysql-m2-master consumer. It seems it could no longer write to the database. | 
  [production] | 
            
  | 19:17 | 
  <Krinkle> | 
  Disabling and relauching Gearman connection from Jenkins. | 
  [production] | 
            
  | 10:28 | 
  <oblivian> | 
  Synchronized wmf-config/jobqueue-eqiad.php: reverting to rdb1001 (duration: 00m 05s) | 
  [production] | 
            
  | 10:13 | 
  <mark> | 
  Rebooted asw-c4-eqiad | 
  [production] | 
            
  | 09:13 | 
  <_joe_> | 
  jobsqueues work again | 
  [production] | 
            
  | 09:08 | 
  <oblivian> | 
  Synchronized wmf-config/jobqueue-eqiad.php: changing the aggregator address as well (duration: 00m 05s) | 
  [production] | 
            
  | 07:27 | 
  <_joe_> | 
  restarted the jobrunner service on all jobrunners | 
  [production] | 
            
  | 07:15 | 
  <oblivian> | 
  Synchronized wmf-config/jobqueue-eqiad.php: (no message) (duration: 00m 05s) | 
  [production] | 
            
  | 05:50 | 
  <ori> | 
  3:50 UTC: switch asw-c-eqiad lost connectivity with cabinet C4. Impact: phabricator down; gap in web request logs and some perf monitoring. Job queue and Recent Changes stream OK b/c redundant servers are up. | 
  [production] | 
            
  | 03:41 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at Sun Nov 30 03:41:03 UTC 2014 (duration 41m 2s) | 
  [production] | 
            
  | 02:18 | 
  <LocalisationUpdate> | 
  completed (1.25wmf10) at 2014-11-30 02:18:22+00:00 | 
  [production] | 
            
  | 02:18 | 
  <l10nupdate> | 
  Synchronized php-1.25wmf10/cache/l10n: (no message) (duration: 00m 02s) | 
  [production] | 
            
  | 02:11 | 
  <LocalisationUpdate> | 
  completed (1.25wmf9) at 2014-11-30 02:11:01+00:00 | 
  [production] | 
            
  | 02:11 | 
  <l10nupdate> | 
  Synchronized php-1.25wmf9/cache/l10n: (no message) (duration: 00m 01s) | 
  [production] | 
            
  
    | 
      
        2014-11-27
      
      §
     | 
  
    
  | 17:55 | 
  <_joe_> | 
  restarted hhvm on mw1224, the alarm may have been lost in the puppet failure shower earlier | 
  [production] | 
            
  | 17:24 | 
  <godog> | 
  removed /var/lib/carbon/whisper/archived/jenkins from tungsten | 
  [production] | 
            
  | 17:09 | 
  <godog> | 
  upload txstatsd 0.7.0~bzr30-0ubuntu0+14 to precise-wikimedia on carbon | 
  [production] | 
            
  | 16:49 | 
  <godog> | 
  upload missing txstatsd 1.0.0-1 _source package_ to carbon  | 
  [production] | 
            
  | 16:48 | 
  <godog> | 
  upload missing txstatsd 1.0.0-1 to carbon | 
  [production] | 
            
  | 15:48 | 
  <hoo> | 
  Synchronized php-1.25wmf10/extensions/Wikidata/: Fixing a data model bug + enable Statements on Properties for testwikidata (duration: 00m 12s) | 
  [production] | 
            
  | 15:34 | 
  <hoo> | 
  Synchronized wmf-config/Wikibase.php: Set "displayStatementsOnProperties" for wikidata/testwikidata (duration: 00m 06s) | 
  [production] | 
            
  | 14:48 | 
  <akosiaris> | 
  upgrading librsvg throughout the fleet | 
  [production] | 
            
  | 04:35 | 
  <springle> | 
  restarted squid3 on carbon, but glitches seem to be upstream | 
  [production] | 
            
  | 04:23 | 
  <LocalisationUpdate> | 
  ResourceLoader cache refresh completed at Thu Nov 27 04:23:45 UTC 2014 (duration 23m 44s) | 
  [production] | 
            
  | 03:45 | 
  <springle> | 
  puppet failures everywhere; transient apt timeout | 
  [production] | 
            
  | 02:33 | 
  <LocalisationUpdate> | 
  completed (1.25wmf10) at 2014-11-27 02:33:36+00:00 | 
  [production] | 
            
  | 02:33 | 
  <l10nupdate> | 
  Synchronized php-1.25wmf10/cache/l10n: (no message) (duration: 00m 01s) | 
  [production] | 
            
  | 02:20 | 
  <LocalisationUpdate> | 
  completed (1.25wmf9) at 2014-11-27 02:20:52+00:00 | 
  [production] |