| 2014-11-26
      
      § | 
    
  | 22:08 | <hashar> | investigating Zuul/Jenkins.  Jenkins potentially has a deadlock | [production] | 
            
  | 22:02 | <cscott> | updated Parsoid to version 67e2596c | [production] | 
            
  | 21:53 | <hashar> | Restarting Gearman client. I am in a meeting, will cleanup later. | [production] | 
            
  | 21:33 | <bd808> | restarted logstash on logstash1001; log2udp events not being received | [production] | 
            
  | 21:22 | <ejegg> | disabled ty sending | [production] | 
            
  | 21:15 | <hashar> | Zuul stuck, restarting Gearman client | [production] | 
            
  | 21:01 | <bd808> | restarted elasticsearch on logstash1002 for OOM | [production] | 
            
  | 21:00 | <bd808> | restarted elasticsearch on logstash1003 for OOM | [production] | 
            
  | 20:57 | <bd808> | All three elasticsaerch nodes in the logstash clsuter think logstash1003 is master but ogstash-2014.11.26 is not allocated on any node | [production] | 
            
  | 20:56 | <ejegg> | enabled queue consumers | [production] | 
            
  | 20:51 | <cscott> | updated OCG to version 7d8f2b8bd496464041e3ef9c092732457cc8f7ef (did not restart ocg) | [production] | 
            
  | 20:50 | <reedy> | Synchronized php-1.25wmf10: (no message) (duration: 00m 47s) | [production] | 
            
  | 20:30 | <reedy> | Purged l10n cache for 1.25wmf6 | [production] | 
            
  | 20:29 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf10 | [production] | 
            
  | 20:28 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.25wmf9 | [production] | 
            
  | 20:24 | <reedy> | Finished scap: testwiki to 1.25wmf10 and build l10n cache (duration: 49m 03s) | [production] | 
            
  | 20:14 | <ejegg> | updated crm from e13cae8c418d29ef444899e0a70bbe03f4b7079d to d0a51250d2bdbf3c818ec0486af284691c7a61ff | [production] | 
            
  | 20:13 | <ejegg> | disabling queue consumers | [production] | 
            
  | 19:35 | <reedy> | Started scap: testwiki to 1.25wmf10 and build l10n cache | [production] | 
            
  | 18:18 | <ebernhardson> | Synchronized wmf-config/InitialiseSettings.php: Whitelist converted lqt pages on officewiki (duration: 00m 07s) | [production] | 
            
  | 16:54 | <gwicke> | Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 05s) | [production] | 
            
  | 16:34 | <Krinkle> | Changed Jenkins default language from "en_US" to "en" ("Ignore browser settings" was already enabled). Not sure why, but it's back to English now. | [production] | 
            
  | 16:16 | <marktraceur> | Synchronized wmf-config/: [SWAT] [config] 174793 Enable VisualEditor as a Beta Feature on most remaining wikis (duration: 00m 06s) | [production] | 
            
  | 16:13 | <Krinkle> | Jenkins is displaying everything in French (both logged-in/logged-out users alike) | [production] | 
            
  | 16:11 | <marktraceur> | Synchronized php-1.25wmf9/extensions/Flow/: [SWAT] [wmf9] 175941 "Provide user to local LQT api calls" for officewiki. (duration: 00m 08s) | [production] | 
            
  | 12:25 | <godog> | stopped ocg on ocg1* | [production] | 
            
  | 12:18 | <godog> | restarting ocg on ocg1001 | [production] | 
            
  | 12:00 | <godog> | removing pdf files older than 14d from ocg100* | [production] | 
            
  | 11:57 | <godog> | removing pdf files older than 14d from ocg1001 | [production] | 
            
  | 06:48 | <tstarling> | Synchronized w/oauth-headers.php: (no message) (duration: 00m 05s) | [production] | 
            
  | 06:43 | <tstarling> | Synchronized w/oauth-headers.php: (no message) (duration: 00m 06s) | [production] | 
            
  | 06:40 | <tstarling> | Synchronized live-1.5/oauth-headers.php: (no message) (duration: 00m 05s) | [production] | 
            
  | 06:34 | <tstarling> | Synchronized php-1.25wmf9/extensions/OAuth/lib/OAuth.php: (no message) (duration: 00m 05s) | [production] | 
            
  | 06:09 | <tstarling> | Synchronized php-1.25wmf9/extensions/OAuth/lib/OAuth.php: (no message) (duration: 00m 06s) | [production] | 
            
  | 06:07 | <tstarling> | Synchronized php-1.25wmf9/extensions/OAuth/lib/OAuth.php: (no message) (duration: 00m 06s) | [production] | 
            
  | 05:15 | <tstarling> | Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 06s) | [production] | 
            
  | 04:24 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Wed Nov 26 04:24:14 UTC 2014 (duration 24m 13s) | [production] | 
            
  | 02:30 | <LocalisationUpdate> | completed (1.25wmf9) at 2014-11-26 02:30:20+00:00 | [production] | 
            
  | 02:30 | <l10nupdate> | Synchronized php-1.25wmf9/cache/l10n: (no message) (duration: 00m 01s) | [production] | 
            
  | 02:18 | <LocalisationUpdate> | completed (1.25wmf8) at 2014-11-26 02:18:29+00:00 | [production] | 
            
  | 02:18 | <l10nupdate> | Synchronized php-1.25wmf8/cache/l10n: (no message) (duration: 00m 03s) | [production] | 
            
  | 01:23 | <bd808> | restarted logstash on logstash1001; no events from log2udp relay being recorded | [production] | 
            
  | 00:48 | <aaron> | Synchronized wmf-config/StartProfiler.php: Remove obsolete profiling settings (duration: 00m 06s) | [production] | 
            
  | 00:33 | <springle> | power down db2033 for reassignement to codfw frack | [production] | 
            
  | 00:04 | <qchris> | restarted eventlogging mysql-m2-master consumer. It seems it could no longer write to the database. | [production] |