| 2014-11-26
      
      § | 
    
  | 20:57 | <bd808> | All three elasticsaerch nodes in the logstash clsuter think logstash1003 is master but ogstash-2014.11.26 is not allocated on any node | [production] | 
            
  | 20:56 | <ejegg> | enabled queue consumers | [production] | 
            
  | 20:51 | <cscott> | updated OCG to version 7d8f2b8bd496464041e3ef9c092732457cc8f7ef (did not restart ocg) | [production] | 
            
  | 20:50 | <reedy> | Synchronized php-1.25wmf10: (no message) (duration: 00m 47s) | [production] | 
            
  | 20:30 | <reedy> | Purged l10n cache for 1.25wmf6 | [production] | 
            
  | 20:29 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.25wmf10 | [production] | 
            
  | 20:28 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.25wmf9 | [production] | 
            
  | 20:24 | <reedy> | Finished scap: testwiki to 1.25wmf10 and build l10n cache (duration: 49m 03s) | [production] | 
            
  | 20:14 | <ejegg> | updated crm from e13cae8c418d29ef444899e0a70bbe03f4b7079d to d0a51250d2bdbf3c818ec0486af284691c7a61ff | [production] | 
            
  | 20:13 | <ejegg> | disabling queue consumers | [production] | 
            
  | 19:35 | <reedy> | Started scap: testwiki to 1.25wmf10 and build l10n cache | [production] | 
            
  | 18:18 | <ebernhardson> | Synchronized wmf-config/InitialiseSettings.php: Whitelist converted lqt pages on officewiki (duration: 00m 07s) | [production] | 
            
  | 16:54 | <gwicke> | Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 05s) | [production] | 
            
  | 16:34 | <Krinkle> | Changed Jenkins default language from "en_US" to "en" ("Ignore browser settings" was already enabled). Not sure why, but it's back to English now. | [production] | 
            
  | 16:16 | <marktraceur> | Synchronized wmf-config/: [SWAT] [config] 174793 Enable VisualEditor as a Beta Feature on most remaining wikis (duration: 00m 06s) | [production] | 
            
  | 16:13 | <Krinkle> | Jenkins is displaying everything in French (both logged-in/logged-out users alike) | [production] | 
            
  | 16:11 | <marktraceur> | Synchronized php-1.25wmf9/extensions/Flow/: [SWAT] [wmf9] 175941 "Provide user to local LQT api calls" for officewiki. (duration: 00m 08s) | [production] | 
            
  | 12:25 | <godog> | stopped ocg on ocg1* | [production] | 
            
  | 12:18 | <godog> | restarting ocg on ocg1001 | [production] | 
            
  | 12:00 | <godog> | removing pdf files older than 14d from ocg100* | [production] | 
            
  | 11:57 | <godog> | removing pdf files older than 14d from ocg1001 | [production] | 
            
  | 06:48 | <tstarling> | Synchronized w/oauth-headers.php: (no message) (duration: 00m 05s) | [production] | 
            
  | 06:43 | <tstarling> | Synchronized w/oauth-headers.php: (no message) (duration: 00m 06s) | [production] | 
            
  | 06:40 | <tstarling> | Synchronized live-1.5/oauth-headers.php: (no message) (duration: 00m 05s) | [production] | 
            
  | 06:34 | <tstarling> | Synchronized php-1.25wmf9/extensions/OAuth/lib/OAuth.php: (no message) (duration: 00m 05s) | [production] | 
            
  | 06:09 | <tstarling> | Synchronized php-1.25wmf9/extensions/OAuth/lib/OAuth.php: (no message) (duration: 00m 06s) | [production] | 
            
  | 06:07 | <tstarling> | Synchronized php-1.25wmf9/extensions/OAuth/lib/OAuth.php: (no message) (duration: 00m 06s) | [production] | 
            
  | 05:15 | <tstarling> | Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 06s) | [production] | 
            
  | 04:24 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Wed Nov 26 04:24:14 UTC 2014 (duration 24m 13s) | [production] | 
            
  | 02:30 | <LocalisationUpdate> | completed (1.25wmf9) at 2014-11-26 02:30:20+00:00 | [production] | 
            
  | 02:30 | <l10nupdate> | Synchronized php-1.25wmf9/cache/l10n: (no message) (duration: 00m 01s) | [production] | 
            
  | 02:18 | <LocalisationUpdate> | completed (1.25wmf8) at 2014-11-26 02:18:29+00:00 | [production] | 
            
  | 02:18 | <l10nupdate> | Synchronized php-1.25wmf8/cache/l10n: (no message) (duration: 00m 03s) | [production] | 
            
  | 01:23 | <bd808> | restarted logstash on logstash1001; no events from log2udp relay being recorded | [production] | 
            
  | 00:48 | <aaron> | Synchronized wmf-config/StartProfiler.php: Remove obsolete profiling settings (duration: 00m 06s) | [production] | 
            
  | 00:33 | <springle> | power down db2033 for reassignement to codfw frack | [production] | 
            
  | 00:04 | <qchris> | restarted eventlogging mysql-m2-master consumer. It seems it could no longer write to the database. | [production] | 
            
  
    | 2014-11-25
      
      § | 
    
  | 23:06 | <Tim> | on osmium: removing stale static pcre and zip libraries in /usr/local , installed by hhvm | [production] | 
            
  | 22:50 | <ebernhardson> | Finished scap: Bump Echo and Flow in 1.25wmf9 for officewiki deployment (duration: 30m 17s) | [production] | 
            
  | 22:20 | <ebernhardson> | Started scap: Bump Echo and Flow in 1.25wmf9 for officewiki deployment | [production] | 
            
  | 22:20 | <ebernhardson> | Synchronized php-1.25wmf9/extensions/Echo/: Bump Echo in 1.25wmf9 (duration: 00m 08s) | [production] | 
            
  | 21:55 | <ejegg> | Synchronized wmf-config/CommonSettings.php: Turn CN client-side banner choice back on everywhere (duration: 00m 05s) | [production] | 
            
  | 21:39 | <ejegg> | Synchronized php-1.25wmf8/extensions/CentralNotice/: One more CentralNotice fix to get out ahead of the winter rush - wmf8 (duration: 00m 07s) | [production] | 
            
  | 21:22 | <ejegg> | Synchronized wmf-config/CommonSettings.php: Turn CN client-side banner choice back on for selected wmf9 wikis (duration: 00m 05s) | [production] | 
            
  | 21:15 | <ejegg> | Synchronized php-1.25wmf9/extensions/CentralNotice/: One more CentralNotice fix to get out ahead of the winter rush (duration: 00m 05s) | [production] | 
            
  | 20:53 | <Nemo_bis> | 100 % packet loss between esams and r1fra1.core.init7.net | [production] | 
            
  | 20:16 | <reedy> | Synchronized php-1.25wmf9/extensions/Wikidata: Ic070ce0beb142e100490940fddaa0bd36b8a50be (duration: 00m 14s) | [production] | 
            
  | 20:09 | <reedy> | Synchronized php-1.25wmf8/extensions/Wikidata: Ensure my sanity (duration: 00m 13s) | [production] | 
            
  | 19:49 | <bd808> | restarted elasticsearch on logstah1002 after OOM | [production] | 
            
  | 19:38 | <reedy> | Synchronized wmf-config/: Config updates (duration: 00m 06s) | [production] |