| 2014-06-13
      
      § | 
    
  | 16:26 | <Reedy> | Creating mathoid tables on all wikis | [production] | 
            
  | 16:11 | <mutante> | manutius - decom, delete salt key, puppet cert, stopped services... | [production] | 
            
  | 15:17 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Fri Jun 13 15:16:09 UTC 2014 (duration 53m 11s) | [production] | 
            
  | 14:59 | <reedy> | Synchronized wmf-config/: Disable MW_MATH_SOURCE for now (duration: 00m 15s) | [production] | 
            
  | 14:46 | <LocalisationUpdate> | completed (1.24wmf9) at 2014-06-13 14:45:40+00:00 | [production] | 
            
  | 14:36 | <LocalisationUpdate> | completed (1.24wmf8) at 2014-06-13 14:35:41+00:00 | [production] | 
            
  | 13:00 | <bblack> | moved ge-3/0/0 - 3/0/15 from public to private vlan on cs2-esams (amssq31-46) | [production] | 
            
  | 10:02 | <springle> | Synchronized wmf-config/db-eqiad.php: depool db1049 (duration: 00m 12s) | [production] | 
            
  | 09:56 | <paravoid> | deactivating eqiad<->HE, excessive packet loss/latency | [production] | 
            
  | 09:33 | <springle> | Synchronized wmf-config/db-eqiad.php: repool db1071 (duration: 00m 07s) | [production] | 
            
  | 08:10 | <springle> | Synchronized wmf-config/db-eqiad.php: repool db1070, depool db1071 (duration: 00m 12s) | [production] | 
            
  | 08:06 | <springle> | Synchronized wmf-config/db-eqiad.php: repool db1070, depool db1071 (duration: 00m 06s) | [production] | 
            
  | 07:48 | <springle> | Synchronized wmf-config/db-eqiad.php: repool db1066, depool db1070 (duration: 00m 07s) | [production] | 
            
  | 07:19 | <springle> | Synchronized wmf-config/db-eqiad.php: repool db1065, depool db1066 (duration: 00m 13s) | [production] | 
            
  | 06:51 | <springle> | Synchronized wmf-config/db-eqiad.php: repool db1062, depool db1065 (duration: 00m 09s) | [production] | 
            
  | 06:09 | <springle> | Synchronized wmf-config/db-eqiad.php: depool db1062 (duration: 00m 12s) | [production] | 
            
  | 05:58 | <springle> | Synchronized wmf-config/db-eqiad.php: repool db1051 (duration: 00m 14s) | [production] | 
            
  | 03:54 | <LocalisationUpdate> | ResourceLoader cache refresh completed at Fri Jun 13 03:53:17 UTC 2014 (duration 53m 16s) | [production] | 
            
  | 03:12 | <LocalisationUpdate> | completed (1.24wmf9) at 2014-06-13 03:11:28+00:00 | [production] | 
            
  | 02:35 | <LocalisationUpdate> | completed (1.24wmf8) at 2014-06-13 02:34:41+00:00 | [production] | 
            
  | 00:45 | <ori> | Synchronized php-1.24wmf8/extensions/Math: Reverting Extension:Math to 1bb3bfa3b5656 (duration: 00m 05s) | [production] | 
            
  | 00:44 | <ori> | Synchronized php-1.24wmf9/extensions/Math: Reverting Extension:Math to 1bb3bfa3b5656 (duration: 00m 06s) | [production] | 
            
  | 00:41 | <ori> | removed Physikerwelt and Frédéric Wang from extension-Math group in Gerrit pending further inquiry into recent changes | [production] | 
            
  | 00:38 | <ori> | Finished scap: fix any lingering inconsistencies in the state of the app servers (see https://gerrit.wikimedia.org/r/139089) (duration: 26m 59s) | [production] | 
            
  | 00:11 | <ori> | Started scap: fix any lingering inconsistencies in the state of the app servers (see https://gerrit.wikimedia.org/r/139089) | [production] | 
            
  
    | 2014-06-12
      
      § | 
    
  | 23:35 | <ori> | Synchronized php-1.24wmf8/extensions/MobileFrontend: Re-syncing after submodule update (duration: 00m 06s) | [production] | 
            
  | 23:34 | <ori> | ran sync-common on mw1151 | [production] | 
            
  | 23:18 | <catrope> | Synchronized php-1.24wmf9/extensions/MobileFrontend: (no message) (duration: 00m 04s) | [production] | 
            
  | 23:17 | <catrope> | Synchronized php-1.24wmf8/extensions/MobileFrontend: (no message) (duration: 00m 05s) | [production] | 
            
  | 23:17 | <catrope> | Synchronized php-1.24wmf8/extensions/VisualEditor: (no message) (duration: 00m 04s) | [production] | 
            
  | 23:07 | <Krinkle> | integration-slave1003 is failing npm-test builds due to a cache corruption (filed as https://github.com/npm/npm/issues/5472). Manually cleared /mnt/home/jenkins-deploy/.npm/async on integration-slave1003.eqiad.wmflabs for now. | [production] | 
            
  | 23:05 | <MaxSem> | Purging PageImages data from Wikibooks and Wikisource | [production] | 
            
  | 22:59 | <catrope> | Synchronized wmf-config/: (no message) (duration: 00m 04s) | [production] | 
            
  | 22:46 | <ori> | Synchronized wmf-config/CommonSettings.php: disable MW_MATH_MATHML until mathoid table is created (BUG 66492) (duration: 00m 04s) | [production] | 
            
  | 22:31 | <ori> | Synchronized php-1.24wmf8/extensions/WikimediaEvents: Update WikimediaEvents for Ibd36da416 (duration: 00m 03s) | [production] | 
            
  | 22:30 | <ori> | Synchronized php-1.24wmf9/extensions/WikimediaEvents: Update WikimediaEvents for Ibd36da416 (duration: 00m 03s) | [production] | 
            
  | 21:11 | <yurik> | Synchronized php-1.24wmf9/extensions/JsonConfig/: JsonConfig ext update, fixing bug 66555 (duration: 01m 03s) | [production] | 
            
  | 21:10 | <yurik> | Synchronized php-1.24wmf8/extensions/JsonConfig/: JsonConfig ext update, fixing bug 66555 (duration: 01m 04s) | [production] | 
            
  | 19:25 | <ottomata> | stopping puppet on an18 | [production] | 
            
  | 19:20 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: group0 to 1.24wmf9 | [production] | 
            
  | 19:19 | <ottomata> | starting hadoop decom of analytics1018.  This node will become a Kafka broker | [production] | 
            
  | 19:05 | <reedy> | rebuilt wikiversions.cdb and synchronized wikiversions files: wikipedias to 1.24wmf8 | [production] | 
            
  | 19:04 | <MaxSem> | Dropping old GeoData tables from everywhere | [production] | 
            
  | 18:52 | <reedy> | Finished scap: 1.24wmf9 staging take 2... (duration: 15m 20s) | [production] | 
            
  | 18:37 | <reedy> | Started scap: 1.24wmf9 staging take 2... | [production] | 
            
  | 18:06 | <reedy> | Started scap: testwiki to 1.24wmf9 and build l10n cache | [production] | 
            
  | 17:49 | <ottomata> | disabling puppet on analytics1012 and analytics1022 | [production] | 
            
  | 17:48 | <ottomata> | starting some kafka failure tests, I have scheduled downtime for some service checks in icinga, hopefully this will not be noisy | [production] | 
            
  | 17:41 | <ottomata> | restarting elasticsearch on logstash servers | [production] | 
            
  | 17:34 | <yurik> | Synchronized wmf-config/InitialiseSettings.php: Enabling new zero ext on all wikis (duration: 01m 03s) | [production] |