| 2015-11-10
      
      § | 
    
  | 23:42 | <twentyafterfour@tin> | Started scap: grr: sync 1.27.0-wmf.6 | [production] | 
            
  | 23:41 | <twentyafterfour@tin> | scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="cawikibooks" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.7rNhS6PIRi" ' returned non-zero exit status 1 (duration: 00m 09s) | [production] | 
            
  | 23:41 | <twentyafterfour@tin> | Started scap: grr: sync 1.27.0-wmf.6 | [production] | 
            
  | 23:40 | <twentyafterfour@tin> | scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="cawikibooks" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.yIUW9abUwO" ' returned non-zero exit status 1 (duration: 00m 15s) | [production] | 
            
  | 23:39 | <twentyafterfour@tin> | Started scap: trying again: sync 1.27.0-wmf.6 | [production] | 
            
  | 23:23 | <ori> | ran sudo chgrp -R wikidev /srv/mediawiki-staging/php-1.27.0-wmf.6 on tin | [production] | 
            
  | 22:43 | <twentyafterfour@tin> | scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="cawikibooks" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.sEpytaqlt1" ' returned non-zero exit status 1 (duration: 01m 42s) | [production] | 
            
  | 22:41 | <twentyafterfour@tin> | Started scap: testwiki to 1.27.0-wmf.6 | [production] | 
            
  | 21:51 | <hashar> | installed Zuul package on scandium.eqiad.wmnet . zuul-merger not running | [production] | 
            
  | 21:15 | <mutante> | scb1001 - mobile endpoint health CRIT | [production] | 
            
  | 21:15 | <mutante> | mira unmerged changes in mw-staging | [production] | 
            
  | 21:14 | <mutante> | restbase2001: cassandra CQl refused | [production] | 
            
  | 21:14 | <mutante> | aqs1001-1003: CRITICAL: Test Get aggregate page views | [production] | 
            
  | 21:12 | <mutante> | tungsten - signed new puppet certs, re-installed | [production] | 
            
  | 20:48 | <moritzm> | kafka re-enabled on kafka1013 | [production] | 
            
  | 20:41 | <moritzm> | stopping kafka broker on kafka1013 to activate ferm | [production] | 
            
  | 20:26 | <ottomata> | starting kafka broker on kafka1012 after applying base ferm | [production] | 
            
  | 20:14 | <ottomata> | stopping kafka broker on kafka1012 to apply base ferm | [production] | 
            
  | 20:09 | <moritzm> | disabled puppet on kafk1012/1013 in preparation of ferm activation | [production] | 
            
  | 20:08 | <_joe_> | restarting jobchron across the cluster | [production] | 
            
  | 18:41 | <ottomata> | powercycling analytics1031 | [production] | 
            
  | 16:51 | <krenair@tin> | Synchronized php-1.27.0-wmf.5/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: https://gerrit.wikimedia.org/r/#/c/252231/ (duration: 00m 35s) | [production] | 
            
  | 16:45 | <krenair@tin> | Synchronized wmf-config/Wikibase-labs.php: (no message) (duration: 00m 34s) | [production] | 
            
  | 16:44 | <krenair@tin> | Synchronized wmf-config/Wikibase.php: https://gerrit.wikimedia.org/r/#/c/252204/ (duration: 00m 34s) | [production] | 
            
  | 16:41 | <krenair@tin> | Synchronized wmf-config/Wikibase.php: https://gerrit.wikimedia.org/r/#/c/250000/ (duration: 00m 34s) | [production] | 
            
  | 16:34 | <krenair@tin> | Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/250471/ (duration: 00m 34s) | [production] | 
            
  | 16:28 | <krenair@tin> | Synchronized php-1.27.0-wmf.5/extensions/ContentTranslation/api/ApiQueryContentTranslationSuggestions.php: https://gerrit.wikimedia.org/r/#/c/252172/ (duration: 00m 35s) | [production] | 
            
  | 16:22 | <krenair@tin> | Synchronized php-1.27.0-wmf.5/extensions/Wikidata/Wikidata.php: https://gerrit.wikimedia.org/r/#/c/252227/ (duration: 00m 45s) | [production] | 
            
  | 15:51 | <mutante> | restarted krypton | [production] | 
            
  | 15:32 | <godog> | upgrade rsync on ms-be1* per T93587 | [production] | 
            
  | 14:02 | <godog> | swift codfw-prod: ms-be2017 / ms-be2019 / ms-be2021 weight 3000 | [production] | 
            
  | 10:00 | <jynus> | performing mysql maintenance of db1035 (depooled from production). Replication delay and reboots are expected. | [production] | 
            
  | 09:46 | <godog> | swift codfw-prod: ms-be2017 / ms-be2019 / ms-be2021 weight 2000 | [production] | 
            
  | 09:39 | <jynus@tin> | Synchronized wmf-config/db-eqiad.php: Depool db1035 for maintenance (duration: 00m 36s) | [production] | 
            
  | 03:12 | <ottomata> | restarted statsv on hafnium just in case it stopped after kafka1013 broker crash and restart (ping ori :) ) | [production] | 
            
  | 03:10 | <ottomata> | kafka preferred-replica-election to bring kafka1013  back as leader for its partitions | [production] | 
            
  | 03:00 | <mutante> | started kafka on kafka1013 | [production] | 
            
  | 02:55 | <mutante> | tried to start kafka on kafka1018 | [production] | 
            
  | 02:31 | <mutante> | switched people.wm.org to new backend. please use "rutherfordium.eqiad.wmnet" instead of terbium now. all files have been rsynced already | [production] | 
            
  | 02:22 | <l10nupdate@tin> | Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 06m 49s) | [production] | 
            
  | 00:13 | <hoo> | Attached Richardelainechambers@commonswiki to the global account of the same name | [production] | 
            
  | 00:06 | <bd808@tin> | Synchronized wmf-config/logging.php: Disable microsecond timestamps on all loggers (bf61bfc) (duration: 00m 34s) | [production] |