| 
      
        2015-11-04
      
      §
     | 
  
    
  | 05:02 | 
  <demon@tin> | 
  scap failed: CalledProcessError Command '['sudo', '-u', 'mwdeploy', '-n', '--', '/usr/bin/rsync', '--archive', '--delete-delay', '--delay-updates', '--compress', '--delete', '--exclude=**/.svn/lock', '--exclude=**/.git/objects', '--exclude=**/.git/**/objects', '--exclude=**/cache/l10n/*.cdb', '--no-perms', 'tin.eqiad.wmnet::common', '/srv/mediawiki']' returned non-zero exit status 24 (duration: 02m 45s) | 
  [production] | 
            
  | 04:59 | 
  <demon@tin> | 
  Started scap: using wfLoadExtension() for ContentTranslation, shut up warning spam | 
  [production] | 
            
  | 03:09 | 
  <l10nupdate@tin> | 
  LocalisationUpdate completed (1.27.0-wmf.5) at 2015-11-04 03:09:44+00:00 | 
  [production] | 
            
  | 03:03 | 
  <l10nupdate@tin> | 
  Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 10m 08s) | 
  [production] | 
            
  | 02:36 | 
  <l10nupdate@tin> | 
  LocalisationUpdate completed (1.27.0-wmf.4) at 2015-11-04 02:36:48+00:00 | 
  [production] | 
            
  | 02:32 | 
  <l10nupdate@tin> | 
  Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 07m 34s) | 
  [production] | 
            
  | 01:36 | 
  <legoktm> | 
  ran delete from globaluser where gu_name="" limit 1 on centralauth. (T96233) | 
  [production] | 
            
  | 01:10 | 
  <krenair@tin> | 
  Synchronized php-1.27.0-wmf.5/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: https://gerrit.wikimedia.org/r/#/c/250874/ (duration: 00m 17s) | 
  [production] | 
            
  | 01:02 | 
  <hoo> | 
  Manually recreated https://dumps.wikimedia.org/wikidatawiki/entities/dcatap.rdf using Zend php (instead of HHVM). See T117534. | 
  [production] | 
            
  | 00:25 | 
  <krenair@tin> | 
  Synchronized php-1.27.0-wmf.5/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/250869/ (duration: 00m 19s) | 
  [production] | 
            
  | 00:09 | 
  <krenair@tin> | 
  Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/250712/ (duration: 00m 18s) | 
  [production] | 
            
  | 00:03 | 
  <krenair@tin> | 
  Synchronized portals: https://gerrit.wikimedia.org/r/#/c/250851/ (duration: 00m 17s) | 
  [production] | 
            
  
    | 
      
        2015-11-03
      
      §
     | 
  
    
  | 22:36 | 
  <ostriches> | 
  tin: scap now pointing to Phab repo instead of Gerrit. | 
  [production] | 
            
  | 21:59 | 
  <krenair@tin> | 
  Synchronized README: rv, was just demonstrating for https://phabricator.wikimedia.org/T117574#1779384 (duration: 00m 17s) | 
  [production] | 
            
  | 21:42 | 
  <krenair@tin> | 
  Synchronized README: (no message) (duration: 00m 18s) | 
  [production] | 
            
  | 21:21 | 
  <mutante> | 
  ms-be2018,ms-be2019: sign puppet certs, initial run | 
  [production] | 
            
  | 20:45 | 
  <csteipp> | 
  deployed patch for T109724 to wmf4/5 | 
  [production] | 
            
  | 20:41 | 
  <gwicke> | 
  restbase cassandra: switched local_group_wikipedia_T_parsoid_html to the Date-Tiered Compaction Strategy (DTCS) | 
  [production] | 
            
  | 20:38 | 
  <gwicke> | 
  cleared cassandra snapshots on aqs servers | 
  [production] | 
            
  | 19:59 | 
  <twentyafterfour@tin> | 
  rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.27.0-wmf.5 | 
  [production] | 
            
  | 19:55 | 
  <mutante> | 
  hafnium - revoke puppet cert, salt key for the .wikimedia.org name | 
  [production] | 
            
  | 19:54 | 
  <mutante> | 
  hafnium - reboot into PXE for jessie reinstall | 
  [production] | 
            
  | 19:47 | 
  <twentyafterfour@tin> | 
  Finished scap: testwiki to 1.27.0-wmf.5 (duration: 30m 00s) | 
  [production] | 
            
  | 19:17 | 
  <twentyafterfour@tin> | 
  Started scap: testwiki to 1.27.0-wmf.5 | 
  [production] | 
            
  | 19:11 | 
  <ori@tin> | 
  Synchronized php-1.27.0-wmf.4/includes/libs/objectcache/BagOStuff.php: 46dff75b5b: Make makeKeyInternal() limit more conservative (duration: 00m 18s) | 
  [production] | 
            
  | 19:06 | 
  <twentyafterfour> | 
  deploying 1.27.0-wmf.5 to group0 | 
  [production] | 
            
  | 18:34 | 
  <ori> | 
  Moved statsv script from hafnium to a screen session on eventlog2001 in preparation for hafnium reimage | 
  [production] | 
            
  | 18:29 | 
  <ori> | 
  Moved navtiming script from hafnium to a screen session on eventlog2001 in preparation for hafnium reimage | 
  [production] | 
            
  | 17:13 | 
  <moritzm> | 
  installed security updates on the internal ntpds (system-local ones to follow) | 
  [production] | 
            
  | 16:27 | 
  <dcausse> | 
  restarting elastic on nobelium to check new settings | 
  [production] | 
            
  | 16:19 | 
  <moritzm> | 
  installed unzip and audiofile security updates across the fleet | 
  [production] | 
            
  | 16:12 | 
  <thcipriani@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove sampling of CirrusSearchRequestSet log channel [[gerrit:250691]] (duration: 00m 18s) | 
  [production] | 
            
  | 16:07 | 
  <thcipriani@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor for 10% of new accounts on eswiki [[gerrit:250470]] (duration: 00m 18s) | 
  [production] | 
            
  | 14:19 | 
  <jynus> | 
  disabling puppet on dbstore1002 and db1047 to debug replication issue | 
  [production] | 
            
  | 14:07 | 
  <godog> | 
  manual invocation of /usr/local/bin/archive-instances on labmon1001 to test | 
  [production] | 
            
  | 13:33 | 
  <jynus> | 
  migrating eventlogging_sync from terbium to db1047 and dbstore1002 (analytics-store) | 
  [production] | 
            
  | 12:15 | 
  <_joe_> | 
  updated pybal for jessie to 1.12 | 
  [production] | 
            
  | 11:00 | 
  <godog> | 
  start cassandra-a on xenon | 
  [production] | 
            
  | 09:56 | 
  <godog> | 
  reimage xenon.eqiad.wmnet | 
  [production] | 
            
  | 09:44 | 
  <godog> | 
  decomission xenon.eqiad.wmnet from cassandra, pending conversion to multi-instance | 
  [production] | 
            
  | 05:38 | 
  <l10nupdate@tin> | 
  ResourceLoader cache refresh completed at Tue Nov  3 05:38:33 UTC 2015 (duration 38m 32s) | 
  [production] | 
            
  | 02:26 | 
  <l10nupdate@tin> | 
  LocalisationUpdate completed (1.27.0-wmf.4) at 2015-11-03 02:26:14+00:00 | 
  [production] | 
            
  | 02:22 | 
  <l10nupdate@tin> | 
  Synchronized php-1.27.0-wmf.4/cache/l10n: l10nupdate for 1.27.0-wmf.4 (duration: 07m 00s) | 
  [production] | 
            
  | 01:03 | 
  <krinkle@tin> | 
  Synchronized php-1.27.0-wmf.4/extensions/CentralAuth/includes/CentralAuthUser.php: Ia9673a448da81: User cache key (duration: 00m 17s) | 
  [production] | 
            
  | 01:03 | 
  <krinkle@tin> | 
  Synchronized php-1.27.0-wmf.4/includes/: I58836a24b9e239f: User cache key (duration: 00m 21s) | 
  [production] | 
            
  | 00:37 | 
  <ebernhardson@tin> | 
  Synchronized wmf-config/logging.php: https://gerrit.wikimedia.org/r/#/c/240615/ (duration: 00m 19s) | 
  [production] | 
            
  | 00:35 | 
  <ebernhardson@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/240615/ (duration: 00m 17s) | 
  [production] | 
            
  | 00:34 | 
  <ebernhardson@tin> | 
  Synchronized wmf-config/avro/CirrusSearchRequestSet.avsc: https://gerrit.wikimedia.org/r/#/c/240615/ (duration: 00m 17s) | 
  [production] | 
            
  | 00:27 | 
  <ebernhardson> | 
  sync-common on mw1017 for https://gerrit.wikimedia.org/r/240615 | 
  [production] | 
            
  | 00:26 | 
  <ebernhardson@tin> | 
  Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/241226/4 (duration: 00m 17s) | 
  [production] |