| 2018-07-10
      
      § | 
    
  | 23:50 | <ejegg> | disabled ProcessSmashPigRecurring in Civi scheduled jobs, to be replaced with process-control job | [production] | 
            
  | 22:27 | <ejegg> | updated payments-wiki from 383f667171 to 63372154d0 | [production] | 
            
  | 21:40 | <chasemp> | reboot labnet100[34] | [production] | 
            
  | 21:04 | <XioNoX> | re-configure GTT circuit in eqiad/knams | [production] | 
            
  | 20:50 | <aaron@deploy1001> | Synchronized php-1.32.0-wmf.10/tests/phpunit/includes/libs/objectcache/MultiWriteBagOStuffTest.php: 4fba9f6a032 (duration: 00m 56s) | [production] | 
            
  | 20:48 | <aaron@deploy1001> | Synchronized php-1.32.0-wmf.10/includes/libs/objectcache/MultiWriteBagOStuff.php: 4fba9f6a032 (duration: 00m 57s) | [production] | 
            
  | 18:33 | <aaron@deploy1001> | Synchronized wmf-config/mc.php: Make all non-test wikis write to both nutcracker and mcrouter again (duration: 00m 57s) | [production] | 
            
  | 16:51 | <zfilipin@deploy1001> | rebuilt and synchronized wikiversions files: group0 to 1.32.0-wmf.12 | [production] | 
            
  | 16:03 | <vgutierrez> | replacing baham with authdns2001 - T196664 | [production] | 
            
  | 15:27 | <ema> | merge alternate_domains vcl patch T164609 | [production] | 
            
  | 15:19 | <zfilipin@deploy1001> | Finished scap: testwiki to php-1.32.0-wmf.12 and rebuild l10n cache (duration: 61m 27s) | [production] | 
            
  | 14:31 | <marostegui> | Set disk #0 offline for replacement - T199056 | [production] | 
            
  | 14:17 | <zfilipin@deploy1001> | Started scap: testwiki to php-1.32.0-wmf.12 and rebuild l10n cache | [production] | 
            
  | 14:15 | <moritzm> | installing tiff security updates on jessie | [production] | 
            
  | 13:53 | <moritzm> | installing ruby-sprockets security updates | [production] | 
            
  | 13:48 | <moritzm> | installing subversion updates from jessie 9.11 point release | [production] | 
            
  | 13:40 | <moritzm> | rolling restart of thumbor to pick up new exiv2/openssl | [production] | 
            
  | 13:29 | <thcipriani@deploy1001> | Pruned MediaWiki: 1.32.0-wmf.5 (duration: 03m 13s) | [production] | 
            
  | 13:26 | <moritzm> | installing cups security updates on jessie | [production] | 
            
  | 13:25 | <thcipriani@deploy1001> | Pruned MediaWiki: 1.32.0-wmf.4 (duration: 06m 18s) | [production] | 
            
  | 13:14 | <thcipriani@deploy1001> | Synchronized scap/plugins/clean.py: no op sync for consistancy [[gerrit:441920|Scap clean: remove remote cache directory]] (duration: 00m 51s) | [production] | 
            
  | 12:53 | <aaron@deploy1001> | Synchronized php-1.32.0-wmf.10/includes/libs/objectcache/ReplicatedBagOStuff.php: 4ad6b70ba132c66e14a706eae240887885946a42 (duration: 00m 51s) | [production] | 
            
  | 12:24 | <aaron@deploy1001> | Synchronized php-1.32.0-wmf.10/extensions/SpamBlacklist/includes/SpamBlacklist.php: 08a2153f7aa (duration: 00m 51s) | [production] | 
            
  | 12:20 | <aaron@deploy1001> | Synchronized php-1.32.0-wmf.12/extensions/SpamBlacklist/includes/SpamBlacklist.php: 583dc7a92f9b (duration: 00m 51s) | [production] | 
            
  | 12:06 | <aaron@deploy1001> | Synchronized wmf-config/mc.php: Revert "Make all non-test wikis write to both nutcracker and mcrouter" (duration: 00m 56s) | [production] | 
            
  | 11:56 | <ppchelko@deploy1001> | Finished deploy [restbase/deploy@2674971]: Roll out cassandra-driver@3.5.0 to restbase2001 (duration: 02m 52s) | [production] | 
            
  | 11:53 | <ppchelko@deploy1001> | Started deploy [restbase/deploy@2674971]: Roll out cassandra-driver@3.5.0 to restbase2001 | [production] | 
            
  | 11:43 | <arturo> | updated compiler facts `PUPPET_MASTERS=puppetmaster1001.eqiad.wmnet PUPPET_COMPILER=compiler02.puppet3-diffs.eqiad.wmflabs modules/puppet_compiler/files/compiler-update-facts` | [production] | 
            
  | 11:32 | <aaron@deploy1001> | Synchronized wmf-config/mc.php: Make all non-test wikis write to both nutcracker and mcrouter (duration: 00m 51s) | [production] | 
            
  | 11:25 | <dcausse@deploy1001> | Synchronized ./wmf-config/InitialiseSettings.php: [cirrus] cleanup unused config vars 2/2 (duration: 01m 40s) | [production] | 
            
  | 11:24 | <moritzm> | installing PHP 7 security updates on stretch | [production] | 
            
  | 11:09 | <dcausse@deploy1001> | Synchronized ./wmf-config/: [cirrus] cleanup unused config vars 1/2 (duration: 00m 53s) | [production] | 
            
  | 10:27 | <ppchelko@deploy1001> | Finished deploy [restbase/deploy@d724ad1]: Fix up the invalid Vary header, take 2, checer timed out (duration: 06m 39s) | [production] | 
            
  | 10:20 | <ppchelko@deploy1001> | Started deploy [restbase/deploy@d724ad1]: Fix up the invalid Vary header, take 2, checer timed out | [production] | 
            
  | 10:20 | <ppchelko@deploy1001> | deploy aborted: Fix up the invalid Vary header (duration: 12m 50s) | [production] | 
            
  | 10:07 | <ppchelko@deploy1001> | Started deploy [restbase/deploy@d724ad1]: Fix up the invalid Vary header | [production] | 
            
  | 10:07 | <moritzm> | restarting thumbor on thumbor1001 to pick up exiv2 security updates | [production] | 
            
  | 10:03 | <elukey> | restart analytics100[1,2]'s hadoop resource managers, some I/O socket errors after the ip6 interface change | [production] | 
            
  | 09:34 | <elukey> | forced umount of /mnt/hdfs on stat1004, several processes hang for it (causing load) and transport not connected | [production] | 
            
  | 09:25 | <godog> | swift eqiad-prod add ms-be1036 back gradually - T196873 | [production] | 
            
  | 09:05 | <moritzm> | installing libsoup security updates on jessie/stretch | [production] | 
            
  | 08:38 | <moritzm> | installing libjpeg-turbo security updates on trusty | [production] | 
            
  | 08:37 | <godog> | drain and restart cassandra-a on restbase2001 to test a restart | [production] | 
            
  | 08:34 | <elukey> | rolling restart of AQS to apply the new config | [production] | 
            
  | 08:09 | <godog> | disable puppet on hosts running cassandra before merging 444247 and 443114 | [production] | 
            
  | 08:08 | <moritzm> | installing openslp security updates on trusty | [production] | 
            
  | 07:58 | <moritzm> | installing ntp security updates on trusty (may trigger some Icinga warnings about clocks, these recover after a while) | [production] | 
            
  | 07:33 | <twentyafterfour> | deploying fix for phabricator userpage having hard-coded my username | [production] |