| 
      
        2019-03-28
      
      ยง
     | 
  
    
  | 18:17 | 
  <jforrester@deploy1001> | 
  Synchronized php-1.33.0-wmf.23/extensions/AdvancedSearch/: AdvancedSearch: Fix two UBNs T219455 T219539 (duration: 00m 59s) | 
  [production] | 
            
  | 18:03 | 
  <ejegg> | 
  updated payments-wiki from 6661655e37 to 4b49bb7333 | 
  [production] | 
            
  | 17:46 | 
  <ebernhardson@deploy1001> | 
  Finished deploy [search/mjolnir/deploy@ff9d424]: Deploy logging @cee: prefixing bugfix (duration: 03m 24s) | 
  [production] | 
            
  | 17:43 | 
  <ebernhardson@deploy1001> | 
  Started deploy [search/mjolnir/deploy@ff9d424]: Deploy logging @cee: prefixing bugfix | 
  [production] | 
            
  | 16:39 | 
  <XioNoX> | 
  enable cr2-codfw:xe-5/0/0 (to cr2-eqdfw) | 
  [production] | 
            
  | 16:36 | 
  <mutante> | 
  wikitech-static - changing [renewalparams] authenticator = to 'apache' from 'standalone' (installer = was already apache) (T214640) | 
  [production] | 
            
  | 16:36 | 
  <jbond42> | 
  move python3-requests and python3-urllib3 from jessie-wikimedia backports to component/kube2proxy | 
  [production] | 
            
  | 16:33 | 
  <XioNoX> | 
  disable cr2-codfw:xe-5/0/0 (to cr2-eqdfw) | 
  [production] | 
            
  | 16:00 | 
  <akosiaris> | 
  poweroff sessionstore2001 for a re-racking | 
  [production] | 
            
  | 15:15 | 
  <mutante> | 
  wikitech-static - removing acme-setup cron jobs from root's crontab. this was used before the switch to certbot, is unrelated and added to confusion and maybe the problem (T214640) | 
  [production] | 
            
  | 15:07 | 
  <otto@deploy1001> | 
  scap-helm eventgate-analytics finished | 
  [production] | 
            
  | 15:07 | 
  <otto@deploy1001> | 
  scap-helm eventgate-analytics cluster eqiad completed | 
  [production] | 
            
  | 15:07 | 
  <otto@deploy1001> | 
  scap-helm eventgate-analytics upgrade production -f eventgate-analytics-eqiad-values.yaml --reset-values stable/eventgate-analytics [namespace: eventgate-analytics, clusters: eqiad] | 
  [production] | 
            
  | 15:07 | 
  <otto@deploy1001> | 
  scap-helm eventgate-analytics finished | 
  [production] | 
            
  | 15:07 | 
  <otto@deploy1001> | 
  scap-helm eventgate-analytics cluster codfw completed | 
  [production] | 
            
  | 15:07 | 
  <otto@deploy1001> | 
  scap-helm eventgate-analytics upgrade production -f eventgate-analytics-codfw-values.yaml --reset-values stable/eventgate-analytics [namespace: eventgate-analytics, clusters: codfw] | 
  [production] | 
            
  | 15:06 | 
  <otto@deploy1001> | 
  scap-helm eventgate-analytics finished | 
  [production] | 
            
  | 15:06 | 
  <otto@deploy1001> | 
  scap-helm eventgate-analytics cluster staging completed | 
  [production] | 
            
  | 15:06 | 
  <otto@deploy1001> | 
  scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml --reset-values stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging] | 
  [production] | 
            
  | 14:46 | 
  <ppchelko@deploy1001> | 
  Finished deploy [cpjobqueue/deploy@4deeb04]: Partition htmlCacheUpdate topic, final cleanup stage T219159 (duration: 00m 52s) | 
  [production] | 
            
  | 14:45 | 
  <ppchelko@deploy1001> | 
  Started deploy [cpjobqueue/deploy@4deeb04]: Partition htmlCacheUpdate topic, final cleanup stage T219159 | 
  [production] | 
            
  | 14:32 | 
  <ppchelko@deploy1001> | 
  Finished deploy [cpjobqueue/deploy@3a8a889]: Partition htmlCacheUpdate topic, step 2 T219159 (duration: 00m 53s) | 
  [production] | 
            
  | 14:31 | 
  <ppchelko@deploy1001> | 
  Started deploy [cpjobqueue/deploy@3a8a889]: Partition htmlCacheUpdate topic, step 2 T219159 | 
  [production] | 
            
  | 14:07 | 
  <gehel> | 
  reindexing changes from '2019-03-26T12:00:00Z' to '2019-03-28T12:00:00Z' into cirrus / elasticsearch - T218878 | 
  [production] | 
            
  | 13:59 | 
  <gehel> | 
  restarting elasticsearch on elastic2050 to validate JVM upgrade | 
  [production] | 
            
  | 13:57 | 
  <moritzm> | 
  upgrading Java on elasticsearch hosts | 
  [production] | 
            
  | 13:50 | 
  <filippo@puppetmaster1001> | 
  conftool action : set/pooled=no; selector: name=prometheus2004.codfw.wmnet | 
  [production] | 
            
  | 13:49 | 
  <filippo@puppetmaster1001> | 
  conftool action : set/pooled=yes; selector: name=prometheus2003.codfw.wmnet | 
  [production] | 
            
  | 13:22 | 
  <ppchelko@deploy1001> | 
  Finished deploy [cpjobqueue/deploy@c120b38]: Partition htmlCacheUpdate topic, explicitly exclude htmlCacheUpdate T219159 (duration: 00m 48s) | 
  [production] | 
            
  | 13:21 | 
  <ppchelko@deploy1001> | 
  Started deploy [cpjobqueue/deploy@c120b38]: Partition htmlCacheUpdate topic, explicitly exclude htmlCacheUpdate T219159 | 
  [production] | 
            
  | 13:14 | 
  <ppchelko@deploy1001> | 
  Finished deploy [cpjobqueue/deploy@17285f8]: Partition htmlCacheUpdate topic, step 1 T219159 (duration: 01m 46s) | 
  [production] | 
            
  | 13:12 | 
  <ppchelko@deploy1001> | 
  Started deploy [cpjobqueue/deploy@17285f8]: Partition htmlCacheUpdate topic, step 1 T219159 | 
  [production] | 
            
  | 12:20 | 
  <moritzm> | 
  removing php 7.0 packages from snapshot1005-1007/1009, dumps are only using 7.2  (T218193) | 
  [production] | 
            
  | 12:13 | 
  <jbond42> | 
  move git from jessie-wikimedia backports repo  components/ci | 
  [production] | 
            
  | 12:02 | 
  <lucaswerkmeister-wmde@deploy1001> | 
  Synchronized wmf-config/InitialiseSettings.php: [[gerrit:499756|Revert "SDC: Enable both new-style and old-style Wikibase federation on Commons" (T219450)]] (duration: 00m 57s) | 
  [production] | 
            
  | 11:54 | 
  <moritzm> | 
  upgrading snapshot1005-1007/1009 to component/php72 (T218193) | 
  [production] | 
            
  | 11:53 | 
  <ladsgroup@deploy1001> | 
  rebuilt and synchronized wikiversions files: Revert T212597 | 
  [production] | 
            
  | 11:51 | 
  <ladsgroup@deploy1001> | 
  Synchronized dblists: Revert T212597 (duration: 00m 58s) | 
  [production] | 
            
  | 11:29 | 
  <ladsgroup@deploy1001> | 
  rebuilt and synchronized wikiversions files: T212597 | 
  [production] | 
            
  | 11:27 | 
  <ladsgroup@deploy1001> | 
  Synchronized dblists: T212597 (duration: 00m 56s) | 
  [production] | 
            
  | 11:01 | 
  <godog> | 
  test copying prometheus metrics on bast3002 | 
  [production] | 
            
  | 10:54 | 
  <gehel> | 
  restarting elasticsearch-psi on elastic20[35,36,53] (shards stuck in recovery) - T218878 | 
  [production] | 
            
  | 10:22 | 
  <gehel> | 
  restarting elasticsearch on elastic20[34,36,50] (shards stuck in recovery) - T218878 | 
  [production] | 
            
  | 10:15 | 
  <addshore@deploy1001> | 
  Synchronized php-1.33.0-wmf.23/extensions/Wikibase/lib: T219452 [[gerrit:499738|Revert: Use enableModuleContentVersion() for Wikibase\lib\SitesModule]] (duration: 01m 06s) | 
  [production] | 
            
  | 10:11 | 
  <gehel> | 
  restarting elasticsearch-omega on elastic2050 (shards stuck in recovery) - T218878 | 
  [production] | 
            
  | 09:56 | 
  <gehel> | 
  restarting elasticsearch-omega on elastic2031 (shards stuck in recovery) - T218878 | 
  [production] | 
            
  | 09:42 | 
  <gehel> | 
  restarting elasticsearch on elastic20[28,29,41] (shards stuck in recovery) - T218878 | 
  [production] | 
            
  | 09:37 | 
  <gehel> | 
  restarting elasticsearch-psi on elastic20[39,40] (shards stuck in recovery) - T218878 | 
  [production] | 
            
  | 09:33 | 
  <marostegui@deploy1001> | 
  Synchronized wmf-config/db-eqiad.php: Repool db1123 (duration: 00m 56s) | 
  [production] | 
            
  | 09:28 | 
  <gehel> | 
  restarting elasticsearch on elastic20[25,27] (shards stuck in recovery) - T218878 | 
  [production] |