| 
      
        2019-04-17
      
      ยง
     | 
  
    
  | 21:10 | 
  <thcipriani> | 
  gerrit back | 
  [production] | 
            
  | 21:07 | 
  <thcipriani@deploy1001> | 
  Finished deploy [gerrit/gerrit@4dcb851]: Gerrit update (cobalt -- restart incoming) (duration: 00m 10s) | 
  [production] | 
            
  | 21:07 | 
  <thcipriani@deploy1001> | 
  Started deploy [gerrit/gerrit@4dcb851]: Gerrit update (cobalt -- restart incoming) | 
  [production] | 
            
  | 21:06 | 
  <thcipriani@deploy1001> | 
  Finished deploy [gerrit/gerrit@4dcb851]: Gerrit update (gerrit2001 only) (duration: 00m 11s) | 
  [production] | 
            
  | 21:06 | 
  <thcipriani@deploy1001> | 
  Started deploy [gerrit/gerrit@4dcb851]: Gerrit update (gerrit2001 only) | 
  [production] | 
            
  | 19:14 | 
  <twentyafterfour@deploy1001> | 
  Synchronized php: group1 wikis to 1.34.0-wmf.1  refs T220726 (duration: 01m 49s) | 
  [production] | 
            
  | 19:13 | 
  <twentyafterfour@deploy1001> | 
  rebuilt and synchronized wikiversions files: group1 wikis to 1.34.0-wmf.1  refs T220726 | 
  [production] | 
            
  | 18:04 | 
  <thcipriani> | 
  gerrit back | 
  [production] | 
            
  | 18:01 | 
  <thcipriani> | 
  gerrit restart for https://gerrit.wikimedia.org/r/504611/ | 
  [production] | 
            
  | 17:59 | 
  <jforrester@deploy1001> | 
  Synchronized wmf-config/InitialiseSettings.php: SDC: Enable Wikidata federation on Commons again T214075 (duration: 01m 00s) | 
  [production] | 
            
  | 17:20 | 
  <otto@deploy1001> | 
  Synchronized wmf-config/InitialiseSettings.php: Enabling EventGate api-request logging on group1 wikis (duration: 01m 00s) | 
  [production] | 
            
  | 17:18 | 
  <mutante> | 
  LDAP - added 'brennen' to group 'gerritadmin' (T218858) | 
  [production] | 
            
  | 17:18 | 
  <jforrester@deploy1001> | 
  Synchronized php-1.34.0-wmf.1/extensions/OATHAuth/: UBN T221257 train un-blocker (duration: 01m 02s) | 
  [production] | 
            
  | 17:09 | 
  <jforrester@deploy1001> | 
  Synchronized php-1.34.0-wmf.1/extensions/Echo/includes/formatters/: Notifications: Revert 7121b9c4 per I8f9a6a19ba (duration: 01m 01s) | 
  [production] | 
            
  | 16:49 | 
  <tzatziki> | 
  deleting three files for legal compliance | 
  [production] | 
            
  | 16:47 | 
  <jforrester@deploy1001> | 
  Synchronized php-1.34.0-wmf.1/extensions/WikibaseMediaInfo/: SDC: Various fixes T218922 T221071 T221110 T221123 (duration: 01m 02s) | 
  [production] | 
            
  | 16:41 | 
  <jforrester@deploy1001> | 
  Synchronized php-1.34.0-wmf.1/autoload.php: Update to point to new maintenance scripts (duration: 01m 00s) | 
  [production] | 
            
  | 16:39 | 
  <jforrester@deploy1001> | 
  Synchronized php-1.34.0-wmf.1/maintenance/language/generateUpperCharTable.php: Maintenance script for _joe_ (duration: 00m 59s) | 
  [production] | 
            
  | 16:38 | 
  <jforrester@deploy1001> | 
  Synchronized php-1.34.0-wmf.1/maintenance/language/generateUcfirstOverrides.php: Maintenance script for _joe_ (duration: 01m 00s) | 
  [production] | 
            
  | 16:20 | 
  <jforrester@deploy1001> | 
  Synchronized php-1.34.0-wmf.1/languages/Language.php: T219279 Ability to set wgOverrideUcfirstCharacters part 1 try two (duration: 01m 00s) | 
  [production] | 
            
  | 16:18 | 
  <jforrester@deploy1001> | 
  Synchronized php-1.34.0-wmf.1/includes/DefaultSettings.php: T219279 Ability to set wgOverrideUcfirstCharacters part 1b (duration: 01m 03s) | 
  [production] | 
            
  | 16:13 | 
  <jforrester@deploy1001> | 
  scap failed: average error rate on 4/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/db09a36be5ed3e81155041f7d46ad040 for details) | 
  [production] | 
            
  | 16:11 | 
  <XioNoX> | 
  set fasw-c-eqiad:ge-[0-1]/0/17 in admin vlan - T221232 | 
  [production] | 
            
  | 16:08 | 
  <jforrester@deploy1001> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT T220434 Deploy Partial blocks to Chinese Wikipedia (duration: 01m 02s) | 
  [production] | 
            
  | 14:37 | 
  <ariel@deploy1001> | 
  Finished deploy [dumps/dumps@dcf04a0]: fix up paths for 1.34_wmf.1 for AbstractFilter (duration: 00m 04s) | 
  [production] | 
            
  | 14:36 | 
  <ariel@deploy1001> | 
  Started deploy [dumps/dumps@dcf04a0]: fix up paths for 1.34_wmf.1 for AbstractFilter | 
  [production] | 
            
  | 14:35 | 
  <otto@deploy1001> | 
  scap-helm eventgate-analytics finished | 
  [production] | 
            
  | 14:35 | 
  <otto@deploy1001> | 
  scap-helm eventgate-analytics cluster eqiad completed | 
  [production] | 
            
  | 14:35 | 
  <otto@deploy1001> | 
  scap-helm eventgate-analytics upgrade production -f eventgate-analytics-eqiad-values.yaml --reset-values stable/eventgate-analytics [namespace: eventgate-analytics, clusters: eqiad] | 
  [production] | 
            
  | 14:34 | 
  <otto@deploy1001> | 
  scap-helm eventgate-analytics finished | 
  [production] | 
            
  | 14:34 | 
  <otto@deploy1001> | 
  scap-helm eventgate-analytics cluster codfw completed | 
  [production] | 
            
  | 14:34 | 
  <otto@deploy1001> | 
  scap-helm eventgate-analytics upgrade production -f eventgate-analytics-codfw-values.yaml --reset-values stable/eventgate-analytics [namespace: eventgate-analytics, clusters: codfw] | 
  [production] | 
            
  | 14:13 | 
  <elukey@cumin1001> | 
  END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) | 
  [production] | 
            
  | 14:12 | 
  <elukey@cumin1001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 13:56 | 
  <otto@deploy1001> | 
  scap-helm eventgate-analytics finished | 
  [production] | 
            
  | 13:56 | 
  <otto@deploy1001> | 
  scap-helm eventgate-analytics cluster staging completed | 
  [production] | 
            
  | 13:56 | 
  <otto@deploy1001> | 
  scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml --reset-values stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging] | 
  [production] | 
            
  | 13:52 | 
  <elukey> | 
  upgrading hadoop cdh distrubition to 5.16.1 on all the Hadoop-related nodes - T218343 | 
  [production] | 
            
  | 13:48 | 
  <elukey@cumin1001> | 
  END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) | 
  [production] | 
            
  | 13:48 | 
  <elukey@cumin1001> | 
  START - Cookbook sre.hosts.downtime | 
  [production] | 
            
  | 13:47 | 
  <godog> | 
  reimage prometheus2004 - T187987 | 
  [production] | 
            
  | 12:57 | 
  <filippo@puppetmaster1001> | 
  conftool action : set/pooled=yes; selector: name=prometheus1004.eqiad.wmnet | 
  [production] | 
            
  | 12:44 | 
  <godog> | 
  bounce prometheus instances on prometheus[12]003 after https://gerrit.wikimedia.org/r/c/operations/puppet/+/499742 | 
  [production] | 
            
  | 12:33 | 
  <moritzm> | 
  running some ferm tests on graphite2002 | 
  [production] | 
            
  | 12:10 | 
  <godog> | 
  briefly stop all prometheus on prometheus1003 to finish metrics rsync - T187987 | 
  [production] | 
            
  | 11:39 | 
  <Lucas_WMDE> | 
  EU SWAT done | 
  [production] | 
            
  | 11:38 | 
  <lucaswerkmeister-wmde@deploy1001> | 
  Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:504380|Enable suggestion constraint status on testwikidata (T221108, T204439)]] (duration: 01m 01s) | 
  [production] | 
            
  | 10:58 | 
  <volans@deploy1001> | 
  Finished deploy [debmonitor/deploy@f049b3b]: Deploy Debmonitor v0.1.9 (duration: 01m 00s) | 
  [production] | 
            
  | 10:57 | 
  <volans@deploy1001> | 
  Started deploy [debmonitor/deploy@f049b3b]: Deploy Debmonitor v0.1.9 | 
  [production] | 
            
  | 10:40 | 
  <moritzm> | 
  installing Java security updates on kafka/analytics cluster | 
  [production] |