| 2021-09-30
      
      ยง | 
    
  | 19:26 | <mwdebug-deploy@deploy1002> | helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 19:21 | <mwdebug-deploy@deploy1002> | helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 19:14 | <jhuneidi@deploy1002> | rebuilt and synchronized wikiversions files: all wikis to 1.38.0-wmf.2  refs T281166 | [production] | 
            
  | 19:11 | <mwdebug-deploy@deploy1002> | helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 19:08 | <mwdebug-deploy@deploy1002> | helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 19:07 | <thcipriani@deploy1002> | Synchronized php-1.38.0-wmf.2/extensions/MobileFrontend: Backport: [[gerrit:724979|Fix search within pages alignment (T292107)]] (duration: 01m 09s) | [production] | 
            
  | 19:05 | <thcipriani@deploy1002> | Synchronized php-1.38.0-wmf.1/extensions/EventBus/includes/EventBus.php: Backport: [[gerrit:724481|Guard against undefined index notice when setting x-client-ip (T288853)]] (duration: 01m 09s) | [production] | 
            
  | 19:04 | <thcipriani@deploy1002> | Synchronized php-1.38.0-wmf.2/extensions/EventBus/includes/EventBus.php: Backport: [[gerrit:724480|Guard against undefined index notice when setting x-client-ip (T288853)]] (duration: 01m 09s) | [production] | 
            
  | 18:58 | <mwdebug-deploy@deploy1002> | helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 18:58 | <thcipriani@deploy1002> | Synchronized php-1.38.0-wmf.2/skins/Vector/resources/skins.vector.styles.legacy/components/MenuDropdown.less: Backport: [[gerrit:724798|Restore original more menu padding in legacy Vector (T289163)]] (duration: 01m 08s) | [production] | 
            
  | 18:54 | <mwdebug-deploy@deploy1002> | helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 18:44 | <mwdebug-deploy@deploy1002> | helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 18:43 | <thcipriani@deploy1002> | Scap failed!: Call to mwscript eval.php stderr: not empty | [production] | 
            
  | 18:42 | <moritzm> | imported gitlab 14.2.5 to thirdparty/gitlab T292219 | [production] | 
            
  | 18:41 | <mwdebug-deploy@deploy1002> | helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 18:38 | <thcipriani@deploy1002> | Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:704167|Use Wikimania's logo in a new vector (T286405)]] Part III (duration: 01m 07s) | [production] | 
            
  | 18:37 | <thcipriani@deploy1002> | Synchronized static/images/mobile/copyright/wikimania-wordmark.svg: Config: [[gerrit:704167|Use Wikimania's logo in a new vector (T286405)]] Part II (duration: 01m 07s) | [production] | 
            
  | 18:35 | <thcipriani@deploy1002> | Synchronized static/images/mobile/copyright/wikimania.svg: Config: [[gerrit:704167|Use Wikimania's logo in a new vector (T286405)]] part I (duration: 01m 07s) | [production] | 
            
  | 18:32 | <cmooney@cumin1001> | END (PASS) - Cookbook sre.dns.netbox (exit_code=0) | [production] | 
            
  | 18:31 | <mwdebug-deploy@deploy1002> | helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 18:31 | <thcipriani@deploy1002> | Synchronized wmf-config: Config: [[gerrit:724514|Enable sticky header on beta cluster (T289721)]] (duration: 01m 08s) | [production] | 
            
  | 18:29 | <cmooney@cumin1001> | START - Cookbook sre.dns.netbox | [production] | 
            
  | 18:28 | <mwdebug-deploy@deploy1002> | helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 18:27 | <otto@cumin1001> | END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts thorium.eqiad.wmnet | [production] | 
            
  | 18:22 | <bd808@deploy1002> | helmfile [eqiad] Ran 'sync' command on namespace 'toolhub' for release 'main' . | [production] | 
            
  | 18:20 | <thcipriani@deploy1002> | Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:724861|Disable legacy media dom on a few more wikis (T51097)]] (duration: 01m 08s) | [production] | 
            
  | 18:07 | <bd808@deploy1002> | helmfile [staging] Ran 'sync' command on namespace 'toolhub' for release 'main' . | [production] | 
            
  | 17:49 | <elukey@deploy1002> | helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. | [production] | 
            
  | 17:49 | <elukey@deploy1002> | helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. | [production] | 
            
  | 17:49 | <otto@cumin1001> | START - Cookbook sre.hosts.decommission for hosts thorium.eqiad.wmnet | [production] | 
            
  | 17:42 | <bstorm> | updating packages for thirdparty/kubeadm-k8s-1-20 and thirdparty/kubeadm-k8s-1-19 in stretch-wikimedia on apt1001 T292131 | [production] | 
            
  | 17:09 | <jgiannelos@deploy1002> | Finished deploy [kartotherian/deploy@8fbf87c] (eqiad): Increase mirrored traffic to 50% for eqiad (duration: 00m 55s) | [production] | 
            
  | 17:08 | <jgiannelos@deploy1002> | Started deploy [kartotherian/deploy@8fbf87c] (eqiad): Increase mirrored traffic to 50% for eqiad | [production] | 
            
  | 17:03 | <jgiannelos@deploy1002> | Finished deploy [kartotherian/deploy@8fbf87c] (eqiad): Increase mirrored traffic to 50% for eqiad (duration: 00m 08s) | [production] | 
            
  | 17:02 | <jgiannelos@deploy1002> | Started deploy [kartotherian/deploy@8fbf87c] (eqiad): Increase mirrored traffic to 50% for eqiad | [production] | 
            
  | 17:02 | <elukey@deploy1002> | helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality' for release 'main' . | [production] | 
            
  | 17:00 | <jgiannelos@deploy1002> | Finished deploy [kartotherian/deploy@8fbf87c] (eqiad): Increase mirrored traffic to 50% for eqiad (duration: 00m 11s) | [production] | 
            
  | 17:00 | <jgiannelos@deploy1002> | Started deploy [kartotherian/deploy@8fbf87c] (eqiad): Increase mirrored traffic to 50% for eqiad | [production] | 
            
  | 16:49 | <sukhe> | restart dnsdist.service on doh[1001-1002,2001-2002,3001-3002,4001-4002,5001-5002].wikimedia.org | [production] | 
            
  | 16:43 | <jgiannelos@deploy1002> | Finished deploy [kartotherian/deploy@67a4d22]: Increase mirrored traffic to 10% (duration: 02m 33s) | [production] | 
            
  | 16:40 | <jgiannelos@deploy1002> | Started deploy [kartotherian/deploy@67a4d22]: Increase mirrored traffic to 10% | [production] | 
            
  | 16:38 | <jgiannelos@deploy1002> | Finished deploy [kartotherian/deploy@67a4d22] (eqiad): Increase mirrored traffic to 10% (duration: 00m 40s) | [production] | 
            
  | 16:37 | <jgiannelos@deploy1002> | Started deploy [kartotherian/deploy@67a4d22] (eqiad): Increase mirrored traffic to 10% | [production] | 
            
  | 16:37 | <mwdebug-deploy@deploy1002> | helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 16:33 | <mwdebug-deploy@deploy1002> | helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 16:32 | <hnowlan> | Ran `GRANT pg_monitor TO prometheus` for maps in eqiad and codfw to fix empty prometheus connection metrics | [production] | 
            
  | 16:30 | <jgiannelos@deploy1002> | Finished deploy [kartotherian/deploy@67a4d22] (eqiad): Increase mirrored traffic to 10% (duration: 00m 16s) | [production] | 
            
  | 16:30 | <jgiannelos@deploy1002> | Started deploy [kartotherian/deploy@67a4d22] (eqiad): Increase mirrored traffic to 10% | [production] | 
            
  | 16:24 | <mwdebug-deploy@deploy1002> | helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] | 
            
  | 16:20 | <mwdebug-deploy@deploy1002> | helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | [production] |