| 
      
        2021-09-30
      
      ยง
     | 
  
    
  | 23:41 | 
  <dpifke@deploy1002> | 
  Finished deploy [performance/coal@1be49f8]: Deploy Coal with Kafka TLS support (not yet enabled) T290131 (duration: 01m 07s) | 
  [production] | 
            
  | 23:40 | 
  <dpifke@deploy1002> | 
  Started deploy [performance/coal@1be49f8]: Deploy Coal with Kafka TLS support (not yet enabled) T290131 | 
  [production] | 
            
  | 23:39 | 
  <dpifke@deploy1002> | 
  Finished deploy [performance/navtiming@29264fb]: Deploy Navtiming with Kafka TLS support (not yet enabled) T290131 (duration: 00m 05s) | 
  [production] | 
            
  | 23:39 | 
  <dpifke@deploy1002> | 
  Started deploy [performance/navtiming@29264fb]: Deploy Navtiming with Kafka TLS support (not yet enabled) T290131 | 
  [production] | 
            
  | 23:34 | 
  <ejegg> | 
  updated Fundraising CiviCRM from d4da344274 to d74e9aa0a1 | 
  [production] | 
            
  | 22:09 | 
  <legoktm@deploy1002> | 
  helmfile [codfw] Ran 'sync' command on namespace 'shellbox-media' for release 'main' . | 
  [production] | 
            
  | 22:07 | 
  <legoktm@deploy1002> | 
  helmfile [eqiad] Ran 'sync' command on namespace 'shellbox-media' for release 'main' . | 
  [production] | 
            
  | 22:06 | 
  <legoktm@deploy1002> | 
  helmfile [staging] Ran 'sync' command on namespace 'shellbox-media' for release 'main' . | 
  [production] | 
            
  | 21:53 | 
  <legoktm@deploy1002> | 
  helmfile [staging] Ran 'sync' command on namespace 'shellbox-timeline' for release 'main' . | 
  [production] | 
            
  | 21:06 | 
  <eileen> | 
  civicrm revision changed from 2ecb8f0bcd to d4da344274, config revision is 77cb7ec866 | 
  [production] | 
            
  | 20:54 | 
  <ryankemper> | 
  [WCQS] `ryankemper@wcqs1003:~$ sudo pool` (merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/725110 to unbreak readiness probe) | 
  [production] | 
            
  | 20:54 | 
  <topranks> | 
  Routinator on rpki1001 upgraded to  0.10.0 and working again after force refresh. | 
  [production] | 
            
  | 20:49 | 
  <brennen> | 
  gitlab1001: upgrade to 14.2.5 complete | 
  [production] | 
            
  | 20:32 | 
  <brennen> | 
  gitlab2001, gitlab1001: downtime for upgrades to 14.2.5 | 
  [production] | 
            
  | 20:18 | 
  <ryankemper> | 
  [WCQS] `ryankemper@wcqs1003:~$ sudo depool` (not sure why pybal can't depool it, the other 2 servers are pooled) | 
  [production] | 
            
  | 19:51 | 
  <topranks> | 
  Updating routinator on rpki1001 T291543 | 
  [production] | 
            
  | 19:39 | 
  <legoktm@deploy1002> | 
  helmfile [codfw] Ran 'sync' command on namespace 'shellbox-timeline' for release 'main' . | 
  [production] | 
            
  | 19:38 | 
  <mwdebug-deploy@deploy1002> | 
  helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | 
  [production] | 
            
  | 19:37 | 
  <legoktm@deploy1002> | 
  helmfile [eqiad] Ran 'sync' command on namespace 'shellbox-timeline' for release 'main' . | 
  [production] | 
            
  | 19:36 | 
  <mwdebug-deploy@deploy1002> | 
  helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | 
  [production] | 
            
  | 19:26 | 
  <mwdebug-deploy@deploy1002> | 
  helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | 
  [production] | 
            
  | 19:21 | 
  <mwdebug-deploy@deploy1002> | 
  helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | 
  [production] | 
            
  | 19:14 | 
  <jhuneidi@deploy1002> | 
  rebuilt and synchronized wikiversions files: all wikis to 1.38.0-wmf.2  refs T281166 | 
  [production] | 
            
  | 19:11 | 
  <mwdebug-deploy@deploy1002> | 
  helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | 
  [production] | 
            
  | 19:08 | 
  <mwdebug-deploy@deploy1002> | 
  helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | 
  [production] | 
            
  | 19:07 | 
  <thcipriani@deploy1002> | 
  Synchronized php-1.38.0-wmf.2/extensions/MobileFrontend: Backport: [[gerrit:724979|Fix search within pages alignment (T292107)]] (duration: 01m 09s) | 
  [production] | 
            
  | 19:05 | 
  <thcipriani@deploy1002> | 
  Synchronized php-1.38.0-wmf.1/extensions/EventBus/includes/EventBus.php: Backport: [[gerrit:724481|Guard against undefined index notice when setting x-client-ip (T288853)]] (duration: 01m 09s) | 
  [production] | 
            
  | 19:04 | 
  <thcipriani@deploy1002> | 
  Synchronized php-1.38.0-wmf.2/extensions/EventBus/includes/EventBus.php: Backport: [[gerrit:724480|Guard against undefined index notice when setting x-client-ip (T288853)]] (duration: 01m 09s) | 
  [production] | 
            
  | 18:58 | 
  <mwdebug-deploy@deploy1002> | 
  helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | 
  [production] | 
            
  | 18:58 | 
  <thcipriani@deploy1002> | 
  Synchronized php-1.38.0-wmf.2/skins/Vector/resources/skins.vector.styles.legacy/components/MenuDropdown.less: Backport: [[gerrit:724798|Restore original more menu padding in legacy Vector (T289163)]] (duration: 01m 08s) | 
  [production] | 
            
  | 18:54 | 
  <mwdebug-deploy@deploy1002> | 
  helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | 
  [production] | 
            
  | 18:44 | 
  <mwdebug-deploy@deploy1002> | 
  helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | 
  [production] | 
            
  | 18:43 | 
  <thcipriani@deploy1002> | 
  Scap failed!: Call to mwscript eval.php stderr: not empty | 
  [production] | 
            
  | 18:42 | 
  <moritzm> | 
  imported gitlab 14.2.5 to thirdparty/gitlab T292219 | 
  [production] | 
            
  | 18:41 | 
  <mwdebug-deploy@deploy1002> | 
  helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | 
  [production] | 
            
  | 18:38 | 
  <thcipriani@deploy1002> | 
  Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:704167|Use Wikimania's logo in a new vector (T286405)]] Part III (duration: 01m 07s) | 
  [production] | 
            
  | 18:37 | 
  <thcipriani@deploy1002> | 
  Synchronized static/images/mobile/copyright/wikimania-wordmark.svg: Config: [[gerrit:704167|Use Wikimania's logo in a new vector (T286405)]] Part II (duration: 01m 07s) | 
  [production] | 
            
  | 18:35 | 
  <thcipriani@deploy1002> | 
  Synchronized static/images/mobile/copyright/wikimania.svg: Config: [[gerrit:704167|Use Wikimania's logo in a new vector (T286405)]] part I (duration: 01m 07s) | 
  [production] | 
            
  | 18:32 | 
  <cmooney@cumin1001> | 
  END (PASS) - Cookbook sre.dns.netbox (exit_code=0) | 
  [production] | 
            
  | 18:31 | 
  <mwdebug-deploy@deploy1002> | 
  helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | 
  [production] | 
            
  | 18:31 | 
  <thcipriani@deploy1002> | 
  Synchronized wmf-config: Config: [[gerrit:724514|Enable sticky header on beta cluster (T289721)]] (duration: 01m 08s) | 
  [production] | 
            
  | 18:29 | 
  <cmooney@cumin1001> | 
  START - Cookbook sre.dns.netbox | 
  [production] | 
            
  | 18:28 | 
  <mwdebug-deploy@deploy1002> | 
  helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . | 
  [production] | 
            
  | 18:27 | 
  <otto@cumin1001> | 
  END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts thorium.eqiad.wmnet | 
  [production] | 
            
  | 18:22 | 
  <bd808@deploy1002> | 
  helmfile [eqiad] Ran 'sync' command on namespace 'toolhub' for release 'main' . | 
  [production] | 
            
  | 18:20 | 
  <thcipriani@deploy1002> | 
  Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:724861|Disable legacy media dom on a few more wikis (T51097)]] (duration: 01m 08s) | 
  [production] | 
            
  | 18:07 | 
  <bd808@deploy1002> | 
  helmfile [staging] Ran 'sync' command on namespace 'toolhub' for release 'main' . | 
  [production] | 
            
  | 17:49 | 
  <elukey@deploy1002> | 
  helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. | 
  [production] | 
            
  | 17:49 | 
  <elukey@deploy1002> | 
  helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. | 
  [production] | 
            
  | 17:49 | 
  <otto@cumin1001> | 
  START - Cookbook sre.hosts.decommission for hosts thorium.eqiad.wmnet | 
  [production] |