|
2021-09-30
ยง
|
| 20:54 |
<ryankemper> |
[WCQS] `ryankemper@wcqs1003:~$ sudo pool` (merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/725110 to unbreak readiness probe) |
[production] |
| 20:54 |
<topranks> |
Routinator on rpki1001 upgraded to 0.10.0 and working again after force refresh. |
[production] |
| 20:49 |
<brennen> |
gitlab1001: upgrade to 14.2.5 complete |
[production] |
| 20:32 |
<brennen> |
gitlab2001, gitlab1001: downtime for upgrades to 14.2.5 |
[production] |
| 20:18 |
<ryankemper> |
[WCQS] `ryankemper@wcqs1003:~$ sudo depool` (not sure why pybal can't depool it, the other 2 servers are pooled) |
[production] |
| 19:51 |
<topranks> |
Updating routinator on rpki1001 T291543 |
[production] |
| 19:39 |
<legoktm@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'shellbox-timeline' for release 'main' . |
[production] |
| 19:38 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:37 |
<legoktm@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'shellbox-timeline' for release 'main' . |
[production] |
| 19:36 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:26 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:21 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:14 |
<jhuneidi@deploy1002> |
rebuilt and synchronized wikiversions files: all wikis to 1.38.0-wmf.2 refs T281166 |
[production] |
| 19:11 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:08 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:07 |
<thcipriani@deploy1002> |
Synchronized php-1.38.0-wmf.2/extensions/MobileFrontend: Backport: [[gerrit:724979|Fix search within pages alignment (T292107)]] (duration: 01m 09s) |
[production] |
| 19:05 |
<thcipriani@deploy1002> |
Synchronized php-1.38.0-wmf.1/extensions/EventBus/includes/EventBus.php: Backport: [[gerrit:724481|Guard against undefined index notice when setting x-client-ip (T288853)]] (duration: 01m 09s) |
[production] |
| 19:04 |
<thcipriani@deploy1002> |
Synchronized php-1.38.0-wmf.2/extensions/EventBus/includes/EventBus.php: Backport: [[gerrit:724480|Guard against undefined index notice when setting x-client-ip (T288853)]] (duration: 01m 09s) |
[production] |
| 18:58 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:58 |
<thcipriani@deploy1002> |
Synchronized php-1.38.0-wmf.2/skins/Vector/resources/skins.vector.styles.legacy/components/MenuDropdown.less: Backport: [[gerrit:724798|Restore original more menu padding in legacy Vector (T289163)]] (duration: 01m 08s) |
[production] |
| 18:54 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:44 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:43 |
<thcipriani@deploy1002> |
Scap failed!: Call to mwscript eval.php stderr: not empty |
[production] |
| 18:42 |
<moritzm> |
imported gitlab 14.2.5 to thirdparty/gitlab T292219 |
[production] |
| 18:41 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:38 |
<thcipriani@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:704167|Use Wikimania's logo in a new vector (T286405)]] Part III (duration: 01m 07s) |
[production] |
| 18:37 |
<thcipriani@deploy1002> |
Synchronized static/images/mobile/copyright/wikimania-wordmark.svg: Config: [[gerrit:704167|Use Wikimania's logo in a new vector (T286405)]] Part II (duration: 01m 07s) |
[production] |
| 18:35 |
<thcipriani@deploy1002> |
Synchronized static/images/mobile/copyright/wikimania.svg: Config: [[gerrit:704167|Use Wikimania's logo in a new vector (T286405)]] part I (duration: 01m 07s) |
[production] |
| 18:32 |
<cmooney@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 18:31 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:31 |
<thcipriani@deploy1002> |
Synchronized wmf-config: Config: [[gerrit:724514|Enable sticky header on beta cluster (T289721)]] (duration: 01m 08s) |
[production] |
| 18:29 |
<cmooney@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 18:28 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 18:27 |
<otto@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts thorium.eqiad.wmnet |
[production] |
| 18:22 |
<bd808@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'toolhub' for release 'main' . |
[production] |
| 18:20 |
<thcipriani@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:724861|Disable legacy media dom on a few more wikis (T51097)]] (duration: 01m 08s) |
[production] |
| 18:07 |
<bd808@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'toolhub' for release 'main' . |
[production] |
| 17:49 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. |
[production] |
| 17:49 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. |
[production] |
| 17:49 |
<otto@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts thorium.eqiad.wmnet |
[production] |
| 17:42 |
<bstorm> |
updating packages for thirdparty/kubeadm-k8s-1-20 and thirdparty/kubeadm-k8s-1-19 in stretch-wikimedia on apt1001 T292131 |
[production] |
| 17:09 |
<jgiannelos@deploy1002> |
Finished deploy [kartotherian/deploy@8fbf87c] (eqiad): Increase mirrored traffic to 50% for eqiad (duration: 00m 55s) |
[production] |
| 17:08 |
<jgiannelos@deploy1002> |
Started deploy [kartotherian/deploy@8fbf87c] (eqiad): Increase mirrored traffic to 50% for eqiad |
[production] |
| 17:03 |
<jgiannelos@deploy1002> |
Finished deploy [kartotherian/deploy@8fbf87c] (eqiad): Increase mirrored traffic to 50% for eqiad (duration: 00m 08s) |
[production] |
| 17:02 |
<jgiannelos@deploy1002> |
Started deploy [kartotherian/deploy@8fbf87c] (eqiad): Increase mirrored traffic to 50% for eqiad |
[production] |
| 17:02 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality' for release 'main' . |
[production] |
| 17:00 |
<jgiannelos@deploy1002> |
Finished deploy [kartotherian/deploy@8fbf87c] (eqiad): Increase mirrored traffic to 50% for eqiad (duration: 00m 11s) |
[production] |
| 17:00 |
<jgiannelos@deploy1002> |
Started deploy [kartotherian/deploy@8fbf87c] (eqiad): Increase mirrored traffic to 50% for eqiad |
[production] |
| 16:49 |
<sukhe> |
restart dnsdist.service on doh[1001-1002,2001-2002,3001-3002,4001-4002,5001-5002].wikimedia.org |
[production] |
| 16:43 |
<jgiannelos@deploy1002> |
Finished deploy [kartotherian/deploy@67a4d22]: Increase mirrored traffic to 10% (duration: 02m 33s) |
[production] |