|
2021-09-30
ยง
|
| 18:22 |
<bd808@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'toolhub' for release 'main' . |
[production] |
| 18:20 |
<thcipriani@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:724861|Disable legacy media dom on a few more wikis (T51097)]] (duration: 01m 08s) |
[production] |
| 18:07 |
<bd808@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'toolhub' for release 'main' . |
[production] |
| 17:49 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. |
[production] |
| 17:49 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. |
[production] |
| 17:49 |
<otto@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts thorium.eqiad.wmnet |
[production] |
| 17:42 |
<bstorm> |
updating packages for thirdparty/kubeadm-k8s-1-20 and thirdparty/kubeadm-k8s-1-19 in stretch-wikimedia on apt1001 T292131 |
[production] |
| 17:09 |
<jgiannelos@deploy1002> |
Finished deploy [kartotherian/deploy@8fbf87c] (eqiad): Increase mirrored traffic to 50% for eqiad (duration: 00m 55s) |
[production] |
| 17:08 |
<jgiannelos@deploy1002> |
Started deploy [kartotherian/deploy@8fbf87c] (eqiad): Increase mirrored traffic to 50% for eqiad |
[production] |
| 17:03 |
<jgiannelos@deploy1002> |
Finished deploy [kartotherian/deploy@8fbf87c] (eqiad): Increase mirrored traffic to 50% for eqiad (duration: 00m 08s) |
[production] |
| 17:02 |
<jgiannelos@deploy1002> |
Started deploy [kartotherian/deploy@8fbf87c] (eqiad): Increase mirrored traffic to 50% for eqiad |
[production] |
| 17:02 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality' for release 'main' . |
[production] |
| 17:00 |
<jgiannelos@deploy1002> |
Finished deploy [kartotherian/deploy@8fbf87c] (eqiad): Increase mirrored traffic to 50% for eqiad (duration: 00m 11s) |
[production] |
| 17:00 |
<jgiannelos@deploy1002> |
Started deploy [kartotherian/deploy@8fbf87c] (eqiad): Increase mirrored traffic to 50% for eqiad |
[production] |
| 16:49 |
<sukhe> |
restart dnsdist.service on doh[1001-1002,2001-2002,3001-3002,4001-4002,5001-5002].wikimedia.org |
[production] |
| 16:43 |
<jgiannelos@deploy1002> |
Finished deploy [kartotherian/deploy@67a4d22]: Increase mirrored traffic to 10% (duration: 02m 33s) |
[production] |
| 16:40 |
<jgiannelos@deploy1002> |
Started deploy [kartotherian/deploy@67a4d22]: Increase mirrored traffic to 10% |
[production] |
| 16:38 |
<jgiannelos@deploy1002> |
Finished deploy [kartotherian/deploy@67a4d22] (eqiad): Increase mirrored traffic to 10% (duration: 00m 40s) |
[production] |
| 16:37 |
<jgiannelos@deploy1002> |
Started deploy [kartotherian/deploy@67a4d22] (eqiad): Increase mirrored traffic to 10% |
[production] |
| 16:37 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 16:33 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 16:32 |
<hnowlan> |
Ran `GRANT pg_monitor TO prometheus` for maps in eqiad and codfw to fix empty prometheus connection metrics |
[production] |
| 16:30 |
<jgiannelos@deploy1002> |
Finished deploy [kartotherian/deploy@67a4d22] (eqiad): Increase mirrored traffic to 10% (duration: 00m 16s) |
[production] |
| 16:30 |
<jgiannelos@deploy1002> |
Started deploy [kartotherian/deploy@67a4d22] (eqiad): Increase mirrored traffic to 10% |
[production] |
| 16:24 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 16:20 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 16:11 |
<ladsgroup@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:725032|Disable jQuery migrate in metawiki (T280944)]] (duration: 01m 09s) |
[production] |
| 16:08 |
<ladsgroup@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:725019|Enable dispatching via job to 10 prod wikis]] (duration: 01m 09s) |
[production] |
| 15:55 |
<elukey@deploy1002> |
helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality' for release 'main' . |
[production] |
| 14:36 |
<elukey> |
drop /etc/helmfile-defaults/private/backup_old_paths from deploy1002 (old data not needed anymore) |
[production] |
| 14:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1111 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17374 and previous config saved to /var/cache/conftool/dbconfig/20210930-143325-root.json |
[production] |
| 14:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1134 (re)pooling @ 100%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17373 and previous config saved to /var/cache/conftool/dbconfig/20210930-143044-root.json |
[production] |
| 14:18 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1111 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17372 and previous config saved to /var/cache/conftool/dbconfig/20210930-141822-root.json |
[production] |
| 14:15 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1134 (re)pooling @ 75%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17370 and previous config saved to /var/cache/conftool/dbconfig/20210930-141540-root.json |
[production] |
| 14:03 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1111 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17369 and previous config saved to /var/cache/conftool/dbconfig/20210930-140318-root.json |
[production] |
| 14:00 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1134 (re)pooling @ 50%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17368 and previous config saved to /var/cache/conftool/dbconfig/20210930-140037-root.json |
[production] |
| 13:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1111 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17367 and previous config saved to /var/cache/conftool/dbconfig/20210930-134815-root.json |
[production] |
| 13:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1134 (re)pooling @ 25%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17366 and previous config saved to /var/cache/conftool/dbconfig/20210930-134533-root.json |
[production] |
| 13:42 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host testvm2001.codfw.wmnet |
[production] |
| 13:40 |
<jgiannelos@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . |
[production] |
| 13:38 |
<akosiaris@deploy1002> |
helmfile [eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 13:38 |
<akosiaris@deploy1002> |
helmfile [eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 13:37 |
<akosiaris@deploy1002> |
helmfile [codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 13:36 |
<akosiaris@deploy1002> |
helmfile [codfw] START helmfile.d/admin 'apply'. |
[production] |
| 13:33 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1111 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17365 and previous config saved to /var/cache/conftool/dbconfig/20210930-133311-root.json |
[production] |
| 13:30 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1134 (re)pooling @ 10%: After upgrade', diff saved to https://phabricator.wikimedia.org/P17364 and previous config saved to /var/cache/conftool/dbconfig/20210930-133029-root.json |
[production] |
| 13:29 |
<marostegui> |
Upgrade db1111 |
[production] |
| 13:28 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1111 for upgrade', diff saved to https://phabricator.wikimedia.org/P17363 and previous config saved to /var/cache/conftool/dbconfig/20210930-132831-marostegui.json |
[production] |
| 13:27 |
<marostegui> |
Upgrade db1134 |
[production] |
| 13:27 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1134 (s1) for upgrade', diff saved to https://phabricator.wikimedia.org/P17362 and previous config saved to /var/cache/conftool/dbconfig/20210930-132700-marostegui.json |
[production] |