|
2022-05-22
§
|
| 18:02 |
<Krinkle> |
Upgrade webservice from kubernetes php7.2 to php7.4 |
[tools.coverme] |
| 18:01 |
<Krinkle> |
Switch webservice from old grid default to kubernetes and update to php7.4 image |
[tools.guc] |
| 17:14 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1138 (T298555)', diff saved to https://phabricator.wikimedia.org/P28274 and previous config saved to /var/cache/conftool/dbconfig/20220522-171444-ladsgroup.json |
[production] |
| 17:04 |
<taavi> |
failover tools-redis to the updated cluster T278541 |
[tools] |
| 16:42 |
<wm-bot2> |
removing grid node tools-sgeexec-0940.tools.eqiad1.wikimedia.cloud (T308982) - cookbook ran by taavi@runko |
[tools] |
| 15:37 |
<bd808> |
Update demo server to 4eed41 |
[toolhub] |
| 15:36 |
<wm-bot> |
<bd808> Restart to clear errors reported on irc |
[tools.sal] |
| 14:49 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1138 (T298555)', diff saved to https://phabricator.wikimedia.org/P28273 and previous config saved to /var/cache/conftool/dbconfig/20220522-144855-ladsgroup.json |
[production] |
| 14:48 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1138.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:48 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 10:00:00 on db1138.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:48 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1144:3314 (T298555)', diff saved to https://phabricator.wikimedia.org/P28272 and previous config saved to /var/cache/conftool/dbconfig/20220522-144847-ladsgroup.json |
[production] |
| 14:27 |
<krinkle@deploy1002> |
Synchronized src/: Ia0a6d4794faaafc (duration: 00m 50s) |
[production] |
| 14:23 |
<krinkle@deploy1002> |
Synchronized docroot/noc/: Ia0a6d4794faaafc (duration: 00m 50s) |
[production] |
| 14:18 |
<krinkle@deploy1002> |
Synchronized wmf-config/: Ia0a6d4794faaafcb (2/2) (duration: 00m 42s) |
[production] |
| 14:15 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 14:14 |
<krinkle@deploy1002> |
scap failed: average error rate on 3/8 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org for details) |
[production] |
| 14:14 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 14:14 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 14:12 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 14:11 |
<krinkle@deploy1002> |
Synchronized multiversion/: Ia0a6d4794faaafcb (1/2) (duration: 00m 50s) |
[production] |
| 14:07 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 14:03 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 14:03 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 14:02 |
<krinkle@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: I31b1bfb1808b9523 (duration: 00m 52s) |
[production] |
| 13:59 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 13:44 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 13:43 |
<wm-bot> |
<lucaswerkmeister> deployed 15ade38230 (Diffusion → GitLab) |
[tools.wd-image-positions] |
| 13:40 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 13:40 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 13:36 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 13:28 |
<krinkle@deploy1002> |
Synchronized multiversion/: I3759179dba75a9419 (duration: 00m 53s) |
[production] |
| 13:25 |
<krinkle@deploy1002> |
Synchronized wmf-config/CommonSettings.php: I97878f8e6 (duration: 00m 50s) |
[production] |
| 13:21 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 13:20 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 13:20 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 13:19 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 13:18 |
<krinkle@deploy1002> |
Scap failed!: 7/8 canaries failed their endpoint checks(https://en.wikipedia.org). WARNING: canaries have not been rolled back. |
[production] |
| 13:17 |
<krinkle@deploy1002> |
scap failed: average error rate on 7/8 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org for details) |
[production] |
| 12:59 |
<Rook> |
deploying T308862: Fix version of nbvconvert #158 47a7db6b320e70a4183b0a1891096ec5b00129fe |
[paws] |
| 12:51 |
<wm-bot> |
<lucaswerkmeister> deployed 2264a649ae (flake8, updated dependencies) |
[tools.wd-image-positions] |
| 12:24 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1144:3314 (T298555)', diff saved to https://phabricator.wikimedia.org/P28270 and previous config saved to /var/cache/conftool/dbconfig/20220522-122410-ladsgroup.json |
[production] |
| 12:24 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1144.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:24 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 10:00:00 on db1144.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:24 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1149 (T298555)', diff saved to https://phabricator.wikimedia.org/P28269 and previous config saved to /var/cache/conftool/dbconfig/20220522-122402-ladsgroup.json |
[production] |
| 10:04 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1149 (T298555)', diff saved to https://phabricator.wikimedia.org/P28267 and previous config saved to /var/cache/conftool/dbconfig/20220522-100436-ladsgroup.json |
[production] |
| 10:04 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10:00:00 on db1149.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:04 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 10:00:00 on db1149.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:04 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1148 (T298555)', diff saved to https://phabricator.wikimedia.org/P28266 and previous config saved to /var/cache/conftool/dbconfig/20220522-100429-ladsgroup.json |
[production] |
| 09:53 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1167 (T303603)', diff saved to https://phabricator.wikimedia.org/P28265 and previous config saved to /var/cache/conftool/dbconfig/20220522-095327-ladsgroup.json |
[production] |
| 09:49 |
<wm-bot> |
<lucaswerkmeister> deployed 4630899d1e (tweak message) |
[tools.wd-image-positions] |