|
2022-04-13
ยง
|
| 13:19 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 13:19 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 13:19 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 13:19 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 13:19 |
<bking@cumin2002> |
END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: relforge testing - bking@cumin2002 - T301955 |
[production] |
| 13:16 |
<reedy@deploy1002> |
Synchronized wmf-config/CommonSettings.php: Use namespaced GerritExtDistProvider (duration: 00m 55s) |
[production] |
| 13:16 |
<bking@cumin2002> |
START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: relforge testing - bking@cumin2002 - T301955 |
[production] |
| 13:15 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1134', diff saved to https://phabricator.wikimedia.org/P24596 and previous config saved to /var/cache/conftool/dbconfig/20220413-131555-ladsgroup.json |
[production] |
| 13:15 |
<bking@cumin1001> |
END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: relforge testing - bking@cumin1001 - T301955 |
[production] |
| 13:14 |
<bking@cumin1001> |
START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: relforge testing - bking@cumin1001 - T301955 |
[production] |
| 13:13 |
<otto@deploy1002> |
Finished deploy [airflow-dags/research@b029f10]: (no justification provided) (duration: 00m 34s) |
[production] |
| 13:13 |
<bking@cumin2002> |
END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: relforge testing - bking@cumin2002 - T301955 |
[production] |
| 13:13 |
<bking@cumin2002> |
START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: relforge testing - bking@cumin2002 - T301955 |
[production] |
| 13:13 |
<otto@deploy1002> |
Started deploy [airflow-dags/research@b029f10]: (no justification provided) |
[production] |
| 13:10 |
<volans@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:05:00 on sretest[1001-1002].eqiad.wmnet with reason: testing spicerack |
[production] |
| 13:10 |
<volans@cumin2002> |
START - Cookbook sre.hosts.downtime for 0:05:00 on sretest[1001-1002].eqiad.wmnet with reason: testing spicerack |
[production] |
| 13:04 |
<volans> |
installed spicerack v2.4.1 on cumin2002 |
[production] |
| 13:00 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1134 (T298565)', diff saved to https://phabricator.wikimedia.org/P24595 and previous config saved to /var/cache/conftool/dbconfig/20220413-130050-ladsgroup.json |
[production] |
| 12:07 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1134 (T298565)', diff saved to https://phabricator.wikimedia.org/P24594 and previous config saved to /var/cache/conftool/dbconfig/20220413-120704-ladsgroup.json |
[production] |
| 12:07 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1134.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:07 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1134.eqiad.wmnet with reason: Maintenance |
[production] |
| 12:06 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1135 (T298565)', diff saved to https://phabricator.wikimedia.org/P24593 and previous config saved to /var/cache/conftool/dbconfig/20220413-120656-ladsgroup.json |
[production] |
| 11:51 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P24592 and previous config saved to /var/cache/conftool/dbconfig/20220413-115151-ladsgroup.json |
[production] |
| 11:46 |
<btullis@cumin1001> |
END (PASS) - Cookbook sre.hadoop.reboot-workers (exit_code=0) for Hadoop analytics cluster |
[production] |
| 11:40 |
<topranks> |
Remove IPv6 router-advertisement config for fxp0 management interface on cr1-drmrs. |
[production] |
| 11:38 |
<gmodena@deploy1002> |
Finished deploy [airflow-dags/research@b029f10]: (no justification provided) (duration: 00m 07s) |
[production] |
| 11:38 |
<gmodena@deploy1002> |
Started deploy [airflow-dags/research@b029f10]: (no justification provided) |
[production] |
| 11:36 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1135', diff saved to https://phabricator.wikimedia.org/P24591 and previous config saved to /var/cache/conftool/dbconfig/20220413-113645-ladsgroup.json |
[production] |
| 11:21 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1135 (T298565)', diff saved to https://phabricator.wikimedia.org/P24590 and previous config saved to /var/cache/conftool/dbconfig/20220413-112140-ladsgroup.json |
[production] |
| 10:46 |
<btullis@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/datahub: sync on main |
[production] |
| 10:46 |
<btullis@deploy1002> |
helmfile [eqiad] START helmfile.d/services/datahub: apply on main |
[production] |
| 10:42 |
<btullis@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/datahub: sync on main |
[production] |
| 10:41 |
<btullis@deploy1002> |
helmfile [codfw] START helmfile.d/services/datahub: apply on main |
[production] |
| 10:40 |
<btullis@deploy1002> |
helmfile [staging] DONE helmfile.d/services/datahub: sync on main |
[production] |
| 10:40 |
<btullis@deploy1002> |
helmfile [staging] START helmfile.d/services/datahub: apply on main |
[production] |
| 10:29 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1135 (T298565)', diff saved to https://phabricator.wikimedia.org/P24589 and previous config saved to /var/cache/conftool/dbconfig/20220413-102904-ladsgroup.json |
[production] |
| 10:29 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1135.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:29 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on db1135.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:28 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1119 (T298565)', diff saved to https://phabricator.wikimedia.org/P24588 and previous config saved to /var/cache/conftool/dbconfig/20220413-102856-ladsgroup.json |
[production] |
| 10:13 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P24587 and previous config saved to /var/cache/conftool/dbconfig/20220413-101351-ladsgroup.json |
[production] |
| 09:58 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1119', diff saved to https://phabricator.wikimedia.org/P24586 and previous config saved to /var/cache/conftool/dbconfig/20220413-095846-ladsgroup.json |
[production] |
| 09:44 |
<btullis@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/datahub: sync on main |
[production] |
| 09:43 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1119 (T298565)', diff saved to https://phabricator.wikimedia.org/P24585 and previous config saved to /var/cache/conftool/dbconfig/20220413-094341-ladsgroup.json |
[production] |
| 09:43 |
<btullis@deploy1002> |
helmfile [eqiad] START helmfile.d/services/datahub: apply on main |
[production] |
| 09:24 |
<jnuche@deploy1002> |
Finished deploy [restbase/deploy@627f7d7] (dev-cluster): (no justification provided) (duration: 02m 51s) |
[production] |
| 09:21 |
<jnuche@deploy1002> |
Started deploy [restbase/deploy@627f7d7] (dev-cluster): (no justification provided) |
[production] |
| 09:14 |
<btullis@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/datahub: sync on main |
[production] |
| 09:12 |
<btullis@deploy1002> |
helmfile [codfw] START helmfile.d/services/datahub: apply on main |
[production] |
| 08:47 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db1119 (T298565)', diff saved to https://phabricator.wikimedia.org/P24582 and previous config saved to /var/cache/conftool/dbconfig/20220413-084749-ladsgroup.json |
[production] |
| 08:47 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1119.eqiad.wmnet with reason: Maintenance |
[production] |