|
2026-10-20
ยง
|
| 20:29 |
<arlolra@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1357389|Undeploy Parsoid readviews on hewikisource (T441087)]] |
[production] |
| 20:25 |
<arlolra@deploy1003> |
Finished scap sync-world: Backport for Add "proofread-index" to LINTABLE_CONTENT_MODELS (T441874) ([[gerrit:1357368|1.48.0-wmf.1]], [[gerrit:1357370|1.47.0-wmf.24]]) (duration: 18m 59s) |
[production] |
| 20:19 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db1282', diff saved to https://phabricator.wikimedia.org/P96930 and previous config saved to /var/cache/conftool/dbconfig/20261020-201954-ladsgroup.json |
[production] |
| 20:11 |
<arlolra@deploy1003> |
arlolra: Continuing with deployment |
[production] |
| 20:11 |
<bking@cumin2003> |
START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply kernel updates - bking@cumin2003 - T441838 |
[production] |
| 20:11 |
<bking@cumin2003> |
END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: apply kernel updates - bking@cumin2003 - T441838 |
[production] |
| 20:11 |
<bking@cumin2003> |
START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply kernel updates - bking@cumin2003 - T441838 |
[production] |
| 20:10 |
<arlolra@deploy1003> |
arlolra: Backport for Add "proofread-index" to LINTABLE_CONTENT_MODELS (T441874) ([[gerrit:1357368|1.48.0-wmf.1]], [[gerrit:1357370|1.47.0-wmf.24]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 20:09 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db1282', diff saved to https://phabricator.wikimedia.org/P96929 and previous config saved to /var/cache/conftool/dbconfig/20261020-200953-ladsgroup.json |
[production] |
| 20:08 |
<brett@cumin2003> |
END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp70[11-16].magru.wmnet} and A:cp |
[production] |
| 20:08 |
<brett@cumin2003> |
cookbooks.sre.cdn.roll-reboot finished rebooting cp7016.magru.wmnet |
[production] |
| 20:06 |
<arlolra@deploy1003> |
Started scap sync-world: Backport for Add "proofread-index" to LINTABLE_CONTENT_MODELS (T441874) ([[gerrit:1357368|1.48.0-wmf.1]], [[gerrit:1357370|1.47.0-wmf.24]]) |
[production] |
| 19:59 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db1282 (T441633)', diff saved to https://phabricator.wikimedia.org/P96928 and previous config saved to /var/cache/conftool/dbconfig/20261020-195951-ladsgroup.json |
[production] |
| 19:58 |
<bking@cumin2003> |
END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) |
[production] |
| 19:54 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Depooling db1282 (T441633)', diff saved to https://phabricator.wikimedia.org/P96927 and previous config saved to /var/cache/conftool/dbconfig/20261020-195450-ladsgroup.json |
[production] |
| 19:54 |
<ladsgroup@cumin1004> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1282.eqiad.wmnet with reason: Maintenance |
[production] |
| 19:54 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db1279 (T441633)', diff saved to https://phabricator.wikimedia.org/P96926 and previous config saved to /var/cache/conftool/dbconfig/20261020-195430-ladsgroup.json |
[production] |
| 19:44 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db1279', diff saved to https://phabricator.wikimedia.org/P96925 and previous config saved to /var/cache/conftool/dbconfig/20261020-194428-ladsgroup.json |
[production] |
| 19:40 |
<bking@cumin2003> |
START - Cookbook sre.wdqs.reboot |
[production] |
| 19:34 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db1279', diff saved to https://phabricator.wikimedia.org/P96924 and previous config saved to /var/cache/conftool/dbconfig/20261020-193426-ladsgroup.json |
[production] |
| 19:28 |
<brett@cumin2003> |
cookbooks.sre.cdn.roll-reboot finished rebooting cp7015.magru.wmnet |
[production] |
| 19:24 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db1279 (T441633)', diff saved to https://phabricator.wikimedia.org/P96923 and previous config saved to /var/cache/conftool/dbconfig/20261020-192424-ladsgroup.json |
[production] |
| 19:23 |
<bking@cumin2003> |
END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) |
[production] |
| 19:19 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Depooling db1279 (T441633)', diff saved to https://phabricator.wikimedia.org/P96922 and previous config saved to /var/cache/conftool/dbconfig/20261020-191925-ladsgroup.json |
[production] |
| 19:19 |
<ladsgroup@cumin1004> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance |
[production] |
| 19:19 |
<ladsgroup@cumin1004> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1279.eqiad.wmnet with reason: Maintenance |
[production] |
| 19:18 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db1241 (T441633)', diff saved to https://phabricator.wikimedia.org/P96921 and previous config saved to /var/cache/conftool/dbconfig/20261020-191844-ladsgroup.json |
[production] |
| 19:08 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P96920 and previous config saved to /var/cache/conftool/dbconfig/20261020-190842-ladsgroup.json |
[production] |
| 19:06 |
<bking@cumin2003> |
END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply kernel updates - bking@cumin2003 - T441838 |
[production] |
| 19:05 |
<bking@cumin2003> |
START - Cookbook sre.wdqs.reboot |
[production] |
| 19:05 |
<dancy@deploy1003> |
Installation of scap version "4.302.0" completed for 158 hosts |
[production] |
| 19:03 |
<bking@cumin2003> |
END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99) |
[production] |
| 19:03 |
<bking@cumin2003> |
START - Cookbook sre.wdqs.reboot |
[production] |
| 19:02 |
<bking@cumin2003> |
START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: apply kernel updates - bking@cumin2003 - T441838 |
[production] |
| 19:01 |
<dancy@deploy1003> |
Installing scap version "4.302.0" for 158 host(s) |
[production] |
| 18:58 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P96919 and previous config saved to /var/cache/conftool/dbconfig/20261020-185840-ladsgroup.json |
[production] |
| 18:49 |
<brett@cumin2003> |
cookbooks.sre.cdn.roll-reboot finished rebooting cp7014.magru.wmnet |
[production] |
| 18:48 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db1241 (T441633)', diff saved to https://phabricator.wikimedia.org/P96918 and previous config saved to /var/cache/conftool/dbconfig/20261020-184838-ladsgroup.json |
[production] |
| 18:48 |
<brett@cumin2003> |
END (PASS) - Cookbook sre.cdn.roll-restart-reboot-ncredir (exit_code=0) rolling reboot on A:ncredir |
[production] |
| 18:41 |
<bking@cumin2003> |
START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply kernel updates - bking@cumin2003 - T441838 |
[production] |
| 18:36 |
<ladsgroup@cumin1004> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1225.eqiad.wmnet with reason: Maintenance |
[production] |
| 18:35 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db1201 (T441633)', diff saved to https://phabricator.wikimedia.org/P96916 and previous config saved to /var/cache/conftool/dbconfig/20261020-183542-ladsgroup.json |
[production] |
| 18:25 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P96915 and previous config saved to /var/cache/conftool/dbconfig/20261020-182540-ladsgroup.json |
[production] |
| 18:24 |
<bking@cumin2003> |
END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster relforge: apply kernel updates - bking@cumin2003 - T441838 |
[production] |
| 18:15 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P96914 and previous config saved to /var/cache/conftool/dbconfig/20261020-181538-ladsgroup.json |
[production] |
| 18:14 |
<dancy@deploy1003> |
rebuilt and synchronized wikiversions files: group0 to 1.48.0-wmf.1 refs T438222 |
[production] |
| 18:10 |
<bking@cumin2003> |
START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster relforge: apply kernel updates - bking@cumin2003 - T441838 |
[production] |
| 18:07 |
<brett@cumin2003> |
cookbooks.sre.cdn.roll-reboot finished rebooting cp7013.magru.wmnet |
[production] |
| 18:05 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db1201 (T441633)', diff saved to https://phabricator.wikimedia.org/P96912 and previous config saved to /var/cache/conftool/dbconfig/20261020-180536-ladsgroup.json |
[production] |
| 17:58 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Depooling db1201 (T441633)', diff saved to https://phabricator.wikimedia.org/P96911 and previous config saved to /var/cache/conftool/dbconfig/20261020-175833-ladsgroup.json |
[production] |