|
2026-10-20
ยง
|
| 21:24 |
<thcipriani@deploy1003> |
Locking from deployment [ALL REPOSITORIES]: Deployments locked for the week |
[production] |
| 21:14 |
<bking@cumin2003> |
END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) |
[production] |
| 21:05 |
<ladsgroup@cumin1004> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance |
[production] |
| 21:05 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db1287 (T441633)', diff saved to https://phabricator.wikimedia.org/P96937 and previous config saved to /var/cache/conftool/dbconfig/20261020-210526-ladsgroup.json |
[production] |
| 20:55 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db1287', diff saved to https://phabricator.wikimedia.org/P96935 and previous config saved to /var/cache/conftool/dbconfig/20261020-205524-ladsgroup.json |
[production] |
| 20:53 |
<bking@cumin2003> |
START - Cookbook sre.wdqs.reboot |
[production] |
| 20:53 |
<bking@cumin2003> |
END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99) |
[production] |
| 20:53 |
<bking@cumin2003> |
START - Cookbook sre.wdqs.reboot |
[production] |
| 20:45 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db1287', diff saved to https://phabricator.wikimedia.org/P96934 and previous config saved to /var/cache/conftool/dbconfig/20261020-204522-ladsgroup.json |
[production] |
| 20:39 |
<arlolra@deploy1003> |
Finished scap sync-world: Backport for [[gerrit:1357389|Undeploy Parsoid readviews on hewikisource (T441087)]] (duration: 10m 37s) |
[production] |
| 20:37 |
<bking@cumin2003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2025.codfw.wmnet |
[production] |
| 20:35 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db1287 (T441633)', diff saved to https://phabricator.wikimedia.org/P96933 and previous config saved to /var/cache/conftool/dbconfig/20261020-203520-ladsgroup.json |
[production] |
| 20:34 |
<arlolra@deploy1003> |
ssastry, arlolra: Continuing with deployment |
[production] |
| 20:33 |
<arlolra@deploy1003> |
ssastry, arlolra: Backport for [[gerrit:1357389|Undeploy Parsoid readviews on hewikisource (T441087)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 20:30 |
<bking@cumin2003> |
START - Cookbook sre.hosts.reboot-single for host wdqs2025.codfw.wmnet |
[production] |
| 20:30 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Depooling db1287 (T441633)', diff saved to https://phabricator.wikimedia.org/P96932 and previous config saved to /var/cache/conftool/dbconfig/20261020-203016-ladsgroup.json |
[production] |
| 20:30 |
<ladsgroup@cumin1004> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1287.eqiad.wmnet with reason: Maintenance |
[production] |
| 20:29 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db1282 (T441633)', diff saved to https://phabricator.wikimedia.org/P96931 and previous config saved to /var/cache/conftool/dbconfig/20261020-202957-ladsgroup.json |
[production] |
| 20:29 |
<arlolra@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1357389|Undeploy Parsoid readviews on hewikisource (T441087)]] |
[production] |
| 20:25 |
<arlolra@deploy1003> |
Finished scap sync-world: Backport for Add "proofread-index" to LINTABLE_CONTENT_MODELS (T441874) ([[gerrit:1357368|1.48.0-wmf.1]], [[gerrit:1357370|1.47.0-wmf.24]]) (duration: 18m 59s) |
[production] |
| 20:19 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db1282', diff saved to https://phabricator.wikimedia.org/P96930 and previous config saved to /var/cache/conftool/dbconfig/20261020-201954-ladsgroup.json |
[production] |
| 20:11 |
<arlolra@deploy1003> |
arlolra: Continuing with deployment |
[production] |
| 20:11 |
<bking@cumin2003> |
START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_codfw: apply kernel updates - bking@cumin2003 - T441838 |
[production] |
| 20:11 |
<bking@cumin2003> |
END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: apply kernel updates - bking@cumin2003 - T441838 |
[production] |
| 20:11 |
<bking@cumin2003> |
START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply kernel updates - bking@cumin2003 - T441838 |
[production] |
| 20:10 |
<arlolra@deploy1003> |
arlolra: Backport for Add "proofread-index" to LINTABLE_CONTENT_MODELS (T441874) ([[gerrit:1357368|1.48.0-wmf.1]], [[gerrit:1357370|1.47.0-wmf.24]]) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 20:09 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db1282', diff saved to https://phabricator.wikimedia.org/P96929 and previous config saved to /var/cache/conftool/dbconfig/20261020-200953-ladsgroup.json |
[production] |
| 20:08 |
<brett@cumin2003> |
END (PASS) - Cookbook sre.cdn.roll-reboot (exit_code=0) rolling reboot on P{cp70[11-16].magru.wmnet} and A:cp |
[production] |
| 20:08 |
<brett@cumin2003> |
cookbooks.sre.cdn.roll-reboot finished rebooting cp7016.magru.wmnet |
[production] |
| 20:06 |
<arlolra@deploy1003> |
Started scap sync-world: Backport for Add "proofread-index" to LINTABLE_CONTENT_MODELS (T441874) ([[gerrit:1357368|1.48.0-wmf.1]], [[gerrit:1357370|1.47.0-wmf.24]]) |
[production] |
| 19:59 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db1282 (T441633)', diff saved to https://phabricator.wikimedia.org/P96928 and previous config saved to /var/cache/conftool/dbconfig/20261020-195951-ladsgroup.json |
[production] |
| 19:58 |
<bking@cumin2003> |
END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) |
[production] |
| 19:54 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Depooling db1282 (T441633)', diff saved to https://phabricator.wikimedia.org/P96927 and previous config saved to /var/cache/conftool/dbconfig/20261020-195450-ladsgroup.json |
[production] |
| 19:54 |
<ladsgroup@cumin1004> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1282.eqiad.wmnet with reason: Maintenance |
[production] |
| 19:54 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db1279 (T441633)', diff saved to https://phabricator.wikimedia.org/P96926 and previous config saved to /var/cache/conftool/dbconfig/20261020-195430-ladsgroup.json |
[production] |
| 19:44 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db1279', diff saved to https://phabricator.wikimedia.org/P96925 and previous config saved to /var/cache/conftool/dbconfig/20261020-194428-ladsgroup.json |
[production] |
| 19:40 |
<bking@cumin2003> |
START - Cookbook sre.wdqs.reboot |
[production] |
| 19:34 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db1279', diff saved to https://phabricator.wikimedia.org/P96924 and previous config saved to /var/cache/conftool/dbconfig/20261020-193426-ladsgroup.json |
[production] |
| 19:28 |
<brett@cumin2003> |
cookbooks.sre.cdn.roll-reboot finished rebooting cp7015.magru.wmnet |
[production] |
| 19:24 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db1279 (T441633)', diff saved to https://phabricator.wikimedia.org/P96923 and previous config saved to /var/cache/conftool/dbconfig/20261020-192424-ladsgroup.json |
[production] |
| 19:23 |
<bking@cumin2003> |
END (PASS) - Cookbook sre.wdqs.reboot (exit_code=0) |
[production] |
| 19:19 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Depooling db1279 (T441633)', diff saved to https://phabricator.wikimedia.org/P96922 and previous config saved to /var/cache/conftool/dbconfig/20261020-191925-ladsgroup.json |
[production] |
| 19:19 |
<ladsgroup@cumin1004> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 6 hosts with reason: Maintenance |
[production] |
| 19:19 |
<ladsgroup@cumin1004> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1279.eqiad.wmnet with reason: Maintenance |
[production] |
| 19:18 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db1241 (T441633)', diff saved to https://phabricator.wikimedia.org/P96921 and previous config saved to /var/cache/conftool/dbconfig/20261020-191844-ladsgroup.json |
[production] |
| 19:08 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P96920 and previous config saved to /var/cache/conftool/dbconfig/20261020-190842-ladsgroup.json |
[production] |
| 19:06 |
<bking@cumin2003> |
END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: apply kernel updates - bking@cumin2003 - T441838 |
[production] |
| 19:05 |
<bking@cumin2003> |
START - Cookbook sre.wdqs.reboot |
[production] |
| 19:05 |
<dancy@deploy1003> |
Installation of scap version "4.302.0" completed for 158 hosts |
[production] |
| 19:03 |
<bking@cumin2003> |
END (FAIL) - Cookbook sre.wdqs.reboot (exit_code=99) |
[production] |