|
2025-10-15
ยง
|
| 16:58 |
<bking@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-test: apply |
[production] |
| 16:55 |
<sukhe@cumin1003> |
cookbooks.sre.cdn.roll-reboot finished rebooting cp6014.drmrs.wmnet |
[production] |
| 16:53 |
<sukhe@cumin1003> |
cookbooks.sre.cdn.roll-reboot finished rebooting cp6006.drmrs.wmnet |
[production] |
| 16:53 |
<bking@deploy2002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-test: apply |
[production] |
| 16:53 |
<bking@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-test: apply |
[production] |
| 16:52 |
<bking@deploy2002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-test: apply |
[production] |
| 16:52 |
<bking@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-test: apply |
[production] |
| 16:52 |
<bking@deploy2002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-test: apply |
[production] |
| 16:52 |
<bking@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-test: apply |
[production] |
| 16:49 |
<bking@deploy2002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-test: apply |
[production] |
| 16:49 |
<bking@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-test: apply |
[production] |
| 16:47 |
<btullis@deploy2002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 16:46 |
<btullis@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 16:40 |
<fceratto@cumin1003> |
END (PASS) - Cookbook sre.mysql.pool (exit_code=0) es2053 slowly with 10 steps - Pooling in new host |
[production] |
| 16:39 |
<bking@deploy2002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-test: apply |
[production] |
| 16:37 |
<bking@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-test: apply |
[production] |
| 16:37 |
<eevans@cumin1003> |
START - Cookbook sre.hosts.dhcp for host aqs1012.eqiad.wmnet |
[production] |
| 16:37 |
<bking@deploy2002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-test: apply |
[production] |
| 16:37 |
<bking@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-test: apply |
[production] |
| 16:20 |
<sukhe@cumin1003> |
cookbooks.sre.cdn.roll-reboot finished rebooting cp7013.magru.wmnet |
[production] |
| 16:19 |
<sukhe@cumin1003> |
cookbooks.sre.cdn.roll-reboot finished rebooting cp7006.magru.wmnet |
[production] |
| 16:16 |
<eevans@cumin1003> |
END (FAIL) - Cookbook sre.cassandra.roll-reboot (exit_code=1) rolling reboot on A:aqs-eqiad |
[production] |
| 16:14 |
<sukhe@cumin1003> |
cookbooks.sre.cdn.roll-reboot finished rebooting cp6013.drmrs.wmnet |
[production] |
| 16:12 |
<sukhe@cumin1003> |
cookbooks.sre.cdn.roll-reboot finished rebooting cp6005.drmrs.wmnet |
[production] |
| 15:57 |
<elukey@puppetserver1001> |
conftool action : set/pooled=true; selector: dnsdisc=kartotherian,name=codfw |
[production] |
| 15:49 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2206.codfw.wmnet onto db2247.codfw.wmnet |
[production] |
| 15:49 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db2206 gradually with 4 steps - Pool db2206.codfw.wmnet in after cloning |
[production] |
| 15:37 |
<sukhe@cumin1003> |
cookbooks.sre.cdn.roll-reboot finished rebooting cp7012.magru.wmnet |
[production] |
| 15:37 |
<sukhe@cumin1003> |
cookbooks.sre.cdn.roll-reboot finished rebooting cp7005.magru.wmnet |
[production] |
| 15:33 |
<sukhe@cumin1003> |
cookbooks.sre.cdn.roll-reboot finished rebooting cp6012.drmrs.wmnet |
[production] |
| 15:31 |
<sukhe@cumin1003> |
cookbooks.sre.cdn.roll-reboot finished rebooting cp6004.drmrs.wmnet |
[production] |
| 15:29 |
<mforns@deploy2002> |
Finished deploy [analytics/refinery@94efa6e] (thin): Regular analytics weekly train THIN [analytics/refinery@94efa6e8] (duration: 01m 06s) |
[production] |
| 15:28 |
<mforns@deploy2002> |
Started deploy [analytics/refinery@94efa6e] (thin): Regular analytics weekly train THIN [analytics/refinery@94efa6e8] |
[production] |
| 15:28 |
<mforns@deploy2002> |
Finished deploy [analytics/refinery@94efa6e]: Regular analytics weekly train [analytics/refinery@94efa6e8] (duration: 06m 37s) |
[production] |
| 15:21 |
<mforns@deploy2002> |
Started deploy [analytics/refinery@94efa6e]: Regular analytics weekly train [analytics/refinery@94efa6e8] |
[production] |
| 15:21 |
<mforns@deploy2002> |
Finished deploy [analytics/refinery@94efa6e] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@94efa6e8] (duration: 02m 17s) |
[production] |
| 15:19 |
<mforns@deploy2002> |
Started deploy [analytics/refinery@94efa6e] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@94efa6e8] |
[production] |
| 15:03 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.pool db2206 gradually with 4 steps - Pool db2206.codfw.wmnet in after cloning |
[production] |
| 14:54 |
<sukhe@cumin1003> |
cookbooks.sre.cdn.roll-reboot finished rebooting cp7004.magru.wmnet |
[production] |
| 14:54 |
<sukhe@cumin1003> |
cookbooks.sre.cdn.roll-reboot finished rebooting cp7011.magru.wmnet |
[production] |
| 14:51 |
<sukhe@cumin1003> |
cookbooks.sre.cdn.roll-reboot finished rebooting cp6011.drmrs.wmnet |
[production] |
| 14:51 |
<sukhe@cumin1003> |
cookbooks.sre.cdn.roll-reboot finished rebooting cp6003.drmrs.wmnet |
[production] |
| 14:44 |
<fceratto@cumin1003> |
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) es2033 - Depool es2033.codfw.wmnet to then clone it to es2054.codfw.wmnet - fceratto@cumin1003 |
[production] |
| 14:43 |
<fceratto@cumin1003> |
START - Cookbook sre.mysql.depool es2033 - Depool es2033.codfw.wmnet to then clone it to es2054.codfw.wmnet - fceratto@cumin1003 |
[production] |
| 14:43 |
<fceratto@cumin1003> |
START - Cookbook sre.mysql.clone_es of es2033.codfw.wmnet onto es2054.codfw.wmnet |
[production] |
| 14:41 |
<claime> |
armed keyholder on deploy[1003|2002] following reboots |
[production] |
| 14:40 |
<cgoubert@cumin1003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host deploy2002.codfw.wmnet |
[production] |
| 14:39 |
<fceratto@deploy2002> |
helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . |
[production] |
| 14:37 |
<moritzm> |
armed keyholder on cumin1002 following reboot |
[production] |
| 14:35 |
<fceratto@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on es2054.codfw.wmnet with reason: Setting up new ES host |
[production] |