|
2024-10-22
ยง
|
| 11:39 |
<klausman@cumin2002> |
START - Cookbook sre.cassandra.roll-restart for nodes matching A:ml-cache-codfw: Java 11 security updates - klausman@cumin2002 |
[production] |
| 11:34 |
<jayme@cumin1002> |
END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) check for host kubestagemaster2005.codfw.wmnet |
[production] |
| 11:34 |
<jayme@cumin1002> |
START - Cookbook sre.k8s.pool-depool-node check for host kubestagemaster2005.codfw.wmnet |
[production] |
| 11:33 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1186 (T376905)', diff saved to https://phabricator.wikimedia.org/P70476 and previous config saved to /var/cache/conftool/dbconfig/20241022-113342-ladsgroup.json |
[production] |
| 11:27 |
<moritzm> |
installing Java 11 security updates |
[production] |
| 11:24 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db1186 (T376905)', diff saved to https://phabricator.wikimedia.org/P70475 and previous config saved to /var/cache/conftool/dbconfig/20241022-112408-ladsgroup.json |
[production] |
| 11:24 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1186.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:23 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1186.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:23 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1184 (T376905)', diff saved to https://phabricator.wikimedia.org/P70474 and previous config saved to /var/cache/conftool/dbconfig/20241022-112343-ladsgroup.json |
[production] |
| 11:21 |
<hnowlan@deploy1003> |
helmfile [codfw] DONE helmfile.d/services/shellbox-video: sync |
[production] |
| 11:21 |
<hnowlan@deploy1003> |
helmfile [codfw] START helmfile.d/services/shellbox-video: sync |
[production] |
| 11:08 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P70473 and previous config saved to /var/cache/conftool/dbconfig/20241022-110836-ladsgroup.json |
[production] |
| 11:07 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2149 (re)pooling @ 100%: post clone', diff saved to https://phabricator.wikimedia.org/P70472 and previous config saved to /var/cache/conftool/dbconfig/20241022-110744-arnaudb.json |
[production] |
| 10:53 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P70471 and previous config saved to /var/cache/conftool/dbconfig/20241022-105329-ladsgroup.json |
[production] |
| 10:52 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2149 (re)pooling @ 75%: post clone', diff saved to https://phabricator.wikimedia.org/P70470 and previous config saved to /var/cache/conftool/dbconfig/20241022-105238-arnaudb.json |
[production] |
| 10:38 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1184 (T376905)', diff saved to https://phabricator.wikimedia.org/P70469 and previous config saved to /var/cache/conftool/dbconfig/20241022-103822-ladsgroup.json |
[production] |
| 10:37 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2149 (re)pooling @ 50%: post clone', diff saved to https://phabricator.wikimedia.org/P70468 and previous config saved to /var/cache/conftool/dbconfig/20241022-103733-arnaudb.json |
[production] |
| 10:29 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db1184 (T376905)', diff saved to https://phabricator.wikimedia.org/P70467 and previous config saved to /var/cache/conftool/dbconfig/20241022-102907-ladsgroup.json |
[production] |
| 10:29 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1184.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:28 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1184.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:28 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1169 (T376905)', diff saved to https://phabricator.wikimedia.org/P70466 and previous config saved to /var/cache/conftool/dbconfig/20241022-102843-ladsgroup.json |
[production] |
| 10:22 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2149 (re)pooling @ 25%: post clone', diff saved to https://phabricator.wikimedia.org/P70465 and previous config saved to /var/cache/conftool/dbconfig/20241022-102227-arnaudb.json |
[production] |
| 10:13 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P70464 and previous config saved to /var/cache/conftool/dbconfig/20241022-101336-ladsgroup.json |
[production] |
| 10:12 |
<brouberol@deploy2002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub: sync on production |
[production] |
| 10:08 |
<brouberol@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub: apply on production |
[production] |
| 10:07 |
<brouberol@deploy2002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: sync on staging |
[production] |
| 10:04 |
<hnowlan@deploy1003> |
helmfile [codfw] DONE helmfile.d/services/shellbox-video: sync |
[production] |
| 10:04 |
<brouberol@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply on staging |
[production] |
| 10:03 |
<hnowlan@deploy1003> |
helmfile [codfw] START helmfile.d/services/shellbox-video: sync |
[production] |
| 10:03 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2149.codfw.wmnet onto db2205.codfw.wmnet |
[production] |
| 10:03 |
<andrewtavis-wmde@deploy2002> |
Finished deploy [airflow-dags/wmde@dcf019d]: (no justification provided) (duration: 00m 11s) |
[production] |
| 10:02 |
<andrewtavis-wmde@deploy2002> |
Started deploy [airflow-dags/wmde@dcf019d]: (no justification provided) |
[production] |
| 09:58 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P70463 and previous config saved to /var/cache/conftool/dbconfig/20241022-095829-ladsgroup.json |
[production] |
| 09:51 |
<brouberol@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply on staging |
[production] |
| 09:43 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1169 (T376905)', diff saved to https://phabricator.wikimedia.org/P70461 and previous config saved to /var/cache/conftool/dbconfig/20241022-094322-ladsgroup.json |
[production] |
| 09:36 |
<brouberol@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub: apply on production |
[production] |
| 09:33 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db1169 (T376905)', diff saved to https://phabricator.wikimedia.org/P70460 and previous config saved to /var/cache/conftool/dbconfig/20241022-093345-ladsgroup.json |
[production] |
| 09:33 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1169.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:33 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1169.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:32 |
<brouberol@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply on staging |
[production] |
| 09:28 |
<elukey@cumin1002> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be2082.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 09:27 |
<elukey@cumin1002> |
START - Cookbook sre.hosts.provision for host ms-be2082.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 09:22 |
<hashar> |
Restarting CI Jenkins |
[production] |
| 09:06 |
<hashar> |
Restarting Gerrit |
[production] |
| 08:57 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on pc1013.eqiad.wmnet with reason: maintenance |
[production] |
| 08:57 |
<arnaudb@cumin1002> |
START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on pc1013.eqiad.wmnet with reason: maintenance |
[production] |
| 08:37 |
<arnaudb@cumin1002> |
START - Cookbook sre.mysql.clone of db2149.codfw.wmnet onto db2205.codfw.wmnet |
[production] |
| 08:35 |
<elukey@cumin1002> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be2082.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 08:34 |
<elukey@cumin1002> |
START - Cookbook sre.hosts.provision for host ms-be2082.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART |
[production] |
| 08:33 |
<elukey@cumin1002> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ms-be2082.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART |
[production] |