|
2024-10-22
ยง
|
| 11:54 |
<jmm@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti2009.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" |
[production] |
| 11:48 |
<jmm@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 11:48 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P70477 and previous config saved to /var/cache/conftool/dbconfig/20241022-114849-ladsgroup.json |
[production] |
| 11:48 |
<kart_> |
Updated cxserver to 2024-10-22-112806-production (T357950) |
[production] |
| 11:47 |
<lucaswerkmeister-wmde@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1082180|tests: Don't depend on Message implementation details (T377778)]], [[gerrit:1082181|Update for Message/MessageValue changes (T377778)]] |
[production] |
| 11:47 |
<kartik@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/cxserver: apply |
[production] |
| 11:46 |
<kartik@deploy2002> |
helmfile [eqiad] START helmfile.d/services/cxserver: apply |
[production] |
| 11:46 |
<kartik@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/cxserver: apply |
[production] |
| 11:45 |
<kartik@deploy2002> |
helmfile [codfw] START helmfile.d/services/cxserver: apply |
[production] |
| 11:44 |
<kartik@deploy2002> |
helmfile [staging] DONE helmfile.d/services/cxserver: apply |
[production] |
| 11:44 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.decommission for hosts ganeti2009.codfw.wmnet |
[production] |
| 11:43 |
<kartik@deploy2002> |
helmfile [staging] START helmfile.d/services/cxserver: apply |
[production] |
| 11:43 |
<jayme@cumin1002> |
END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) check for host wikikube-worker2085.codfw.wmnet |
[production] |
| 11:43 |
<jayme@cumin1002> |
START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker2085.codfw.wmnet |
[production] |
| 11:41 |
<akosiaris> |
remove faidon from WMCS projects maps, visualeditor, swift, testlabs per his request. Keep the bastion project. cc paravoid |
[production] |
| 11:39 |
<klausman@cumin2002> |
START - Cookbook sre.cassandra.roll-restart for nodes matching A:ml-cache-codfw: Java 11 security updates - klausman@cumin2002 |
[production] |
| 11:34 |
<jayme@cumin1002> |
END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) check for host kubestagemaster2005.codfw.wmnet |
[production] |
| 11:34 |
<jayme@cumin1002> |
START - Cookbook sre.k8s.pool-depool-node check for host kubestagemaster2005.codfw.wmnet |
[production] |
| 11:33 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1186 (T376905)', diff saved to https://phabricator.wikimedia.org/P70476 and previous config saved to /var/cache/conftool/dbconfig/20241022-113342-ladsgroup.json |
[production] |
| 11:27 |
<moritzm> |
installing Java 11 security updates |
[production] |
| 11:24 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db1186 (T376905)', diff saved to https://phabricator.wikimedia.org/P70475 and previous config saved to /var/cache/conftool/dbconfig/20241022-112408-ladsgroup.json |
[production] |
| 11:24 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1186.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:23 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1186.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:23 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1184 (T376905)', diff saved to https://phabricator.wikimedia.org/P70474 and previous config saved to /var/cache/conftool/dbconfig/20241022-112343-ladsgroup.json |
[production] |
| 11:21 |
<hnowlan@deploy1003> |
helmfile [codfw] DONE helmfile.d/services/shellbox-video: sync |
[production] |
| 11:21 |
<hnowlan@deploy1003> |
helmfile [codfw] START helmfile.d/services/shellbox-video: sync |
[production] |
| 11:08 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P70473 and previous config saved to /var/cache/conftool/dbconfig/20241022-110836-ladsgroup.json |
[production] |
| 11:07 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2149 (re)pooling @ 100%: post clone', diff saved to https://phabricator.wikimedia.org/P70472 and previous config saved to /var/cache/conftool/dbconfig/20241022-110744-arnaudb.json |
[production] |
| 10:53 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P70471 and previous config saved to /var/cache/conftool/dbconfig/20241022-105329-ladsgroup.json |
[production] |
| 10:52 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2149 (re)pooling @ 75%: post clone', diff saved to https://phabricator.wikimedia.org/P70470 and previous config saved to /var/cache/conftool/dbconfig/20241022-105238-arnaudb.json |
[production] |
| 10:38 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1184 (T376905)', diff saved to https://phabricator.wikimedia.org/P70469 and previous config saved to /var/cache/conftool/dbconfig/20241022-103822-ladsgroup.json |
[production] |
| 10:37 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2149 (re)pooling @ 50%: post clone', diff saved to https://phabricator.wikimedia.org/P70468 and previous config saved to /var/cache/conftool/dbconfig/20241022-103733-arnaudb.json |
[production] |
| 10:29 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db1184 (T376905)', diff saved to https://phabricator.wikimedia.org/P70467 and previous config saved to /var/cache/conftool/dbconfig/20241022-102907-ladsgroup.json |
[production] |
| 10:29 |
<ladsgroup@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1184.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:28 |
<ladsgroup@cumin1002> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1184.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:28 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1169 (T376905)', diff saved to https://phabricator.wikimedia.org/P70466 and previous config saved to /var/cache/conftool/dbconfig/20241022-102843-ladsgroup.json |
[production] |
| 10:22 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2149 (re)pooling @ 25%: post clone', diff saved to https://phabricator.wikimedia.org/P70465 and previous config saved to /var/cache/conftool/dbconfig/20241022-102227-arnaudb.json |
[production] |
| 10:13 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P70464 and previous config saved to /var/cache/conftool/dbconfig/20241022-101336-ladsgroup.json |
[production] |
| 10:12 |
<brouberol@deploy2002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub: sync on production |
[production] |
| 10:08 |
<brouberol@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub: apply on production |
[production] |
| 10:07 |
<brouberol@deploy2002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s_services/services/datahub-next: sync on staging |
[production] |
| 10:04 |
<hnowlan@deploy1003> |
helmfile [codfw] DONE helmfile.d/services/shellbox-video: sync |
[production] |
| 10:04 |
<brouberol@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply on staging |
[production] |
| 10:03 |
<hnowlan@deploy1003> |
helmfile [codfw] START helmfile.d/services/shellbox-video: sync |
[production] |
| 10:03 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2149.codfw.wmnet onto db2205.codfw.wmnet |
[production] |
| 10:03 |
<andrewtavis-wmde@deploy2002> |
Finished deploy [airflow-dags/wmde@dcf019d]: (no justification provided) (duration: 00m 11s) |
[production] |
| 10:02 |
<andrewtavis-wmde@deploy2002> |
Started deploy [airflow-dags/wmde@dcf019d]: (no justification provided) |
[production] |
| 09:58 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P70463 and previous config saved to /var/cache/conftool/dbconfig/20241022-095829-ladsgroup.json |
[production] |
| 09:51 |
<brouberol@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s_services/services/datahub-next: apply on staging |
[production] |
| 09:43 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1169 (T376905)', diff saved to https://phabricator.wikimedia.org/P70461 and previous config saved to /var/cache/conftool/dbconfig/20241022-094322-ladsgroup.json |
[production] |