|
2024-04-26
ยง
|
| 18:23 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2163 (T352010)', diff saved to https://phabricator.wikimedia.org/P61256 and previous config saved to /var/cache/conftool/dbconfig/20240426-182320-ladsgroup.json |
[production] |
| 17:57 |
<dancy@deploy1002> |
Finished scap: Testing T325530 (duration: 09m 14s) |
[production] |
| 17:48 |
<dancy@deploy1002> |
Started scap: Testing T325530 |
[production] |
| 17:47 |
<dancy@deploy1002> |
Installation of scap version "4.80.0" completed for 325 hosts |
[production] |
| 17:47 |
<dancy@deploy1002> |
Installing scap version "4.80.0" for 325 hosts |
[production] |
| 17:27 |
<bking@cumin2002> |
conftool action : set/weight=10:pooled=yes; selector: name=elastic110[3-7]\.eqiad\.wmnet |
[production] |
| 17:14 |
<eoghan@cumin1002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lists2001.wikimedia.org with OS bookworm |
[production] |
| 17:02 |
<isaranto@deploy1002> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' . |
[production] |
| 16:57 |
<bking@cumin2002> |
conftool action : set/pooled=no; selector: name=elastic1107\.eqiad\.wmnet |
[production] |
| 16:36 |
<bking@cumin2002> |
conftool action : set/weight=20:pooled=yes; selector: name=elastic1107\.eqiad\.wmnet |
[production] |
| 16:35 |
<bking@cumin2002> |
conftool action : set/pooled=no; selector: name=elastic1106\.eqiad\.wmnet |
[production] |
| 16:33 |
<bking@cumin2002> |
conftool action : set/weight=20:pooled=yes; selector: name=elastic1106\.eqiad\.wmnet |
[production] |
| 16:32 |
<bking@cumin2002> |
conftool action : set/pooled=no; selector: name=elastic1105\.eqiad\.wmnet |
[production] |
| 16:30 |
<denisse> |
Delete the unused Prometheus PoP TLS certificates in the private repository as part of the cergen to CFSSL migration - T360414 |
[production] |
| 16:22 |
<eoghan@cumin1002> |
START - Cookbook sre.hosts.reimage for host lists2001.wikimedia.org with OS bookworm |
[production] |
| 16:20 |
<elukey@deploy1002> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . |
[production] |
| 16:20 |
<eoghan@cumin1002> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host lists2001.mgmt.codfw.wmnet with reboot policy FORCED |
[production] |
| 16:19 |
<elukey@deploy1002> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . |
[production] |
| 16:18 |
<elukey@deploy1002> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' . |
[production] |
| 16:18 |
<elukey@deploy1002> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . |
[production] |
| 16:17 |
<elukey@deploy1002> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . |
[production] |
| 16:17 |
<elukey@deploy1002> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . |
[production] |
| 16:16 |
<elukey@deploy1002> |
helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . |
[production] |
| 16:06 |
<vgutierrez> |
repool ncredir6001 |
[production] |
| 15:56 |
<bking@cumin2002> |
conftool action : set/pooled=no; selector: name=elastic1104\.eqiad\.wmnet |
[production] |
| 15:56 |
<bking@cumin2002> |
conftool action : set/pooled=no; selector: name=elastic1103\.eqiad\.wmnet |
[production] |
| 15:55 |
<vgutierrez> |
depool ncredir6001 |
[production] |
| 15:53 |
<eoghan@cumin1002> |
START - Cookbook sre.hosts.provision for host lists2001.mgmt.codfw.wmnet with reboot policy FORCED |
[production] |
| 15:46 |
<denisse> |
Enabling Puppet on the Prometheus PoP hosts as part of the cergen to CFSSL migration - T360414 |
[production] |
| 15:43 |
<bking@cumin2002> |
conftool action : set/weight=20:pooled=yes; selector: name=elastic1105\.eqiad\.wmnet |
[production] |
| 15:36 |
<eoghan@cumin1002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 15:34 |
<eoghan@cumin1002> |
START - Cookbook sre.dns.netbox |
[production] |
| 15:31 |
<bking@cumin2002> |
conftool action : set/weight=20:pooled=yes; selector: name=elastic1104\.eqiad\.wmnet |
[production] |
| 15:29 |
<denisse> |
testing patch #1023917 on prometheus6002 - T360414 |
[production] |
| 15:28 |
<denisse> |
testing patch #1023917 on prometheus6002 |
[production] |
| 15:26 |
<jayme@cumin1002> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kubestagemaster2003.codfw.wmnet |
[production] |
| 15:26 |
<jayme@cumin1002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 15:25 |
<jayme@cumin1002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kubestagemaster2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin1002" |
[production] |
| 15:25 |
<denisse@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on prometheus6002.drmrs.wmnet,prometheus5002.eqsin.wmnet,prometheus3003.esams.wmnet,prometheus4002.ulsfo.wmnet with reason: Downtiming the Prometheus PoP hosts part of the cergen to CFSSL migration - T360414 |
[production] |
| 15:25 |
<denisse> |
Disabling Puppet on the Prometheus PoP hosts as part of the cergen to CFSSL migration - T360414 |
[production] |
| 15:24 |
<denisse@cumin2002> |
START - Cookbook sre.hosts.downtime for 0:30:00 on prometheus6002.drmrs.wmnet,prometheus5002.eqsin.wmnet,prometheus3003.esams.wmnet,prometheus4002.ulsfo.wmnet with reason: Downtiming the Prometheus PoP hosts part of the cergen to CFSSL migration - T360414 |
[production] |
| 15:23 |
<jayme@cumin1002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kubestagemaster2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin1002" |
[production] |
| 15:22 |
<denisse> |
Downtiming the Prometheus PoP hosts as part of the cergen to CFSSL migration - T360414 |
[production] |
| 15:20 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host lists2001 |
[production] |
| 15:19 |
<cmooney@cumin1002> |
START - Cookbook sre.network.configure-switch-interfaces for host lists2001 |
[production] |
| 15:19 |
<cmooney@cumin1002> |
END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) |
[production] |
| 15:14 |
<bking@cumin2002> |
conftool action : set/weight=20:pooled=yes; selector: name=elastic1103\.eqiad\.wmnet |
[production] |
| 15:12 |
<jayme@cumin1002> |
START - Cookbook sre.dns.netbox |
[production] |
| 15:07 |
<jayme@cumin1002> |
START - Cookbook sre.hosts.decommission for hosts kubestagemaster2003.codfw.wmnet |
[production] |
| 14:48 |
<elukey@cumin1002> |
END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase2021.codfw.wmnet: Move to PKI TLS certs - elukey@cumin1002 |
[production] |