|
2024-03-27
ยง
|
| 12:33 |
<brouberol> |
redeploying external-services in all k8s clusters to account for the newly exposed ml-cassandra cluster - T360428 |
[production] |
| 12:33 |
<brouberol@deploy1002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 12:33 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2216 (re)pooling @ 10%: Post clone (dst)', diff saved to https://phabricator.wikimedia.org/P58949 and previous config saved to /var/cache/conftool/dbconfig/20240327-123258-arnaudb.json |
[production] |
| 12:32 |
<brouberol@deploy1002> |
helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 12:30 |
<brouberol@deploy1002> |
helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 12:30 |
<brouberol@deploy1002> |
helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 12:28 |
<brouberol@deploy1002> |
helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 12:27 |
<brouberol@deploy1002> |
helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 12:26 |
<brouberol@deploy1002> |
helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 12:26 |
<brouberol@deploy1002> |
helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 12:25 |
<brouberol@deploy1002> |
helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 12:24 |
<brouberol@deploy1002> |
helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 12:21 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2119 (re)pooling @ 100%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P58948 and previous config saved to /var/cache/conftool/dbconfig/20240327-122136-arnaudb.json |
[production] |
| 12:17 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2216 (re)pooling @ 5%: Post clone (dst)', diff saved to https://phabricator.wikimedia.org/P58947 and previous config saved to /var/cache/conftool/dbconfig/20240327-121752-arnaudb.json |
[production] |
| 12:15 |
<sukhe> |
running authdns-update to repool esams |
[production] |
| 12:07 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=yes; selector: dc=esams,cluster=cache_text |
[production] |
| 12:06 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2119 (re)pooling @ 75%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P58946 and previous config saved to /var/cache/conftool/dbconfig/20240327-120630-arnaudb.json |
[production] |
| 12:04 |
<sukhe@cumin2002> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for 8 hosts |
[production] |
| 12:04 |
<sukhe@cumin2002> |
START - Cookbook sre.hosts.remove-downtime for 8 hosts |
[production] |
| 11:51 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2119 (re)pooling @ 50%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P58945 and previous config saved to /var/cache/conftool/dbconfig/20240327-115125-arnaudb.json |
[production] |
| 11:44 |
<elukey@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=registry2004.codfw.wmnet |
[production] |
| 11:41 |
<elukey> |
run `apt-get clean` on registry2004 to free some space on the root partition |
[production] |
| 11:39 |
<elukey@cumin1002> |
END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM registry2004.codfw.wmnet |
[production] |
| 11:39 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2120 (re)pooling @ 100%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P58944 and previous config saved to /var/cache/conftool/dbconfig/20240327-113911-arnaudb.json |
[production] |
| 11:24 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2120 (re)pooling @ 75%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P58942 and previous config saved to /var/cache/conftool/dbconfig/20240327-112405-arnaudb.json |
[production] |
| 11:16 |
<elukey@cumin1002> |
START - Cookbook sre.ganeti.reboot-vm for VM registry2003.codfw.wmnet |
[production] |
| 11:15 |
<elukey> |
expand vram for registry200[3,4] from 4G to 6G - T360637 |
[production] |
| 11:13 |
<brouberol@deploy1002> |
helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 11:13 |
<brouberol@deploy1002> |
helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 11:12 |
<elukey@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=registry2003.codfw.wmnet |
[production] |
| 11:11 |
<elukey@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on registry2004.codfw.wmnet with reason: Increase tmpfs for nginx |
[production] |
| 11:11 |
<elukey@cumin1002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on registry2004.codfw.wmnet with reason: Increase tmpfs for nginx |
[production] |
| 11:11 |
<elukey@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on registry2003.codfw.wmnet with reason: Increase tmpfs for nginx |
[production] |
| 11:10 |
<elukey@cumin1002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on registry2003.codfw.wmnet with reason: Increase tmpfs for nginx |
[production] |
| 11:09 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2120 (re)pooling @ 50%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P58941 and previous config saved to /var/cache/conftool/dbconfig/20240327-110858-arnaudb.json |
[production] |
| 10:53 |
<arnaudb@cumin1002> |
dbctl commit (dc=all): 'db2120 (re)pooling @ 25%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P58940 and previous config saved to /var/cache/conftool/dbconfig/20240327-105353-arnaudb.json |
[production] |
| 10:51 |
<arnaudb@cumin1002> |
END (PASS) - Cookbook sre.mysql.clone (exit_code=0) Will create a clone of db2120.codfw.wmnet onto db2220.codfw.wmnet |
[production] |
| 10:31 |
<jnuche@deploy1002> |
Finished deploy [releng/jenkins-deploy@661e531] (releasing): (no justification provided) (duration: 00m 40s) |
[production] |
| 10:31 |
<jnuche@deploy1002> |
Started deploy [releng/jenkins-deploy@661e531] (releasing): (no justification provided) |
[production] |
| 10:28 |
<jelto@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on releases1003.eqiad.wmnet with reason: Troubleshooting jenkins update |
[production] |
| 10:28 |
<jelto@cumin1002> |
START - Cookbook sre.hosts.downtime for 1:00:00 on releases1003.eqiad.wmnet with reason: Troubleshooting jenkins update |
[production] |
| 10:25 |
<brouberol@deploy1002> |
helmfile [codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 10:25 |
<brouberol@deploy1002> |
helmfile [codfw] START helmfile.d/admin 'apply'. |
[production] |
| 10:23 |
<brouberol@deploy1002> |
helmfile [eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 10:23 |
<brouberol@deploy1002> |
helmfile [eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 10:20 |
<klausman@deploy1002> |
helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 10:19 |
<klausman@deploy1002> |
helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 10:19 |
<brouberol@deploy1002> |
helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 10:19 |
<brouberol@deploy1002> |
helmfile [staging-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 10:14 |
<jnuche@deploy1002> |
Finished deploy [releng/jenkins-deploy@661e531] (releasing): (no justification provided) (duration: 01m 46s) |
[production] |