|
2025-06-10
ยง
|
| 10:31 |
<cmooney@cumin1003> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add AAAA records for WMCS cloud-private IPs in eqiad - cmooney@cumin1003" |
[production] |
| 10:29 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1187 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P77469 and previous config saved to /var/cache/conftool/dbconfig/20250610-102943-root.json |
[production] |
| 10:29 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P77468 and previous config saved to /var/cache/conftool/dbconfig/20250610-102913-fceratto.json |
[production] |
| 10:29 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2032 (re)pooling @ 60%: Repooling', diff saved to https://phabricator.wikimedia.org/P77467 and previous config saved to /var/cache/conftool/dbconfig/20250610-102900-root.json |
[production] |
| 10:27 |
<cmooney@cumin1003> |
START - Cookbook sre.dns.netbox |
[production] |
| 10:26 |
<jmm@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host ganeti2041.codfw.wmnet |
[production] |
| 10:25 |
<cmooney@cumin1003> |
END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) |
[production] |
| 10:25 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2033 (re)pooling @ 40%: Repooling', diff saved to https://phabricator.wikimedia.org/P77466 and previous config saved to /var/cache/conftool/dbconfig/20250610-102538-root.json |
[production] |
| 10:22 |
<cmooney@cumin1003> |
START - Cookbook sre.dns.netbox |
[production] |
| 10:17 |
<jmm@cumin1003> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2041.codfw.wmnet |
[production] |
| 10:17 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P77465 and previous config saved to /var/cache/conftool/dbconfig/20250610-101745-marostegui.json |
[production] |
| 10:17 |
<jmm@cumin1003> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2040.codfw.wmnet |
[production] |
| 10:17 |
<jmm@cumin1003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2040.codfw.wmnet |
[production] |
| 10:16 |
<taavi> |
add cloud-private addresses to eqiad hosts T379283 |
[admin] |
| 10:14 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1187 (re)pooling @ 60%: Repooling', diff saved to https://phabricator.wikimedia.org/P77464 and previous config saved to /var/cache/conftool/dbconfig/20250610-101438-root.json |
[production] |
| 10:14 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1174 (T395241)', diff saved to https://phabricator.wikimedia.org/P77463 and previous config saved to /var/cache/conftool/dbconfig/20250610-101406-fceratto.json |
[production] |
| 10:13 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2032 (re)pooling @ 40%: Repooling', diff saved to https://phabricator.wikimedia.org/P77462 and previous config saved to /var/cache/conftool/dbconfig/20250610-101355-root.json |
[production] |
| 10:12 |
<jmm@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host ganeti2040.codfw.wmnet |
[production] |
| 10:11 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:10 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2033 (re)pooling @ 20%: Repooling', diff saved to https://phabricator.wikimedia.org/P77461 and previous config saved to /var/cache/conftool/dbconfig/20250610-101032-root.json |
[production] |
| 10:08 |
<moritzm> |
installing jinja2 security updates |
[production] |
| 10:08 |
<moritzm> |
installing ninja2 security updates |
[production] |
| 10:06 |
<jmm@cumin1003> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2040.codfw.wmnet |
[production] |
| 10:05 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db1174 (T395241)', diff saved to https://phabricator.wikimedia.org/P77460 and previous config saved to /var/cache/conftool/dbconfig/20250610-100558-fceratto.json |
[production] |
| 10:05 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1174.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:05 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1170 (T395241)', diff saved to https://phabricator.wikimedia.org/P77459 and previous config saved to /var/cache/conftool/dbconfig/20250610-100532-fceratto.json |
[production] |
| 10:02 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P77458 and previous config saved to /var/cache/conftool/dbconfig/20250610-100239-marostegui.json |
[production] |
| 10:01 |
<jmm@cumin1003> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2039.codfw.wmnet |
[production] |
| 10:00 |
<jmm@cumin1003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2039.codfw.wmnet |
[production] |
| 09:59 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1187 (re)pooling @ 40%: Repooling', diff saved to https://phabricator.wikimedia.org/P77457 and previous config saved to /var/cache/conftool/dbconfig/20250610-095933-root.json |
[production] |
| 09:58 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2032 (re)pooling @ 20%: Repooling', diff saved to https://phabricator.wikimedia.org/P77456 and previous config saved to /var/cache/conftool/dbconfig/20250610-095850-root.json |
[production] |
| 09:58 |
<fceratto@deploy1003> |
helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . |
[production] |
| 09:55 |
<jmm@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host ganeti2039.codfw.wmnet |
[production] |
| 09:55 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2033 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P77455 and previous config saved to /var/cache/conftool/dbconfig/20250610-095527-root.json |
[production] |
| 09:54 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for es2033.codfw.wmnet |
[production] |
| 09:54 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:50 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P77454 and previous config saved to /var/cache/conftool/dbconfig/20250610-095025-fceratto.json |
[production] |
| 09:50 |
<jmm@cumin1003> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2039.codfw.wmnet |
[production] |
| 09:49 |
<jmm@cumin1003> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2038.codfw.wmnet |
[production] |
| 09:49 |
<jmm@cumin1003> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2038.codfw.wmnet |
[production] |
| 09:48 |
<fnegri@cumin1003> |
conftool action : set/pooled=yes; selector: name=clouddb1019.eqiad.wmnet |
[production] |
| 09:47 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2148 (T396130)', diff saved to https://phabricator.wikimedia.org/P77453 and previous config saved to /var/cache/conftool/dbconfig/20250610-094731-marostegui.json |
[production] |
| 09:46 |
<moritzm> |
installing postgresql-15 security updates |
[production] |
| 09:45 |
<marostegui@cumin1002> |
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) es2033 - Upgrading es2033.codfw.wmnet |
[production] |
| 09:45 |
<marostegui@cumin1002> |
START - Cookbook sre.mysql.depool es2033 - Upgrading es2033.codfw.wmnet |
[production] |
| 09:44 |
<marostegui@cumin1002> |
START - Cookbook sre.mysql.upgrade for es2033.codfw.wmnet |
[production] |
| 09:44 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'db1187 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P77450 and previous config saved to /var/cache/conftool/dbconfig/20250610-094429-root.json |
[production] |
| 09:44 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Promote es2031 to es2 master T395241', diff saved to https://phabricator.wikimedia.org/P77449 and previous config saved to /var/cache/conftool/dbconfig/20250610-094401-root.json |
[production] |
| 09:43 |
<jmm@cumin1003> |
START - Cookbook sre.hosts.reboot-single for host ganeti2038.codfw.wmnet |
[production] |
| 09:43 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'es2032 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P77448 and previous config saved to /var/cache/conftool/dbconfig/20250610-094345-root.json |
[production] |