|
2025-06-20
§
|
| 09:10 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2210 (T396130)', diff saved to https://phabricator.wikimedia.org/P78521 and previous config saved to /var/cache/conftool/dbconfig/20250620-091005-marostegui.json |
[production] |
| 09:09 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2210.codfw.wmnet with reason: Maintenance |
[production] |
| 09:09 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2206 (T396130)', diff saved to https://phabricator.wikimedia.org/P78520 and previous config saved to /var/cache/conftool/dbconfig/20250620-090943-marostegui.json |
[production] |
| 08:54 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P78519 and previous config saved to /var/cache/conftool/dbconfig/20250620-085435-marostegui.json |
[production] |
| 08:39 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P78518 and previous config saved to /var/cache/conftool/dbconfig/20250620-083928-marostegui.json |
[production] |
| 08:24 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2206 (T396130)', diff saved to https://phabricator.wikimedia.org/P78517 and previous config saved to /var/cache/conftool/dbconfig/20250620-082420-marostegui.json |
[production] |
| 08:16 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2206 (T396130)', diff saved to https://phabricator.wikimedia.org/P78516 and previous config saved to /var/cache/conftool/dbconfig/20250620-081638-marostegui.json |
[production] |
| 08:16 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2206.codfw.wmnet with reason: Maintenance |
[production] |
| 08:11 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2199.codfw.wmnet with reason: Maintenance |
[production] |
| 08:11 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2179 (T396130)', diff saved to https://phabricator.wikimedia.org/P78515 and previous config saved to /var/cache/conftool/dbconfig/20250620-081127-marostegui.json |
[production] |
| 07:59 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repool pc2 T378715', diff saved to https://phabricator.wikimedia.org/P78514 and previous config saved to /var/cache/conftool/dbconfig/20250620-075944-root.json |
[production] |
| 07:56 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P78513 and previous config saved to /var/cache/conftool/dbconfig/20250620-075619-marostegui.json |
[production] |
| 07:41 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P78512 and previous config saved to /var/cache/conftool/dbconfig/20250620-074112-marostegui.json |
[production] |
| 07:26 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2179 (T396130)', diff saved to https://phabricator.wikimedia.org/P78511 and previous config saved to /var/cache/conftool/dbconfig/20250620-072605-marostegui.json |
[production] |
| 07:20 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on es2045.codfw.wmnet with reason: Firmware downgrade pending |
[production] |
| 07:17 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2179 (T396130)', diff saved to https://phabricator.wikimedia.org/P78510 and previous config saved to /var/cache/conftool/dbconfig/20250620-071730-marostegui.json |
[production] |
| 07:17 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2179.codfw.wmnet with reason: Maintenance |
[production] |
| 07:17 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2172 (T396130)', diff saved to https://phabricator.wikimedia.org/P78509 and previous config saved to /var/cache/conftool/dbconfig/20250620-071707-marostegui.json |
[production] |
| 07:12 |
<jmm@cumin1003> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2024.codfw.wmnet |
[production] |
| 07:12 |
<jmm@cumin1003> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 07:12 |
<jmm@cumin1003> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti2024.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003" |
[production] |
| 07:11 |
<jmm@cumin1003> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti2024.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003" |
[production] |
| 07:02 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P78508 and previous config saved to /var/cache/conftool/dbconfig/20250620-070200-marostegui.json |
[production] |
| 06:56 |
<jmm@cumin1003> |
START - Cookbook sre.dns.netbox |
[production] |
| 06:51 |
<jmm@cumin1003> |
START - Cookbook sre.hosts.decommission for hosts ganeti2024.codfw.wmnet |
[production] |
| 06:50 |
<jmm@cumin1003> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2023.codfw.wmnet |
[production] |
| 06:50 |
<jmm@cumin1003> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 06:50 |
<jmm@cumin1003> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti2023.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003" |
[production] |
| 06:48 |
<jmm@cumin1003> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti2023.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin1003" |
[production] |
| 06:46 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P78507 and previous config saved to /var/cache/conftool/dbconfig/20250620-064652-marostegui.json |
[production] |
| 06:44 |
<jmm@cumin1003> |
START - Cookbook sre.dns.netbox |
[production] |
| 06:40 |
<jmm@cumin1003> |
START - Cookbook sre.hosts.decommission for hosts ganeti2023.codfw.wmnet |
[production] |
| 06:31 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2172 (T396130)', diff saved to https://phabricator.wikimedia.org/P78506 and previous config saved to /var/cache/conftool/dbconfig/20250620-063145-marostegui.json |
[production] |
| 06:23 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2172 (T396130)', diff saved to https://phabricator.wikimedia.org/P78504 and previous config saved to /var/cache/conftool/dbconfig/20250620-062307-marostegui.json |
[production] |
| 06:23 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2172.codfw.wmnet with reason: Maintenance |
[production] |
| 06:17 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2155.codfw.wmnet with reason: Maintenance |
[production] |
| 06:17 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2147 (T396130)', diff saved to https://phabricator.wikimedia.org/P78503 and previous config saved to /var/cache/conftool/dbconfig/20250620-061659-marostegui.json |
[production] |
| 06:02 |
<jmm@cumin1003> |
END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of ml-staging-etcd2002.codfw.wmnet to plain |
[production] |
| 06:02 |
<jmm@cumin1003> |
START - Cookbook sre.ganeti.changedisk for changing disk type of ml-staging-etcd2002.codfw.wmnet to plain |
[production] |
| 06:01 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P78502 and previous config saved to /var/cache/conftool/dbconfig/20250620-060151-marostegui.json |
[production] |
| 05:46 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P78501 and previous config saved to /var/cache/conftool/dbconfig/20250620-054644-marostegui.json |
[production] |
| 05:31 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2147 (T396130)', diff saved to https://phabricator.wikimedia.org/P78500 and previous config saved to /var/cache/conftool/dbconfig/20250620-053137-marostegui.json |
[production] |
| 05:23 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depooling db2147 (T396130)', diff saved to https://phabricator.wikimedia.org/P78499 and previous config saved to /var/cache/conftool/dbconfig/20250620-052300-marostegui.json |
[production] |
| 05:22 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2147.codfw.wmnet with reason: Maintenance |
[production] |
| 05:21 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 15 hosts with reason: Upgrade |
[production] |
| 05:18 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1236.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:10 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on pc1012.eqiad.wmnet with reason: Maintenance |
[production] |