1-50 of 10000 results (101ms)
2026-04-24 ยง
10:32 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P91424 and previous config saved to /var/cache/conftool/dbconfig/20260424-103202-fceratto.json [production]
10:31 <fceratto@cumin1003> dbctl commit (dc=all): 'Depooling db1241 (T419961)', diff saved to https://phabricator.wikimedia.org/P91423 and previous config saved to /var/cache/conftool/dbconfig/20260424-103146-fceratto.json [production]
10:31 <fceratto@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1241.eqiad.wmnet with reason: Maintenance [production]
10:31 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1238 (T419961)', diff saved to https://phabricator.wikimedia.org/P91422 and previous config saved to /var/cache/conftool/dbconfig/20260424-103116-fceratto.json [production]
10:30 <klausman@cumin1003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve1015.eqiad.wmnet [production]
10:26 <klausman@cumin1003> START - Cookbook sre.hosts.reboot-single for host ml-serve1015.eqiad.wmnet [production]
10:22 <marostegui@cumin1003> START - Cookbook sre.hosts.reimage for host db2228.codfw.wmnet with OS trixie [production]
10:22 <marostegui@cumin1003> START - Cookbook sre.hosts.reimage for host db1159.eqiad.wmnet with OS trixie [production]
10:21 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P91421 and previous config saved to /var/cache/conftool/dbconfig/20260424-102154-fceratto.json [production]
10:21 <marostegui@cumin1003> END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2228: Reimage to Trixie [production]
10:21 <marostegui@cumin1003> END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1159: Reimage to Trixie [production]
10:21 <marostegui@cumin1003> START - Cookbook sre.mysql.depool depool db2228: Reimage to Trixie [production]
10:21 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2228.codfw.wmnet with reason: Reimage to Trixie [production]
10:21 <marostegui@cumin1003> START - Cookbook sre.mysql.depool depool db1159: Reimage to Trixie [production]
10:21 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1159.eqiad.wmnet with reason: Reimage to Trixie [production]
10:21 <klausman@cumin1003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve1015.eqiad.wmnet [production]
10:21 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1238', diff saved to https://phabricator.wikimedia.org/P91418 and previous config saved to /var/cache/conftool/dbconfig/20260424-102108-fceratto.json [production]
10:17 <klausman@cumin1003> START - Cookbook sre.hosts.reboot-single for host ml-serve1015.eqiad.wmnet [production]
10:15 <klausman@cumin1003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve1015.eqiad.wmnet [production]
10:12 <klausman@cumin1003> START - Cookbook sre.hosts.reboot-single for host ml-serve1015.eqiad.wmnet [production]
10:11 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1186 (T419635)', diff saved to https://phabricator.wikimedia.org/P91417 and previous config saved to /var/cache/conftool/dbconfig/20260424-101146-fceratto.json [production]
10:11 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1238', diff saved to https://phabricator.wikimedia.org/P91416 and previous config saved to /var/cache/conftool/dbconfig/20260424-101056-fceratto.json [production]
10:02 <aokoth@cumin1003> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Release - T424175 [production]
10:00 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1238 (T419961)', diff saved to https://phabricator.wikimedia.org/P91415 and previous config saved to /var/cache/conftool/dbconfig/20260424-100047-fceratto.json [production]
09:57 <cmooney@cumin1003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve1015.eqiad.wmnet [production]
09:57 <cmooney@cumin1003> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ml-serve1015.eqiad.wmnet on all recursors [production]
09:56 <cmooney@cumin1003> START - Cookbook sre.dns.wipe-cache ml-serve1015.eqiad.wmnet on all recursors [production]
09:56 <cmooney@cumin1003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
09:56 <cmooney@cumin1003> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: change records for ml-serve1014 - cmooney@cumin1003" [production]
09:56 <cmooney@cumin1003> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: change records for ml-serve1014 - cmooney@cumin1003" [production]
09:54 <fceratto@cumin1003> dbctl commit (dc=all): 'Depooling db1186 (T419635)', diff saved to https://phabricator.wikimedia.org/P91414 and previous config saved to /var/cache/conftool/dbconfig/20260424-095450-fceratto.json [production]
09:54 <fceratto@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1186.eqiad.wmnet with reason: Maintenance [production]
09:54 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1184 (T419635)', diff saved to https://phabricator.wikimedia.org/P91413 and previous config saved to /var/cache/conftool/dbconfig/20260424-095425-fceratto.json [production]
09:52 <fceratto@cumin1003> dbctl commit (dc=all): 'Depooling db1238 (T419961)', diff saved to https://phabricator.wikimedia.org/P91412 and previous config saved to /var/cache/conftool/dbconfig/20260424-095228-fceratto.json [production]
09:52 <cmooney@cumin1003> START - Cookbook sre.dns.netbox [production]
09:52 <cmooney@cumin1003> START - Cookbook sre.hosts.reboot-single for host ml-serve1015.eqiad.wmnet [production]
09:52 <fceratto@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1238.eqiad.wmnet with reason: Maintenance [production]
09:52 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1221 (T419961)', diff saved to https://phabricator.wikimedia.org/P91411 and previous config saved to /var/cache/conftool/dbconfig/20260424-095159-fceratto.json [production]
09:50 <klausman@cumin1003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve1015.eqiad.wmnet [production]
09:44 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P91410 and previous config saved to /var/cache/conftool/dbconfig/20260424-094417-fceratto.json [production]
09:41 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P91409 and previous config saved to /var/cache/conftool/dbconfig/20260424-094151-fceratto.json [production]
09:40 <klausman@cumin1003> START - Cookbook sre.hosts.reboot-single for host ml-serve1015.eqiad.wmnet [production]
09:34 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P91408 and previous config saved to /var/cache/conftool/dbconfig/20260424-093409-fceratto.json [production]
09:34 <cmooney@cumin1003> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host ml-serve1014.eqiad.wmnet [production]
09:31 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P91407 and previous config saved to /var/cache/conftool/dbconfig/20260424-093143-fceratto.json [production]
09:28 <cmooney@cumin1003> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ml-serve1014.eqiad.wmnet on all recursors [production]
09:28 <cmooney@cumin1003> START - Cookbook sre.dns.wipe-cache ml-serve1014.eqiad.wmnet on all recursors [production]
09:27 <cmooney@cumin1003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
09:27 <cmooney@cumin1003> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: change records for ml-serve1014 - cmooney@cumin1003" [production]
09:27 <cmooney@cumin1003> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: change records for ml-serve1014 - cmooney@cumin1003" [production]