51-100 of 10000 results (26ms)
2026-04-24 ยง
09:01 <klausman@cumin1003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve1015.eqiad.wmnet [production]
08:56 <klausman@cumin1003> START - Cookbook sre.hosts.reboot-single for host ml-serve1015.eqiad.wmnet [production]
08:54 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P91399 and previous config saved to /var/cache/conftool/dbconfig/20260424-085421-fceratto.json [production]
08:52 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P91398 and previous config saved to /var/cache/conftool/dbconfig/20260424-085221-fceratto.json [production]
08:44 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P91397 and previous config saved to /var/cache/conftool/dbconfig/20260424-084414-fceratto.json [production]
08:42 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1199 (T419961)', diff saved to https://phabricator.wikimedia.org/P91396 and previous config saved to /var/cache/conftool/dbconfig/20260424-084213-fceratto.json [production]
08:36 <dcausse> remove the toolhub_tools index from the cirrussearch opensearch cluster in codfw (T423993) [toolhub]
08:34 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1169 (T419635)', diff saved to https://phabricator.wikimedia.org/P91395 and previous config saved to /var/cache/conftool/dbconfig/20260424-083406-fceratto.json [production]
08:31 <fceratto@cumin1003> dbctl commit (dc=all): 'Depooling db1199 (T419961)', diff saved to https://phabricator.wikimedia.org/P91394 and previous config saved to /var/cache/conftool/dbconfig/20260424-083118-fceratto.json [production]
08:31 <fceratto@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1199.eqiad.wmnet with reason: Maintenance [production]
08:30 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1190 (T419961)', diff saved to https://phabricator.wikimedia.org/P91393 and previous config saved to /var/cache/conftool/dbconfig/20260424-083050-fceratto.json [production]
08:29 <jmm@cumin2002> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host hcaptcha-proxy5003.wikimedia.org [production]
08:29 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) hcaptcha-proxy5003.wikimedia.org on all recursors [production]
08:29 <jmm@cumin2002> START - Cookbook sre.dns.wipe-cache hcaptcha-proxy5003.wikimedia.org on all recursors [production]
08:29 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
08:29 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM hcaptcha-proxy5003.wikimedia.org - jmm@cumin2002" [production]
08:29 <jmm@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM hcaptcha-proxy5003.wikimedia.org - jmm@cumin2002" [production]
08:27 <klausman@cumin1003> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve1014.eqiad.wmnet [production]
08:24 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
08:24 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) hcaptcha-proxy5003.wikimedia.org on all recursors [production]
08:24 <jmm@cumin2002> START - Cookbook sre.dns.wipe-cache hcaptcha-proxy5003.wikimedia.org on all recursors [production]
08:24 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
08:24 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM hcaptcha-proxy5003.wikimedia.org - jmm@cumin2002" [production]
08:24 <jmm@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM hcaptcha-proxy5003.wikimedia.org - jmm@cumin2002" [production]
08:22 <klausman@cumin1003> START - Cookbook sre.hosts.reboot-single for host ml-serve1014.eqiad.wmnet [production]
08:20 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P91392 and previous config saved to /var/cache/conftool/dbconfig/20260424-082041-fceratto.json [production]
08:19 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
08:19 <jmm@cumin2002> START - Cookbook sre.ganeti.makevm for new host hcaptcha-proxy5003.wikimedia.org [production]
08:15 <fceratto@cumin1003> dbctl commit (dc=all): 'Depooling db1169 (T419635)', diff saved to https://phabricator.wikimedia.org/P91391 and previous config saved to /var/cache/conftool/dbconfig/20260424-081539-fceratto.json [production]
08:15 <fceratto@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1169.eqiad.wmnet with reason: Maintenance [production]
08:12 <jmm@cumin2002> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host hcaptcha-proxy5003.wikimedia.org [production]
08:12 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) hcaptcha-proxy5003.wikimedia.org on all recursors [production]
08:12 <jmm@cumin2002> START - Cookbook sre.dns.wipe-cache hcaptcha-proxy5003.wikimedia.org on all recursors [production]
08:12 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
08:12 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM hcaptcha-proxy5003.wikimedia.org - jmm@cumin2002" [production]
08:12 <jmm@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Remove records for VM hcaptcha-proxy5003.wikimedia.org - jmm@cumin2002" [production]
08:10 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P91390 and previous config saved to /var/cache/conftool/dbconfig/20260424-081033-fceratto.json [production]
08:09 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
08:08 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
08:08 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
08:08 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) hcaptcha-proxy5003.wikimedia.org on all recursors [production]
08:08 <jmm@cumin2002> START - Cookbook sre.dns.wipe-cache hcaptcha-proxy5003.wikimedia.org on all recursors [production]
08:08 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
08:05 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
08:05 <ayounsi@cumin1003> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts netflow5002.eqsin.wmnet [production]
08:05 <ayounsi@cumin1003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
08:05 <ayounsi@cumin1003> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: netflow5002.eqsin.wmnet decommissioned, removing all IPs except the asset tag one - ayounsi@cumin1003" [production]
08:04 <jmm@cumin2002> END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) [production]
08:03 <ayounsi@cumin1003> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: netflow5002.eqsin.wmnet decommissioned, removing all IPs except the asset tag one - ayounsi@cumin1003" [production]
08:00 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1190 (T419961)', diff saved to https://phabricator.wikimedia.org/P91388 and previous config saved to /var/cache/conftool/dbconfig/20260424-080025-fceratto.json [production]