1-50 of 10000 results (24ms)
2026-08-21 ยง
19:37 <bking@cumin2003> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host krb1002.eqiad.wmnet with OS bookworm [production]
19:00 <lerickson@deploy1003> helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [production]
18:59 <lerickson@deploy1003> helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [production]
18:51 <lerickson@deploy1003> helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [production]
18:51 <lerickson@deploy1003> helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [production]
18:35 <lerickson@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [production]
18:35 <lerickson@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [production]
18:27 <lerickson@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [production]
18:27 <lerickson@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [production]
18:16 <bking@cumin2003> START - Cookbook sre.hosts.reimage for host krb1002.eqiad.wmnet with OS bookworm [production]
17:35 <sukhe@dns1004> END - running authdns-update [production]
17:33 <sukhe@dns1004> START - running authdns-update [production]
17:32 <sukhe@puppetserver1001> conftool action : set/pooled=yes; selector: name=dns5004.wikimedia.org [reason: resolved authdns-update issues] [production]
17:32 <sukhe@cumin1003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
17:32 <sukhe@cumin1003> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: force HEAD to be26e30ae101 - sukhe@cumin1003" [production]
17:32 <sukhe@cumin1003> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: force HEAD to be26e30ae101 - sukhe@cumin1003" [production]
17:28 <sukhe@cumin1003> START - Cookbook sre.dns.netbox [production]
17:28 <sukhe@puppetserver1001> conftool action : set/pooled=yes; selector: service=authdns-update,name=dns5004.wikimedia.org [reason: resolving authdns-update issues] [production]
17:28 <sukhe@cumin1003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
17:28 <sukhe@cumin1003> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: force HEAD to be26e30ae101 - sukhe@cumin1003" [production]
17:28 <sukhe@cumin1003> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: force HEAD to be26e30ae101 - sukhe@cumin1003" [production]
17:24 <sukhe@cumin1003> START - Cookbook sre.dns.netbox [production]
17:24 <sukhe@cumin1003> END (ERROR) - Cookbook sre.dns.netbox (exit_code=97) [production]
17:23 <sukhe@cumin1003> START - Cookbook sre.dns.netbox [production]
17:16 <lerickson@deploy1003> helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [production]
17:12 <lerickson@deploy1003> helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [production]
17:10 <sukhe@dns1004> END - running authdns-update [production]
17:08 <sukhe@dns1004> START - running authdns-update [production]
17:08 <sukhe@puppetserver1001> conftool action : set/pooled=no; selector: name=dns5004.wikimedia.org [reason: resolving authdns-update issues] [production]
17:07 <sukhe@dns1004> FAIL - running authdns-update [production]
17:05 <lerickson@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [production]
17:05 <sukhe@dns1004> START - running authdns-update [production]
17:01 <lerickson@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [production]
16:59 <lerickson@deploy1003> helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply [production]
16:56 <lerickson@deploy1003> helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply [production]
16:53 <cdobbins@cumin1003> conftool action : set/pooled=yes; selector: name=dns5004.* [reason: trixie upgrade] [production]
16:52 <cdobbins@cumin1003> END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for dns5004.wikimedia.org [production]
16:52 <cdobbins@cumin1003> START - Cookbook sre.hosts.remove-downtime for dns5004.wikimedia.org [production]
16:44 <cmooney@dns3003> END - running authdns-update [production]
16:41 <cmooney@dns3003> START - running authdns-update [production]
16:41 <cmooney@cumin1003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:41 <cmooney@cumin1003> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on eqsin<->codfw arelion - cmooney@cumin1003" [production]
16:37 <cmooney@cumin1003> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: remove dns entries for IPs formerly used on eqsin<->codfw arelion - cmooney@cumin1003" [production]
16:33 <cmooney@cumin1003> START - Cookbook sre.dns.netbox [production]
16:11 <lerickson@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [production]
16:11 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.bootstrap_and_add [admin]
16:08 <sukhe@dns1004> END - running authdns-update [production]
16:08 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) [admin]
16:08 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.bootstrap_and_add [admin]
16:08 <lerickson@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [production]