1651-1700 of 10000 results (16ms)
2026-03-23 §
14:36 <bking@cumin2002> START - Cookbook sre.dns.wipe-cache k8s-ingress-dse-aa.discovery.wmnet on all recursors [production]
14:31 <bking@cumin2002> conftool action : set/pooled=true; selector: dnsdisc=k8s-ingress-dse-aa,name=codfw [production]
2026-03-20 §
13:39 <inflatador> bking@deploy2002 restarting opensearch-ipoid cluster to apply new certificates [production]
2026-03-19 §
14:32 <bking@cumin2002> conftool action : set/pooled=yes:weight=1; selector: name=dse-k8s-worker1010.eqiad.wmnet|dse-k8s-worker1011.eqiad.wmnet|dse-k8s-worker1012.eqiad.wmnet|dse-k8s-worker1013.eqiad.wmnet|dse-k8s-worker1015.eqiad.wmnet|dse-k8s-worker1016.eqiad.wmnet|dse-k8s-worker1017.eqiad.wmnet|dse-k8s-worker1018.eqiad.wmnet|dse-k8s-worker1019.eqiad.wmnet [production]
14:25 <bking@cumin2002> conftool action : set/pooled=yes:weight=10; selector: name=dse-k8s-worker1012.eqiad.wmnet|dse-k8s-worker1015.eqiad.wmnet|dse-k8s-worker1016.eqiad.wmnet|dse-k8s-worker1017.eqiad.wmnet [production]
2026-03-17 §
14:38 <inflatador> bking@requestctl remove `wdqs_highest_error_rate_ever_seen` requestctl rule as it is no longer needed [production]
2026-03-12 §
15:14 <bking@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
15:13 <bking@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
13:59 <bking@deploy2002> helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [production]
13:59 <bking@deploy2002> helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [production]
13:57 <bking@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
13:54 <bking@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
2026-03-11 §
18:49 <bking@deploy2002> helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply [production]
18:49 <bking@deploy2002> helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply [production]
18:45 <bking@deploy2002> helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply [production]
18:45 <bking@deploy2002> helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply [production]
18:44 <bking@deploy2002> helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply [production]
18:44 <bking@deploy2002> helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply [production]
14:23 <bking@deploy2002> helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply [production]
14:23 <bking@deploy2002> helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply [production]
14:12 <bking@deploy2002> helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply [production]
14:12 <bking@deploy2002> helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply [production]
14:11 <bking@deploy2002> helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply [production]
14:11 <bking@deploy2002> helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply [production]
13:36 <bking@deploy2002> helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply [production]
13:35 <bking@deploy2002> helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply [production]
2026-03-10 §
06:14 <bking@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wdqs2009.codfw.wmnet with OS bookworm [production]
2026-03-09 §
22:34 <bking@cumin2002> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM dse-k8s-ctrl1001.eqiad.wmnet [production]
22:32 <bking@cumin2002> START - Cookbook sre.ganeti.reboot-vm for VM dse-k8s-ctrl1001.eqiad.wmnet [production]
22:30 <bking@cumin2002> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM dse-k8s-ctrl1002.eqiad.wmnet [production]
22:29 <bking@cumin2002> START - Cookbook sre.ganeti.reboot-vm for VM dse-k8s-ctrl1002.eqiad.wmnet [production]
22:28 <bking@cumin2002> END (FAIL) - Cookbook sre.ganeti.reboot-vm (exit_code=99) for VM dse-k8s-ctrl1002.eqiad.wmnet [production]
22:28 <bking@cumin2002> START - Cookbook sre.ganeti.reboot-vm for VM dse-k8s-ctrl1002.eqiad.wmnet [production]
22:28 <bking@cumin2002> END (FAIL) - Cookbook sre.ganeti.reboot-vm (exit_code=99) for VM dse-k8s-ctrl1002.eqiad.wmnet [production]
22:28 <bking@cumin2002> START - Cookbook sre.ganeti.reboot-vm for VM dse-k8s-ctrl1002.eqiad.wmnet [production]
18:06 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1001.eqiad.wmnet [production]
18:01 <bking@cumin2002> START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1001.eqiad.wmnet [production]
15:12 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2009.codfw.wmnet with reason: host reimage [production]
15:08 <bking@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2009.codfw.wmnet with reason: host reimage [production]
14:50 <bking@cumin2002> START - Cookbook sre.hosts.reimage for host wdqs2009.codfw.wmnet with OS bookworm [production]
14:49 <bking@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wdqs2009.codfw.wmnet with OS bullseye [production]
14:20 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2009.codfw.wmnet with reason: host reimage [production]
14:15 <bking@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2009.codfw.wmnet with reason: host reimage [production]
13:54 <bking@cumin2002> START - Cookbook sre.hosts.reimage for host wdqs2009.codfw.wmnet with OS bullseye [production]
2026-03-06 §
19:48 <bking@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply [production]
19:47 <bking@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply [production]
19:47 <bking@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply [production]
19:46 <bking@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply [production]
19:36 <bking@cumin2002> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host wdqs2009.codfw.wmnet [production]
19:23 <bking@cumin2002> START - Cookbook sre.hosts.reboot-single for host wdqs2009.codfw.wmnet [production]