251-300 of 8775 results (18ms)
2026-03-11 §
14:11 <bking@deploy2002> helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply [production]
14:11 <bking@deploy2002> helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply [production]
13:36 <bking@deploy2002> helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply [production]
13:35 <bking@deploy2002> helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-ipoid-test: apply [production]
2026-03-10 §
06:14 <bking@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wdqs2009.codfw.wmnet with OS bookworm [production]
2026-03-09 §
22:34 <bking@cumin2002> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM dse-k8s-ctrl1001.eqiad.wmnet [production]
22:32 <bking@cumin2002> START - Cookbook sre.ganeti.reboot-vm for VM dse-k8s-ctrl1001.eqiad.wmnet [production]
22:30 <bking@cumin2002> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM dse-k8s-ctrl1002.eqiad.wmnet [production]
22:29 <bking@cumin2002> START - Cookbook sre.ganeti.reboot-vm for VM dse-k8s-ctrl1002.eqiad.wmnet [production]
22:28 <bking@cumin2002> END (FAIL) - Cookbook sre.ganeti.reboot-vm (exit_code=99) for VM dse-k8s-ctrl1002.eqiad.wmnet [production]
22:28 <bking@cumin2002> START - Cookbook sre.ganeti.reboot-vm for VM dse-k8s-ctrl1002.eqiad.wmnet [production]
22:28 <bking@cumin2002> END (FAIL) - Cookbook sre.ganeti.reboot-vm (exit_code=99) for VM dse-k8s-ctrl1002.eqiad.wmnet [production]
22:28 <bking@cumin2002> START - Cookbook sre.ganeti.reboot-vm for VM dse-k8s-ctrl1002.eqiad.wmnet [production]
18:06 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-ctrl1001.eqiad.wmnet [production]
18:01 <bking@cumin2002> START - Cookbook sre.hosts.reboot-single for host dse-k8s-ctrl1001.eqiad.wmnet [production]
15:12 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2009.codfw.wmnet with reason: host reimage [production]
15:08 <bking@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2009.codfw.wmnet with reason: host reimage [production]
14:50 <bking@cumin2002> START - Cookbook sre.hosts.reimage for host wdqs2009.codfw.wmnet with OS bookworm [production]
14:49 <bking@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host wdqs2009.codfw.wmnet with OS bullseye [production]
14:20 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2009.codfw.wmnet with reason: host reimage [production]
14:15 <bking@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2009.codfw.wmnet with reason: host reimage [production]
13:54 <bking@cumin2002> START - Cookbook sre.hosts.reimage for host wdqs2009.codfw.wmnet with OS bullseye [production]
2026-03-06 §
19:48 <bking@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply [production]
19:47 <bking@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply [production]
19:47 <bking@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply [production]
19:46 <bking@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply [production]
19:36 <bking@cumin2002> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=1) for host wdqs2009.codfw.wmnet [production]
19:23 <bking@cumin2002> START - Cookbook sre.hosts.reboot-single for host wdqs2009.codfw.wmnet [production]
19:17 <bking@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on wdqs2009.codfw.wmnet with reason: NFS might be hung, about to reboot [production]
2026-03-05 §
14:24 <bking@dns1004> START - running authdns-update [production]
2026-03-04 §
21:46 <bking@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on dse-k8s-worker1028.eqiad.wmnet with reason: broken networking [production]
21:35 <bking@cumin2002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet [production]
21:30 <bking@cumin2002> START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet [production]
2026-03-03 §
23:04 <bking@cumin2002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet [production]
22:56 <bking@cumin2002> START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet [production]
22:26 <bking@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-test: apply [production]
22:26 <bking@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-test: apply [production]
2026-03-02 §
21:48 <inflatador> bking@desktop restarting wdqs codfw to clear ProbeDown alerts [production]
21:14 <inflatador> bking@apt1002 reprepro --component thirdparty/opensearch3 update trixie-wikimedia T418388 [production]
2026-02-27 §
15:32 <bking@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
15:31 <bking@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
15:31 <bking@deploy2002> helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [production]
15:30 <bking@deploy2002> helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [production]
15:26 <bking@deploy2002> helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [production]
15:26 <bking@deploy2002> helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [production]
2026-02-25 §
21:11 <bking@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset: apply [production]
21:10 <bking@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset: apply [production]
20:49 <bking@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset: apply [production]
20:49 <bking@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset: apply [production]
20:47 <bking@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset: apply [production]