1701-1750 of 10000 results (15ms)
2026-03-06 §
19:17 <bking@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on wdqs2009.codfw.wmnet with reason: NFS might be hung, about to reboot [production]
2026-03-05 §
14:24 <bking@dns1004> START - running authdns-update [production]
2026-03-04 §
21:46 <bking@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 0:00:00 on dse-k8s-worker1028.eqiad.wmnet with reason: broken networking [production]
21:35 <bking@cumin2002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet [production]
21:30 <bking@cumin2002> START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet [production]
2026-03-03 §
23:04 <bking@cumin2002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1028.eqiad.wmnet [production]
22:56 <bking@cumin2002> START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1028.eqiad.wmnet [production]
22:26 <bking@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-test: apply [production]
22:26 <bking@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-test: apply [production]
2026-03-02 §
21:48 <inflatador> bking@desktop restarting wdqs codfw to clear ProbeDown alerts [production]
21:14 <inflatador> bking@apt1002 reprepro --component thirdparty/opensearch3 update trixie-wikimedia T418388 [production]
2026-02-27 §
15:32 <bking@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
15:31 <bking@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
15:31 <bking@deploy2002> helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [production]
15:30 <bking@deploy2002> helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [production]
15:26 <bking@deploy2002> helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'. [production]
15:26 <bking@deploy2002> helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'. [production]
2026-02-25 §
21:11 <bking@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset: apply [production]
21:10 <bking@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset: apply [production]
20:49 <bking@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset: apply [production]
20:49 <bking@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset: apply [production]
20:47 <bking@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset: apply [production]
20:47 <bking@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset: apply [production]
2026-02-24 §
15:59 <inflatador> bking@local restarting wdqs codfw main to deal with 5xx errors [production]
2026-02-21 §
15:33 <bking@cumin2002> END (FAIL) - Cookbook sre.wdqs.restart (exit_code=99) [production]
15:28 <bking@cumin2002> START - Cookbook sre.wdqs.restart [production]
2026-02-19 §
17:30 <inflatador> bking@wmf restart bg on wdqs2022.codfw.wmnet,wdqs2014.codfw.wmnet,wdqs2007.codfw.wmnet to clear ProbeDown alerts [production]
2026-02-18 §
16:10 <bking@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on wdqs1028.eqiad.wmnet with reason: broken puppet [production]
12:52 <bking@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wdqs1028.eqiad.wmnet with OS bookworm [production]
2026-02-17 §
21:49 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs1028.eqiad.wmnet with reason: host reimage [production]
21:43 <bking@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs1028.eqiad.wmnet with reason: host reimage [production]
21:23 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.move-vlan (exit_code=0) for host wdqs1028 [production]
21:23 <bking@cumin2002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs1028 [production]
21:21 <bking@cumin2002> START - Cookbook sre.network.configure-switch-interfaces for host wdqs1028 [production]
21:21 <bking@cumin2002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) wdqs1028.eqiad.wmnet 6.48.64.10.in-addr.arpa 6.0.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [production]
21:21 <bking@cumin2002> START - Cookbook sre.dns.wipe-cache wdqs1028.eqiad.wmnet 6.48.64.10.in-addr.arpa 6.0.0.0.8.4.0.0.4.6.0.0.0.1.0.0.7.0.1.0.1.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors [production]
21:21 <bking@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
21:21 <bking@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1028 - bking@cumin2002" [production]
21:21 <bking@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host wdqs1028 - bking@cumin2002" [production]
21:16 <bking@cumin2002> START - Cookbook sre.dns.netbox [production]
21:16 <bking@cumin2002> START - Cookbook sre.hosts.move-vlan for host wdqs1028 [production]
21:15 <bking@cumin2002> START - Cookbook sre.hosts.reimage for host wdqs1028.eqiad.wmnet with OS bookworm [production]
2026-02-13 §
16:00 <bking@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply [production]
16:00 <bking@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply [production]
15:42 <bking@deploy2002> helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply [production]
15:42 <bking@deploy2002> helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply [production]
15:27 <bking@deploy2002> helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply [production]
15:25 <bking@deploy2002> helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search-test: apply [production]
2026-02-12 §
14:52 <bking@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 21 days, 0:00:00 on 9 hosts with reason: shut off 1Gbps hosts [production]
2026-02-06 §
14:42 <bking@deploy2002> helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-test: apply [production]