3301-3350 of 10000 results (31ms)
2023-04-06 §
10:26 <elukey@deploy2002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [production]
09:38 <elukey@cumin1001> END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-main-codfw cluster: Roll restart of jvm daemons. [production]
09:30 <elukey> kafka main codfw cluster migrated to PKI TLS certs for brokers - T319372 [production]
08:40 <elukey> powercycle ml-serve2004 - host frozen, racadm getsel shows multi-bit errors in various DIMM slots [production]
07:56 <elukey@cumin1001> START - Cookbook sre.kafka.roll-restart-brokers for Kafka A:kafka-main-codfw cluster: Roll restart of jvm daemons. [production]
2023-04-05 §
16:30 <elukey@deploy2002> helmfile [eqiad] DONE helmfile.d/services/changeprop: sync [production]
16:30 <elukey@deploy2002> helmfile [eqiad] START helmfile.d/services/changeprop: sync [production]
16:02 <elukey@cumin1001> END (PASS) - Cookbook sre.ganeti.reimage (exit_code=0) for host kafka-test1010.eqiad.wmnet with OS bullseye [production]
15:44 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-test1010.eqiad.wmnet with reason: host reimage [production]
15:41 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-test1010.eqiad.wmnet with reason: host reimage [production]
15:27 <elukey@cumin1001> START - Cookbook sre.ganeti.reimage for host kafka-test1010.eqiad.wmnet with OS bullseye [production]
15:10 <elukey@cumin1001> END (PASS) - Cookbook sre.ganeti.reimage (exit_code=0) for host kafka-test1009.eqiad.wmnet with OS bullseye [production]
15:09 <elukey@deploy2002> helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync [production]
15:08 <elukey@deploy2002> helmfile [eqiad] START helmfile.d/services/eventgate-main: sync [production]
14:54 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-test1009.eqiad.wmnet with reason: host reimage [production]
14:51 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-test1009.eqiad.wmnet with reason: host reimage [production]
14:36 <elukey@cumin1001> START - Cookbook sre.ganeti.reimage for host kafka-test1009.eqiad.wmnet with OS bullseye [production]
14:33 <elukey> restart kafka on kafka-main1005 to pick up the new TLS certificate (PKI based) - T319372 [production]
14:31 <elukey@cumin1001> END (PASS) - Cookbook sre.ganeti.reimage (exit_code=0) for host kafka-test1008.eqiad.wmnet with OS bullseye [production]
14:31 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on kafka-main1005.eqiad.wmnet with reason: restart kafka, switch to PKI [production]
14:30 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on kafka-main1005.eqiad.wmnet with reason: restart kafka, switch to PKI [production]
14:14 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-test1008.eqiad.wmnet with reason: host reimage [production]
14:11 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-test1008.eqiad.wmnet with reason: host reimage [production]
14:00 <elukey> powercycle an-worker1132 [production]
13:58 <elukey@cumin1001> START - Cookbook sre.ganeti.reimage for host kafka-test1008.eqiad.wmnet with OS bullseye [production]
13:57 <elukey@cumin1001> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM kafka-test1010.eqiad.wmnet [production]
13:52 <elukey@cumin1001> START - Cookbook sre.ganeti.reboot-vm for VM kafka-test1010.eqiad.wmnet [production]
13:52 <elukey@cumin1001> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM kafka-test1009.eqiad.wmnet [production]
13:52 <elukey> restart kafka on kafka-main1004 to pick up the new TLS certificate (PKI based) - T319372 [production]
13:49 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on kafka-main1004.eqiad.wmnet with reason: restart kafka, switch to PKI [production]
13:48 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on kafka-main1004.eqiad.wmnet with reason: restart kafka, switch to PKI [production]
13:48 <elukey@cumin1001> START - Cookbook sre.ganeti.reboot-vm for VM kafka-test1009.eqiad.wmnet [production]
13:03 <elukey@cumin1001> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM kafka-test1008.eqiad.wmnet [production]
12:58 <elukey@cumin1001> START - Cookbook sre.ganeti.reboot-vm for VM kafka-test1008.eqiad.wmnet [production]
10:14 <elukey> restart purged on cp5032, cp1082, cp6004, cp1090 - errors after restart of kafka main eqiad brokers [production]
10:12 <elukey> restart purged on cp6015 to verify if connection to brokers failed are only temporary or not [production]
10:11 <elukey@cumin1001> END (PASS) - Cookbook sre.ganeti.reimage (exit_code=0) for host kafka-test1007.eqiad.wmnet with OS bullseye [production]
09:54 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-test1007.eqiad.wmnet with reason: host reimage [production]
09:51 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-test1007.eqiad.wmnet with reason: host reimage [production]
09:36 <elukey@cumin1001> START - Cookbook sre.ganeti.reimage for host kafka-test1007.eqiad.wmnet with OS bullseye [production]
09:35 <elukey> restart kafka on kafka-main1003 to pick up the new TLS certificate (PKI based) - T319372 [production]
09:34 <elukey@cumin1001> END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM kafka-test1007.eqiad.wmnet [production]
09:34 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on kafka-main1003.eqiad.wmnet with reason: restart kafka, switch to PKI [production]
09:34 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on kafka-main1003.eqiad.wmnet with reason: restart kafka, switch to PKI [production]
09:30 <elukey@cumin1001> START - Cookbook sre.ganeti.reboot-vm for VM kafka-test1007.eqiad.wmnet [production]
08:27 <elukey@cumin1001> END (PASS) - Cookbook sre.ganeti.reimage (exit_code=0) for host kafka-test1006.eqiad.wmnet with OS bullseye [production]
08:09 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kafka-test1006.eqiad.wmnet with reason: host reimage [production]
08:07 <elukey> restart kafka on kafka-main1002 to pick up the new TLS certificate (PKI based) - T319372 [production]
08:06 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on kafka-test1006.eqiad.wmnet with reason: host reimage [production]
08:02 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on kafka-main1002.eqiad.wmnet with reason: restart kafka, switch to PKI [production]