3401-3450 of 10000 results (29ms)
2023-04-05 §
08:02 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on kafka-main1002.eqiad.wmnet with reason: restart kafka, switch to PKI [production]
07:56 <elukey@cumin1001> START - Cookbook sre.ganeti.reimage for host kafka-test1006.eqiad.wmnet with OS bullseye [production]
07:54 <elukey@cumin1001> END (FAIL) - Cookbook sre.ganeti.reimage (exit_code=99) for host kafka-test1006.eqiad.wmnet with OS bullseye [production]
07:24 <elukey@cumin1001> START - Cookbook sre.ganeti.reimage for host kafka-test1006.eqiad.wmnet with OS bullseye [production]
2023-04-03 §
09:19 <elukey> move kafka-jumbo1006's kafka broker cert to PKI - T296064 [production]
09:19 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on kafka-jumbo1006.eqiad.wmnet with reason: restart kafka, switch to PKI [production]
09:19 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on kafka-jumbo1006.eqiad.wmnet with reason: restart kafka, switch to PKI [production]
08:54 <elukey> move kafka-jumbo1009's kafka broker cert to PKI - T296064 [production]
08:53 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on kafka-jumbo1009.eqiad.wmnet with reason: restart kafka, switch to PKI [production]
08:53 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on kafka-jumbo1009.eqiad.wmnet with reason: restart kafka, switch to PKI [production]
08:29 <elukey> move kafka-main1001's kafka broker to PKI - T319372 [production]
08:24 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on kafka-main1001.eqiad.wmnet with reason: restart kafka, switch to PKI [production]
08:23 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on kafka-main1001.eqiad.wmnet with reason: restart kafka, switch to PKI [production]
08:03 <elukey> move kafka-jumbo1008's kafka broker cert to PKI - T296064 [production]
08:03 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on kafka-jumbo1008.eqiad.wmnet with reason: restart kafka, switch to PKI [production]
08:02 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on kafka-jumbo1008.eqiad.wmnet with reason: restart kafka, switch to PKI [production]
07:43 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on kafka-jumbo1007.eqiad.wmnet with reason: restart kafka, switch to PKI [production]
07:43 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on kafka-jumbo1007.eqiad.wmnet with reason: restart kafka, switch to PKI [production]
07:43 <elukey> move kafka-jumbo1006's kafka broker cert to PKI - T296064 [production]
06:53 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on kafka-jumbo1005.eqiad.wmnet with reason: restart kafka, switch to PKI [production]
06:52 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on kafka-jumbo1005.eqiad.wmnet with reason: restart kafka, switch to PKI [production]
06:52 <elukey> move kafka-jumbo1005's kafka broker cert to PKI - T296064 [production]
2023-03-31 §
13:12 <elukey> move kafka-jumbo1004's kafka broker cert to PKI - T296064 [production]
13:11 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on kafka-jumbo1004.eqiad.wmnet with reason: restart kafka, switch to PKI [production]
13:11 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on kafka-jumbo1004.eqiad.wmnet with reason: restart kafka, switch to PKI [production]
13:09 <elukey> restart kafkatee on centrallog2002 - test to see if there are issues connecting to the jumbo brokers running pki [production]
09:54 <elukey> move kafka-jumbo1003's kafka broker cert to PKI - T296064 [production]
09:54 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on kafka-jumbo1003.eqiad.wmnet with reason: restart kafka, switch to PKI [production]
09:53 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on kafka-jumbo1003.eqiad.wmnet with reason: restart kafka, switch to PKI [production]
09:03 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:15:00 on kafka-jumbo1002.eqiad.wmnet with reason: restart kafka, switch to PKI [production]
09:03 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 0:15:00 on kafka-jumbo1002.eqiad.wmnet with reason: restart kafka, switch to PKI [production]
09:02 <elukey> move kafka-jumbo1002's kafka broker cert to PKI - T296064 [production]
2023-03-30 §
14:17 <elukey@cumin1001> END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-main-codfw cluster: Roll restart of jvm daemons. [production]
12:36 <elukey@cumin1001> START - Cookbook sre.kafka.roll-restart-brokers for Kafka A:kafka-main-codfw cluster: Roll restart of jvm daemons. [production]
10:35 <elukey@cumin1001> END (PASS) - Cookbook sre.kafka.roll-restart-brokers (exit_code=0) for Kafka A:kafka-main-eqiad cluster: Roll restart of jvm daemons. [production]
08:55 <elukey> move kafka main clusters to new truststore (PKI+Puppet root CA certs) - T319372 [production]
08:54 <elukey@cumin1001> START - Cookbook sre.kafka.roll-restart-brokers for Kafka A:kafka-main-eqiad cluster: Roll restart of jvm daemons. [production]
2023-03-29 §
15:29 <elukey@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'sync'. [production]
15:29 <elukey@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'sync'. [production]
15:28 <elukey@deploy2002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. [production]
15:27 <elukey@deploy2002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. [production]
15:27 <elukey@deploy2002> helmfile [ml-serve-codfw] DONE helmfile.d/admin 'sync'. [production]
15:27 <elukey@deploy2002> helmfile [ml-serve-codfw] START helmfile.d/admin 'sync'. [production]
15:06 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on kafka-main2001.codfw.wmnet with reason: Stop kafka, dist-upgrade [production]
15:06 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on kafka-main2001.codfw.wmnet with reason: Stop kafka, dist-upgrade [production]
13:42 <elukey> run dist-upgrade on kafka-main2002 to upgrade it to bullseye - T332013 [production]
13:41 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on kafka-main2002.codfw.wmnet with reason: stop kafka, dist-upgrade [production]
13:41 <elukey@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on kafka-main2002.codfw.wmnet with reason: stop kafka, dist-upgrade [production]
09:02 <elukey> move kafka on kafka-jumbo1001 to PKI TLS certs - T296064 [production]
09:02 <elukey@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on kafka-jumbo1001.eqiad.wmnet with reason: restart kafka, upgrade to PKI [production]