4751-4800 of 8776 results (28ms)
2024-08-20 §
17:37 <bking@cumin2002> START - Cookbook sre.hosts.reimage for host wdqs2024.codfw.wmnet with OS bullseye [production]
17:32 <bking@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "recover wdqs2024 from failed status T372919 - bking@cumin2002" [production]
17:20 <bking@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "recover wdqs2024 from failed status T372919 - bking@cumin2002" [production]
2024-08-19 §
14:27 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on wdqs[1022,1024].eqiad.wmnet with reason: noisy alerts, will look at later in the day [production]
14:27 <bking@cumin2002> START - Cookbook sre.hosts.downtime for 5:00:00 on wdqs[1022,1024].eqiad.wmnet with reason: noisy alerts, will look at later in the day [production]
2024-08-13 §
21:56 <inflatador> bking@cumin2002 reboot wdqs101[3-5],1018,1020 from DRAC due to unresponsiveness T372442 [production]
19:32 <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (2 nodes at a time) for ElasticSearch cluster search_eqiad: security update - bking@cumin2002 - T371874 [production]
17:40 <bking@cumin2002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (2 nodes at a time) for ElasticSearch cluster search_eqiad: security update - bking@cumin2002 - T371874 [production]
17:39 <bking@cumin2002> END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: security update - bking@cumin2002 - T371874 [production]
17:26 <bking@cumin2002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_eqiad: security update - bking@cumin2002 - T371874 [production]
2024-08-12 §
16:33 <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: security update - bking@cumin2002 - T371874 [production]
14:44 <bking@cumin2002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: security update - bking@cumin2002 - T371874 [production]
2024-08-09 §
18:10 <inflatador> bking@wdqs-codfw-public mitigate codfw wdqs abuse via nginx hotfix T372074 [production]
2024-08-08 §
21:29 <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: security update - bking@cumin2002 - T371874 [production]
21:05 <bking@cumin2002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: security update - bking@cumin2002 - T371874 [production]
19:23 <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster relforge: security update - bking@cumin2002 - T371874 [production]
19:13 <bking@cumin2002> START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster relforge: security update - bking@cumin2002 - T371874 [production]
17:08 <inflatador> bking@wdqs1020 restart wdqs-blazegraph service due to excessive GC [production]
2024-07-31 §
13:46 <bking@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
13:45 <bking@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply [production]
2024-07-24 §
19:31 <bking@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow: apply [production]
19:31 <bking@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow: apply [production]
19:30 <bking@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow: apply [production]
19:30 <bking@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow: apply [production]
19:27 <bking@deploy1002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow: apply [production]
19:27 <bking@deploy1002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow: apply [production]
2024-07-23 §
22:22 <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Unbanning all hosts in search_eqiad [production]
22:22 <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Unbanning all hosts in search_eqiad [production]
2024-07-22 §
22:37 <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: elastic110[0-2]* for T348977 - bking@cumin2002 [production]
22:36 <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Banning hosts: elastic110[0-2]* for T348977 - bking@cumin2002 [production]
22:35 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on elastic[1100-1102].eqiad.wmnet with reason: T348977 [production]
22:34 <bking@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on elastic[1100-1102].eqiad.wmnet with reason: T348977 [production]
2024-07-18 §
22:19 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on elastic1100.eqiad.wmnet with reason: catch up on indexing [production]
22:19 <bking@cumin2002> START - Cookbook sre.hosts.downtime for 4:00:00 on elastic1100.eqiad.wmnet with reason: catch up on indexing [production]
21:57 <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.force-shard-allocation (exit_code=0) [production]
21:57 <bking@cumin2002> START - Cookbook sre.elasticsearch.force-shard-allocation [production]
21:08 <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Unbanning all hosts in search_eqiad [production]
21:08 <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Unbanning all hosts in search_eqiad [production]
2024-07-17 §
17:13 <inflatador> bking@kafka-main2005 `kafka topics --create --topic ${TOPIC} --partitions 1 --replication-factor 3; kafka configs --entity-type topics --entity-name ${TOPIC} --alter --add-config retention.ms=2592000000 T367510` [production]
16:08 <inflatador> bking@kafka-main1005 `kafka topics --create --topic ${TOPIC} --partitions 1 --replication-factor 3; kafka configs --entity-type topics --entity-name ${TOPIC} --alter --add-config retention.ms=2592000000` T367510 [production]
00:42 <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Unbanning all hosts in search_eqiad [production]
00:42 <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Unbanning all hosts in search_eqiad [production]
2024-07-15 §
19:24 <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: elastic109[8-9]* for T348977 - bking@cumin2002 [production]
19:24 <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Banning hosts: elastic109[8-9]* for T348977 - bking@cumin2002 [production]
19:24 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on elastic[1098-1099].eqiad.wmnet with reason: T348977 [production]
19:23 <bking@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on elastic[1098-1099].eqiad.wmnet with reason: T348977 [production]
2024-07-11 §
21:11 <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Unbanning all hosts in search_eqiad [production]
21:11 <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Unbanning all hosts in search_eqiad [production]
13:04 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6 days, 0:00:00 on relforge[1003-1004].eqiad.wmnet with reason: T368950 [production]
13:03 <bking@cumin2002> START - Cookbook sre.hosts.downtime for 6 days, 0:00:00 on relforge[1003-1004].eqiad.wmnet with reason: T368950 [production]