5101-5150 of 8776 results (20ms)
2024-06-10 §
13:57 <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Banning hosts: elastic1107.eqiad.wmnet for T348977 - bking@cumin2002 [production]
13:57 <bking@cumin2002> END (FAIL) - Cookbook sre.elasticsearch.ban (exit_code=99) Banning hosts: elastic1107 for T348977 - bking@cumin2002 [production]
13:57 <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Banning hosts: elastic1107 for T348977 - bking@cumin2002 [production]
2024-06-05 §
22:50 <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic cluster restart - bking@cumin2002 - T366555 [production]
21:42 <bking@cumin2002> START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic cluster restart - bking@cumin2002 - T366555 [production]
21:41 <bking@cumin2002> END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic cluster restart - bking@cumin2002 - T366555 [production]
21:36 <bking@cumin2002> START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic cluster restart - bking@cumin2002 - T366555 [production]
13:48 <inflatador> bking@an-db1001 install python3-psycopg2 pkg T363001 [production]
13:45 <inflatador> bking@an-db1001 install acl pkg T363001 [production]
2024-05-28 §
19:45 <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Unbanning all hosts in search_eqiad [production]
19:45 <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Unbanning all hosts in search_eqiad [production]
19:01 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host elastic1056.eqiad.wmnet [production]
18:54 <bking@cumin2002> START - Cookbook sre.hosts.reboot-single for host elastic1056.eqiad.wmnet [production]
18:53 <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on elastic1056.eqiad.wmnet with reason: rebooting after abnormally high load [production]
18:53 <bking@cumin2002> START - Cookbook sre.hosts.downtime for 1:00:00 on elastic1056.eqiad.wmnet with reason: rebooting after abnormally high load [production]
17:25 <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: elastic1056.eqiad.wmnet for ban highly-loaded node - bking@cumin2002 [production]
17:25 <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Banning hosts: elastic1056.eqiad.wmnet for ban highly-loaded node - bking@cumin2002 [production]
17:25 <bking@cumin2002> END (FAIL) - Cookbook sre.elasticsearch.ban (exit_code=99) Banning hosts: elastic1056 for ban highly-loaded node - bking@cumin2002 [production]
17:25 <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Banning hosts: elastic1056 for ban highly-loaded node - bking@cumin2002 [production]
2024-05-20 §
21:38 <bking@cumin2002> END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) for Zookeeper A:zookeeper-flink-eqiad cluster: Roll restart of jvm daemons. [production]
21:32 <bking@cumin2002> START - Cookbook sre.zookeeper.roll-restart-zookeeper for Zookeeper A:zookeeper-flink-eqiad cluster: Roll restart of jvm daemons. [production]
21:29 <bking@cumin2002> END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) for Zookeeper A:zookeeper-flink-codfw cluster: Roll restart of jvm daemons. [production]
21:22 <bking@cumin2002> START - Cookbook sre.zookeeper.roll-restart-zookeeper for Zookeeper A:zookeeper-flink-codfw cluster: Roll restart of jvm daemons. [production]
2024-05-13 §
22:55 <bking@cumin2002> conftool action : set/weight=10:pooled=yes; selector: name=elastic110[5|7]\.eqiad\.wmnet [production]
22:27 <bking@cumin2002> conftool action : set/weight=10:pooled=no; selector: name=elastic110[5|7]\.eqiad\.wmnet [production]
2024-05-01 §
14:12 <bking@deploy1002> helmfile [eqiad] DONE helmfile.d/services/rdf-streaming-updater: apply [production]
14:11 <bking@deploy1002> helmfile [eqiad] START helmfile.d/services/rdf-streaming-updater: apply [production]
14:03 <bking@deploy1002> helmfile [codfw] DONE helmfile.d/services/rdf-streaming-updater: apply [production]
14:03 <bking@deploy1002> helmfile [codfw] START helmfile.d/services/rdf-streaming-updater: apply [production]
2024-04-30 §
18:59 <bking@deploy1002> helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply [production]
18:55 <bking@deploy1002> helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply [production]
2024-04-26 §
17:27 <bking@cumin2002> conftool action : set/weight=10:pooled=yes; selector: name=elastic110[3-7]\.eqiad\.wmnet [production]
16:57 <bking@cumin2002> conftool action : set/pooled=no; selector: name=elastic1107\.eqiad\.wmnet [production]
16:36 <bking@cumin2002> conftool action : set/weight=20:pooled=yes; selector: name=elastic1107\.eqiad\.wmnet [production]
16:35 <bking@cumin2002> conftool action : set/pooled=no; selector: name=elastic1106\.eqiad\.wmnet [production]
16:33 <bking@cumin2002> conftool action : set/weight=20:pooled=yes; selector: name=elastic1106\.eqiad\.wmnet [production]
16:32 <bking@cumin2002> conftool action : set/pooled=no; selector: name=elastic1105\.eqiad\.wmnet [production]
15:56 <bking@cumin2002> conftool action : set/pooled=no; selector: name=elastic1104\.eqiad\.wmnet [production]
15:56 <bking@cumin2002> conftool action : set/pooled=no; selector: name=elastic1103\.eqiad\.wmnet [production]
15:43 <bking@cumin2002> conftool action : set/weight=20:pooled=yes; selector: name=elastic1105\.eqiad\.wmnet [production]
15:31 <bking@cumin2002> conftool action : set/weight=20:pooled=yes; selector: name=elastic1104\.eqiad\.wmnet [production]
15:14 <bking@cumin2002> conftool action : set/weight=20:pooled=yes; selector: name=elastic1103\.eqiad\.wmnet [production]
2024-04-24 §
19:15 <bking@deploy1002> helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
19:15 <bking@deploy1002> helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
19:08 <inflatador> bking@deploy1002 stop `consumer-cloudelastic` release to test alerting T359213 [production]
19:07 <bking@deploy1002> helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
19:06 <bking@deploy1002> helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
2024-04-23 §
17:09 <inflatador> bking@mw1461 "restart rsyslog to reclaim fds T357616" [production]
2024-04-19 §
19:56 <bking@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T362508, journal in uncertain state) xfer wikidata from wdqs2022.codfw.wmnet -> wdqs2023.codfw.wmnet w/ force delete existing files, repooling both afterwards [production]
18:34 <bking@cumin2002> START - Cookbook sre.wdqs.data-transfer (T362508, journal in uncertain state) xfer wikidata from wdqs2022.codfw.wmnet -> wdqs2023.codfw.wmnet w/ force delete existing files, repooling both afterwards [production]