1-50 of 773 results (8ms)
2025-06-09 §
13:31 <taavi@deploy1003> Finished scap sync-world: Backport for [[gerrit:1153363|logging: Allow sampling of Logstash logs (T395967)]], [[gerrit:1153364|logging: Sample some high-volume log streams (T394402)]] (duration: 24m 30s) [production]
13:21 <taavi@deploy1003> taavi, tgr: Backport for [[gerrit:1153363|logging: Allow sampling of Logstash logs (T395967)]], [[gerrit:1153364|logging: Sample some high-volume log streams (T394402)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
13:06 <taavi@deploy1003> Started scap sync-world: Backport for [[gerrit:1153363|logging: Allow sampling of Logstash logs (T395967)]], [[gerrit:1153364|logging: Sample some high-volume log streams (T394402)]] [production]
2025-05-20 §
15:21 <jmm@cumin2002> END (PASS) - Cookbook sre.o11y.roll-restart-reboot-logstash-collectors (exit_code=0) rolling restart_daemons on A:logstash-collector [production]
15:13 <jmm@cumin2002> START - Cookbook sre.o11y.roll-restart-reboot-logstash-collectors rolling restart_daemons on A:logstash-collector [production]
2025-03-24 §
16:57 <btullis@cumin1002> END (PASS) - Cookbook sre.apifeatureusage.roll-restart-reboot-logstash (exit_code=0) rolling restart_daemons on A:apifeatureusage [production]
16:55 <btullis@cumin1002> START - Cookbook sre.apifeatureusage.roll-restart-reboot-logstash rolling restart_daemons on A:apifeatureusage [production]
2025-03-13 §
14:35 <jmm@cumin2002> END (PASS) - Cookbook sre.o11y.roll-restart-reboot-logstash-collectors (exit_code=0) rolling restart_daemons on A:logstash-collector [production]
14:27 <jmm@cumin2002> START - Cookbook sre.o11y.roll-restart-reboot-logstash-collectors rolling restart_daemons on A:logstash-collector [production]
2024-12-11 §
14:11 <btullis@cumin1002> END (PASS) - Cookbook sre.apifeatureusage.roll-restart-reboot-logstash (exit_code=0) rolling restart_daemons on A:apifeatureusage [production]
14:08 <btullis@cumin1002> START - Cookbook sre.apifeatureusage.roll-restart-reboot-logstash rolling restart_daemons on A:apifeatureusage [production]
2024-09-25 §
07:19 <jmm@cumin2002> END (PASS) - Cookbook sre.o11y.roll-restart-reboot-logstash-collectors (exit_code=0) rolling restart_daemons on A:logstash-collector [production]
07:02 <jmm@cumin2002> START - Cookbook sre.o11y.roll-restart-reboot-logstash-collectors rolling restart_daemons on A:logstash-collector [production]
2024-09-24 §
16:38 <swfrench@cumin1002> conftool action : set/pooled=false; selector: dnsdisc=logstash,name=codfw [production]
16:37 <swfrench@cumin1002> conftool action : set/pooled=true; selector: dnsdisc=logstash,name=eqiad [production]
2024-09-16 §
16:12 <rzl> rzl@cumin1002:~$ sudo cumin logstash[2023,2025,2030-2032].codfw.wmnet,logstash[1025,1030,1032].eqiad.wmnet 'systemctl restart opensearch-dashboards' # only hosts where status is failed [production]
2024-09-09 §
17:44 <jforrester@deploy1003> Finished scap sync-world: Backport for [[gerrit:1071266|ZObjectStore::findZTesterResult: Trim our own error so we don't break logstash (T374241)]] (duration: 12m 05s) [production]
17:33 <jforrester@deploy1003> jforrester: Backport for [[gerrit:1071266|ZObjectStore::findZTesterResult: Trim our own error so we don't break logstash (T374241)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
17:31 <jforrester@deploy1003> Started scap sync-world: Backport for [[gerrit:1071266|ZObjectStore::findZTesterResult: Trim our own error so we don't break logstash (T374241)]] [production]
2024-09-04 §
10:11 <btullis@cumin1002> END (PASS) - Cookbook sre.apifeatureusage.roll-restart-reboot-logstash (exit_code=0) rolling restart_daemons on A:apifeatureusage [production]
10:08 <btullis@cumin1002> START - Cookbook sre.apifeatureusage.roll-restart-reboot-logstash rolling restart_daemons on A:apifeatureusage [production]
2024-08-27 §
09:04 <tappof@cumin2002> END (PASS) - Cookbook sre.o11y.roll-restart-reboot-logstash-collectors (exit_code=0) rolling restart_daemons on P{O:logging::opensearch::collector and logstash*.codfw.wmnet} and (A:logstash-collector) [production]
09:00 <tappof@cumin2002> START - Cookbook sre.o11y.roll-restart-reboot-logstash-collectors rolling restart_daemons on P{O:logging::opensearch::collector and logstash*.codfw.wmnet} and (A:logstash-collector) [production]
2024-08-16 §
23:27 <eevans@deploy1003> Started deploy [cassandra/logstash-logback-encoder@42653e6] (beta): Test deploy [production]
20:26 <eevans@deploy1003> Finished deploy [cassandra/logstash-logback-encoder@42653e6] (aqs): Test (duration: 00m 32s) [production]
20:26 <eevans@deploy1003> Started deploy [cassandra/logstash-logback-encoder@42653e6] (aqs): Test [production]
20:15 <eevans@deploy1003> Finished deploy [cassandra/logstash-logback-encoder@42653e6] (beta): Beta deploy (duration: 00m 31s) [production]
20:14 <eevans@deploy1003> Started deploy [cassandra/logstash-logback-encoder@42653e6] (beta): Beta deploy [production]
2024-07-29 §
12:46 <godog> upgrade and roll-restart benthos@mw_accesslog_sampler on logstash hosts [production]
10:12 <godog> bounce benthos@mw_accesslog_sampler on logstash collectors [production]
2024-07-24 §
21:44 <ryankemper@cumin2002> END (PASS) - Cookbook sre.apifeatureusage.roll-restart-reboot-logstash (exit_code=0) rolling reboot on A:apifeatureusage [production]
21:35 <ryankemper@cumin2002> START - Cookbook sre.apifeatureusage.roll-restart-reboot-logstash rolling reboot on A:apifeatureusage [production]
2024-06-21 §
23:54 <cwhite> delete remaining 2024.03 log indexes to make room on logstash eqiad and codfw T368180 [production]
2024-06-12 §
15:28 <denisse@cumin2002> conftool action : set/pooled=true; selector: dnsdisc=logstash,name=eqiad [production]
2024-05-13 §
12:44 <brouberol@cumin2002> END (PASS) - Cookbook sre.apifeatureusage.roll-restart-reboot-logstash (exit_code=0) rolling restart_daemons on A:apifeatureusage [production]
2024-05-09 §
15:27 <eevans@deploy1002> Finished deploy [cassandra/logstash-logback-encoder@42653e6] (aqs): (no justification provided) (duration: 00m 33s) [production]
15:27 <eevans@deploy1002> Started deploy [cassandra/logstash-logback-encoder@42653e6] (aqs): (no justification provided) [production]
2024-05-07 §
20:09 <denisse> Restarting envoyproxy and opensearch-dashboards services on the Logstash hosts that serve OpenSearch dashboards to migrate to CFSSL certificates - T360414 [production]
20:06 <denisse> Enabling Puppet on the Logstash hosts that serve OpenSearch dashboards to migrate to CFSSL certificates - T360414 [production]
19:57 <denisse@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 12 hosts with reason: Downtiming the Logstash hosts serving OpenSearch Dashboards as part of the cergen to CFSSL migration - T360414 [production]
19:57 <denisse@cumin2002> START - Cookbook sre.hosts.downtime for 0:30:00 on 12 hosts with reason: Downtiming the Logstash hosts serving OpenSearch Dashboards as part of the cergen to CFSSL migration - T360414 [production]
19:46 <denisse> disabling Puppet on the Logstash hosts that serve OpenSearch dashboards to test the CFSSL certificates - T360414 [production]
2024-04-24 §
15:45 <jmm@cumin2002> END (PASS) - Cookbook sre.o11y.roll-restart-reboot-logstash-collectors (exit_code=0) rolling restart_daemons on A:logstash-collector [production]
15:37 <jmm@cumin2002> START - Cookbook sre.o11y.roll-restart-reboot-logstash-collectors rolling restart_daemons on A:logstash-collector [production]
13:23 <jmm@cumin2002> END (PASS) - Cookbook sre.o11y.roll-restart-reboot-logstash-collectors (exit_code=0) rolling restart_daemons on A:logstash-collector [production]
13:15 <jmm@cumin2002> START - Cookbook sre.o11y.roll-restart-reboot-logstash-collectors rolling restart_daemons on A:logstash-collector [production]
2024-04-10 §
20:15 <cjming@deploy1002> Finished scap: Backport for [[gerrit:1018738|LogStash: log HtmlOutputRendererHelper channel (T356157)]] (duration: 13m 51s) [production]
20:04 <cjming@deploy1002> cjming and daniel: Backport for [[gerrit:1018738|LogStash: log HtmlOutputRendererHelper channel (T356157)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
20:01 <cjming@deploy1002> Started scap: Backport for [[gerrit:1018738|LogStash: log HtmlOutputRendererHelper channel (T356157)]] [production]
2024-04-01 §
16:43 <eevans@deploy1002> Finished deploy [cassandra/logstash-logback-encoder@42653e6]: (no justification provided) (duration: 00m 33s) [production]