101-150 of 722 results (5ms)
2022-01-25 §
20:17 <cwhite> begin transition to logstash output opensearch plugin T299168 [production]
2022-01-19 §
17:58 <herron> beginning logstash apifeatureusage switchover T297239 [production]
2022-01-06 §
18:00 <btullis@deploy1002> Finished deploy [cassandra/logstash-logback-encoder@fb10de1] (aqs): Deploying logstash-logback-encoder to production (duration: 00m 09s) [production]
18:00 <btullis@deploy1002> Started deploy [cassandra/logstash-logback-encoder@fb10de1] (aqs): Deploying logstash-logback-encoder to production [production]
16:42 <otto@deploy1002> Finished deploy [cassandra/logstash-logback-encoder@fb10de1] (aqs): Deploying logstash-logback-encoder to production (duration: 00m 34s) [production]
16:41 <otto@deploy1002> Started deploy [cassandra/logstash-logback-encoder@fb10de1] (aqs): Deploying logstash-logback-encoder to production [production]
16:19 <btullis@deploy1002> Finished deploy [cassandra/logstash-logback-encoder@fb10de1] (aqs): Deploying logstash-logback-encoder to production (duration: 00m 41s) [production]
16:18 <btullis@deploy1002> Started deploy [cassandra/logstash-logback-encoder@fb10de1] (aqs): Deploying logstash-logback-encoder to production [production]
16:18 <btullis@deploy1002> Finished deploy [cassandra/logstash-logback-encoder@fb10de1] (aqs): Deploying logstash-logback-encoder to production (duration: 07m 16s) [production]
16:10 <btullis@deploy1002> Started deploy [cassandra/logstash-logback-encoder@fb10de1] (aqs): Deploying logstash-logback-encoder to production [production]
2021-12-16 §
17:29 <cwhite> pruned jndilookup.class from log4j-core on logstash 7 instances T297468 [production]
16:49 <cwhite> pruned jndilookup.class from log4j-core on logstash 5 instances T297468 [production]
2021-12-01 §
15:54 <godog> bounce logstash on eqiad/codfw to apply template changes [production]
2021-10-26 §
22:59 <legoktm> uploaded python-logstash to buster-wikimedia for T294393 [production]
2021-09-30 §
10:06 <godog> test bounce logstash on logstash1023 [production]
2021-09-22 §
18:12 <legoktm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: logging: send DuplicateParse bucket to Logstash (duration: 01m 05s) [production]
2021-09-20 §
13:45 <moritzm> restarting apache on Logstash ELK5 cluster to pick up GNUTLS update T283165 [production]
2021-08-27 §
10:31 <godog> bounce logstash on logstash1007 [production]
07:00 <godog> bounce logstash on logstash1008 [production]
2021-08-19 §
16:10 <cwhite> remove rotated logstash-plain-* and logstash-json-* logs on logstash collectors [production]
15:04 <godog> clean logstash json logs off logstash hosts [production]
2021-08-18 §
13:41 <godog> bounce logstash on logstash100[89] [production]
2021-08-16 §
16:37 <cwhite> restart logstash on logstash1008 [production]
2021-08-10 §
09:04 <moritzm> removing stale Java 8 packages from logstash1024/1025/2023/2024/2025 (ELK7 Logstash cluster is on Java 11 for a while now) [production]
2021-08-05 §
17:43 <herron> rolling restart eqiad logstash cluster for java updates [production]
15:54 <herron> rolling restart codfw logstash elasticsearch cluster for java updates [production]
2021-07-29 §
09:40 <jelto> uncordon kubestage1002.eqiad.wmnet as rsyslog was restarted and log shipping to logstash works again [production]
2021-07-16 §
09:47 <jayme> cordon kubestage1002.eqiad.wmnet as it currently does not feed logs to logstash [production]
2021-07-13 §
10:18 <moritzm> installing apache security updates on Logstash hosts [production]
2021-07-07 §
07:56 <moritzm> bounced elasticsearch_5@production-logstash-eqiad on logstash1009 [production]
2021-03-12 §
11:22 <hnowlan> corrected git_server for logstash-logback-encoder, cassandra/twcs and cassandra/metrics-collector on deploy1002 [production]
2021-03-10 §
21:30 <brennen> train status: 1.36.0-wmf.34 (T274938): logstash client error board was set up incorrectly; reverting earlier patch for T277094 and will proceed to group1. [production]
2021-03-01 §
21:30 <shdubsh> completed removal of kafka logging inputs to legacy logstash cluster - T234854 [production]
2021-02-17 §
16:46 <godog> roll-restart logstash to apply ulogd filter - T234565 [production]
2021-01-19 §
13:38 <godog> bounce logstash on logstash1025 to debug unindexable logs [production]
2021-01-05 §
20:04 <mutante> mw1344 - restarted apache2 - it was showing the same "partial results" error a mw1362 - no other appservers are showing up in logstash, but these were #1 and #2 source of errors [production]
2020-12-18 §
16:19 <shdubsh> restart logstash on logstash2004 [production]
2020-12-08 §
13:44 <godog> bounce logstash on logstash1023 [production]
13:44 <godog> bounce logstash on logstash102 [production]
2020-12-03 §
19:57 <shdubsh> restart logstash kafka in codfw - java updates [production]
19:44 <shdubsh> restart logstash kafka in eqiad - java updates [production]
2020-11-25 §
22:34 <shdubsh> beginning rolling restart of logstash cluster - eqiad [production]
17:44 <shdubsh> beginning rolling restart of logstash cluster - codfw [production]
2020-11-19 §
20:12 <herron> upgraded logstash-next to kibana 7.10 [production]
2020-10-13 §
15:02 <godog> bounce logstash on logstash1007, GC death [production]
2020-09-24 §
07:49 <godog> roll restart logstash codfw, gc death [production]
01:25 <ryankemper> Root cause of sigkill of `elasticsearch_5@production-logstash-eqiad.service` appears to be OOMKill of the java process: `Killed process 1775 (java) total-vm:8016136kB, anon-rss:4888232kB, file-rss:0kB, shmem-rss:0kB`. Service appears to have restarted itself and is healthy again [production]
01:21 <ryankemper> Observed that `elasticsearch_5@production-logstash-eqiad.service` is in a `failed` state since `Thu 2020-09-24 00:53:53 UTC`; appears the process received a SIGKILL - not sure why [production]
2020-09-14 §
21:30 <cdanis> T257527 ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕠🍺 sudo cumin 'R:Class ~ "(?i)profile::logstash::collector7"' 'enable-puppet "cdanis rolling out Ifa3c68e4"' [production]
21:24 <cdanis> T257527 ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕠🍺 sudo cumin 'R:Class ~ "(?i)profile::logstash::collector7"' 'disable-puppet "cdanis rolling out Ifa3c68e4"' [production]