251-300 of 773 results (8ms)
2019-11-26 §
18:33 <shdubsh> stop logstash on logstash200[5-6] for metrics collection - T215904 [production]
2019-11-15 §
10:24 <godog> roll-restart logstash to apply configuration change [production]
2019-11-08 §
15:37 <herron> beginning rolling service restarts on logstash hosts for java security updates [production]
2019-11-07 §
10:58 <moritzm> installing Java security updates on kafka-main/logstash [production]
2019-10-31 §
20:27 <shdubsh> restarting logstash on logstash1008 to test level->severity filter selector [production]
10:54 <godog> bounce logstash on logstash2004 [production]
10:13 <godog> bounce logstash on logstash2004 [production]
09:37 <godog> temporarily stop logstash on logstash2005 to test performance with two ingesters only - T215904 [production]
09:23 <godog> temporarily stop logstash on logstash2006 to test performance with two ingesters only - T215904 [production]
2019-10-30 §
15:48 <godog> roll restart logstash after https://gerrit.wikimedia.org/r/c/operations/puppet/+/544217 [production]
2019-10-28 §
13:30 <godog> roll restart logstash in codfw/eqiad to apply new config [production]
08:48 <godog> bump udp_localhost kafka-logging topics to 6 partitions and roll-restart logstash and rsyslog - T215904 [production]
2019-10-24 §
07:42 <godog> bump rsyslog- topics partitions to 6 and roll-restart logstash frontends [production]
2019-10-23 §
09:40 <godog> roll restart logstash to pick up new rsyslog-notice partitions [production]
08:23 <godog> roll restart logstash in codfw/eqiad to pick up new kafka partitions [production]
07:28 <hashar> logstash: refreshing index fields for logstash-* indices (via https://logstash.wikimedia.org/app/kibana#/management/kibana/indices/logstash-* ) # T234564 [production]
2019-10-21 §
08:52 <godog> roll-restart logstash to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/544209 [production]
2019-10-07 §
18:16 <godog> roll-restart logstash to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/539978 [production]
2019-10-01 §
14:08 <herron> beginning rolling reboots of eqiad and codfw logstash collectors [production]
2019-09-21 §
01:08 <shdubsh> removed input-kafka-rsyslog-shipper-eqiad/codfw from logstash inputs logstash1008 and logstash1009 [production]
00:46 <shdubsh> restarting logstash on logstash1008 without udp-localhost-eqiad/codfw configs [production]
2019-08-06 §
19:50 <herron> disabling puppet on logstash collectors for rolling deploy of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/528306/ T166107 [production]
14:52 <herron> restarting logstash service on logstash1007 to pick up puppet managed log4j2 config [production]
2019-08-01 §
14:51 <herron> performing rolling restarts of eqiad logstash cluster for security updates [production]
2019-07-31 §
14:28 <herron> beginning rolling reboots of codfw logstash hosts for security updates [production]
2019-07-23 §
23:43 <shdubsh> reverting logstash mitigations and re-enable puppet [production]
22:06 <herron> puppet temporarily disabled on eqiad/codfw logstash collectors while catching up with backlog. see /etc/logstash/conf.d/01-filter_temp_drops.conf [production]
21:52 <herron> logstash - temporarily dropping logs matching [message] =~ /^SlowTimer/ due to UTF-8 parsing errors that are stopping the logstash processing pipeline. will re-enable after logstash has caught up with the backlog [production]
20:59 <shdubsh> temporarily disable input-kafka-rsyslog-shipper and drop memcached logs on logstash nodes [production]
19:06 <herron> restarting logstash on logstash100[789] [production]
17:52 <moritzm> installing Java security updates on kafka/main and Logstash servers [production]
2019-07-16 §
00:03 <shdubsh> restart logstash to revert mitigations - T228089 [production]
2019-07-15 §
19:05 <shdubsh> restarting logstash on logstash1008 [production]
2019-07-11 §
23:49 <eevans@deploy1001> Finished deploy [cassandra/logstash-logback-encoder@d085ffa]: deploy logback to restbase1017 (T222960) (duration: 00m 47s) [production]
23:48 <eevans@deploy1001> Started deploy [cassandra/logstash-logback-encoder@d085ffa]: deploy logback to restbase1017 (T222960) [production]
23:47 <eevans@deploy1001> Finished deploy [cassandra/logstash-logback-encoder@d085ffa]: (no justification provided) (duration: 01m 56s) [production]
23:45 <eevans@deploy1001> Started deploy [cassandra/logstash-logback-encoder@d085ffa]: (no justification provided) [production]
23:36 <eevans@deploy1001> Started deploy [cassandra/logstash-logback-encoder@d085ffa]: (no justification provided) [production]
2019-06-07 §
18:56 <herron> performing rolling reboots of logstash codfw frontends for security updates [production]
2019-06-06 §
19:47 <herron> performing rolling reboot of eqiad logstash hw for MDS security updates [production]
2019-06-04 §
18:10 <herron> correction — performing rolling reboots of codfw logstash hardware hosts for MDS security updates [production]
18:10 <herron> performing rolling reboots of eqiad logstash hardware hosts for MDS security updates [production]
2019-06-03 §
07:58 <elukey> refresh field list for logstash (via kibana Management -> Index patterns -> etc..) [production]
2019-05-14 §
20:44 <herron@deploy1001> Finished deploy [logstash/plugins@7fb8843]: adding logstash-filter-truncate plugin (duration: 00m 07s) [production]
20:43 <herron@deploy1001> Started deploy [logstash/plugins@7fb8843]: adding logstash-filter-truncate plugin [production]
20:41 <herron@deploy1001> Finished deploy [logstash/plugins@7fb8843]: (no justification provided) (duration: 00m 01s) [production]
20:41 <herron@deploy1001> Started deploy [logstash/plugins@7fb8843]: (no justification provided) [production]
19:37 <herron> adding logstash filter truncate plugin to prod logstash collectors [production]
09:02 <godog> statsd_exporter 0.9 upgrade on logstash - T220709 [production]
2019-04-15 §
18:35 <shdubsh> logstash1009: disabling puppet and testing logstash config [production]