651-700 of 799 results (9ms)
2015-08-11 §
17:27 <bd808> logstash event volume recovered after restarting all 3 logstash services [production]
17:14 <bd808> log event volume in logstash dropped dramatically at 16:49; investigating [production]
17:13 <bd808> logstash cluster recovered after upgrade of elasticsearch on logstash1004 [production]
16:42 <bd808> upgraded elasticsearch to 1.7.1 on logstash1004; logstash-2015.08.11 shard recovering [production]
16:23 <bd808> logstash upgrade on logstash1003 complete [production]
16:20 <bd808> logstash upgrade on logstash1002 complete [production]
16:16 <bd808> logstash upgrade on logstash1001 complete [production]
15:49 <bd808> upgrading logstash on logstash1001 [production]
15:47 <bd808> Trebuchet deploy of logstash/plugins: Add logstash-filter-prune 0.1.5 (36144b2) [production]
15:31 <bd808@tin> Synchronized wmf-config/InitialiseSettings.php: logging: Only send info and higher to logstash by default (4388a84) 2/2 (actually rebased this time) (duration: 00m 11s) [production]
15:30 <bd808@tin> Synchronized wmf-config/logging.php: logging: Only send info and higher to logstash by default (4388a84) 1/2 (actually rebased this time) (duration: 00m 11s) [production]
15:12 <bd808@tin> Synchronized wmf-config/InitialiseSettings.php: logging: Only send info and higher to logstash by default (4388a84) 2/2 (duration: 00m 12s) [production]
15:11 <bd808@tin> Synchronized wmf-config/logging.php: logging: Only send info and higher to logstash by default (4388a84) 1/2 (duration: 00m 12s) [production]
2015-07-31 §
15:52 <bd808> Rebuilt grafana-dashboards index to have 1 shard/2 replicas in logstash cluster [production]
15:46 <bd808> Rebuilt kibana-int index to have 1 shard/2 replicas in logstash cluster [production]
2015-07-29 §
20:26 <urandom> bouncing cassandra on restbase1006 to apply logstash config [production]
20:18 <urandom> bouncing cassandra on restbase1005 to apply logstash config [production]
20:15 <urandom> bouncing cassandra on restbase1004 to apply logstash config [production]
20:11 <urandom> bouncing cassandra on restbase1003 to apply logstash config [production]
20:04 <urandom> bouncing cassandra on restbase1002 to apply logstash config [production]
19:59 <urandom> restarting restbase1001 to apply logstash config [production]
2015-07-28 §
16:01 <godog> bounce cassandra on xenon to test logstash logging [production]
15:52 <bd808> installed logstash on logstash1002; forced puppet run [production]
2015-07-27 §
20:54 <bd808> installed libbcprov-java and restarted logstash on logstash1001 [production]
18:38 <bd808> Synchronized wmf-config/InitialiseSettings.php: logstash: change ip address for logstash1001 and logstash1003 (duration: 00m 12s) [production]
2015-07-11 §
04:21 <bd808> Logstash cluster upgrade complete! Kibana working again [production]
2015-07-08 §
23:06 <bd808> Restarted logstash on logstash1001; no hhvm input seen for last hour [production]
2015-07-05 §
22:30 <bd808> Restarted logstash on logstah1001; Hung due to OOM errors [production]
2015-07-02 §
16:40 <bd808> Restarted logstash on logstash1001 due to OOM [production]
2015-06-26 §
23:57 <bd808> Logstash log ingestion working again after forcing recovery of replicas for logstash-2015.06.26; new logs were being rejected with only a primary shard available [production]
23:54 <bd808> re-enabled allocation on logstash elasticsearch cluster [production]
22:36 <bd808> set indices.recovery.concurrent_streams to 4 on logstash ES cluster [production]
22:36 <godog> set indices.recovery.max_bytes_per_sec to 10mb on logstash ES cluster [production]
22:25 <godog> set indices.recovery.max_bytes_per_sec to 50mb on logstash ES cluster [production]
22:09 <bd808> restarted logstash on logstash1001 [production]
20:10 <bd808> Deleted 4 corrupt indices (logstash-2015.05.30 logstash-2015.05.31 logstash-2015.06.03 logstash-2015.06.06) on logstash1004 [production]
2015-06-04 §
21:27 <bd808> restarted logstash and elasticsearch on logstash100[1-3] to pick up latest jre updates [production]
2015-05-19 §
23:20 <bd808> Synchronized wmf-config/InitialiseSettings.php: logstash: Exclude jobrunner debug messages (duration: 00m 12s) [production]
2015-05-11 §
15:05 <manybubbles> Synchronized wmf-config/InitialiseSettings.php: SWAT: send all mediawiki events from all wikis to logstash (duration: 00m 12s) [production]
2015-05-06 §
15:04 <bd808> Synchronized wmf-config/InitialiseSettings.php: Send group0 + group1 MediaWiki events to logstash {{gerrit|209170}} (duration: 00m 16s) [production]
2015-05-04 §
01:13 <bd808> Started logstash cluster relocating indices off of logstash100[1-3] to logstash100[4-6] [production]
2015-04-23 §
22:10 <bd808> Synchronized wmf-config/logging.php: logstash: Fix log level detection (c09014d) (duration: 00m 17s) [production]
2015-04-14 §
15:48 <bd808> Restarted logstash on logstash1003.eqiad.wmnet; subbu reported missing parsoid log events [production]
2015-02-11 §
19:36 <subbu> temporarily turn off logging to logstash till logstash isssues are resolved. [production]
2015-02-05 §
17:47 <ori> Synchronized wmf-config/logging.php: Live hack: disable Logstash logging on suspicion that it is acting up (duration: 00m 05s) [production]
00:54 <bd808> truncated redis input queues for logstash on all 3 hosts to see if cluster can keep up now with 3 elasticsearch writer threads [production]
2015-02-03 §
22:53 <bd808> starting rolling restart of logstash elasticsearch cluster to pick up index.merge.scheduler.max_thread_count puppet change [production]
01:51 <bd808> restarted logstash on logstash1001 [production]
2015-01-25 §
18:44 <bd808> trimmed Logstash redis input queues to 0 events; dropped ~4M backlogged events [production]
2015-01-24 §
22:40 <bd808> Emptied logstash redis lists on all 3 hosts [production]