701-750 of 799 results (8ms)
2015-01-24 §
22:27 <bd808> Full restart of logstash elasticsearch cluster [production]
2015-01-23 §
17:14 <bd808> logstash elasticsearch cluster split brained; logstash1002 thinks it is a lone master [production]
2015-01-16 §
23:33 <bd808> ran `LTRIM logstash -50000 9999999` on redis queues to drop ~4M events in backlog [production]
20:17 <bd808> Synchronized wmf-config/InitialiseSettings.php: Allow wgDebugLogGroups to exclude logstash append (e808e690) (duration: 00m 05s) [production]
20:17 <bd808> Synchronized wmf-config/logging.php: Allow wgDebugLogGroups to exclude logstash append (e808e690) (duration: 00m 07s) [production]
18:13 <bd808> document count not changing for logstash-2015.01.16 index [production]
17:59 <bd808> Synchronized wmf-config/logging-labs.php: beta: Allow wgDebugLogGroups to exclude logstash append (03c3ab27) (duration: 00m 06s) [production]
16:48 <bd808> Upgraded elasticsearch and restarted on all logstash nodes [production]
16:43 <bd808> shutdown whole elasticsearch cluster for logstash [production]
2015-01-15 §
20:04 <bd808> logstash redis queue backlog 384k events and climbing; likely related to the elasticsearch cluster flapping [production]
16:09 <bd808> Deleted 2015-12-* indices from logstash elasticsearch cluster [production]
16:01 <bd808> Elasticsearch cluster for logstash has indices for events dated 2015-12-* again [production]
2015-01-12 §
16:28 <bd808> deleted 2014-01-* and 2015-12-* indices from logstash elasticsearch cluster [production]
16:13 <bd808> logs on logstash1001 reporting elasticserch connection errors; restarted logstash service [production]
16:09 <bd808> logstash elasticsearch cluster has strange indices dated 2014-01-* and 2015-12-* again [production]
15:59 <bd808> logstash not showing any events at all since 2015-01-12T13:58:59.728Z [production]
2015-01-08 §
16:02 <marktraceur> Synchronized wmf-config/logging.php: [SWAT] Honor log sampling and levels for logstash on group0 wikis (duration: 00m 05s) [production]
01:04 <bd808> cleaned up logstash indices dated 2014-01-* and 2015-12-* that look to have been created by some sort of syslog input parsing bug [production]
01:01 <bd808> accidentally deleted 2015-01-07 logstash index when cleaning up rogue indices for 2014-01-* [production]
00:38 <bd808> elasticsearch cluster for logstash is split brain. [production]
00:34 <bd808> restarted logstash on logstash1001 [production]
2015-01-05 §
17:00 <bd808> restarted logstash on logstash1001 to see if that will make syslog events come back [production]
16:58 <bd808> syslog events not being recorded in logstash as expected (apache2, hhvm) [production]
2014-12-31 §
00:42 <bd808> log2udp events still not making it into logstash; possibly related to earlier elasticsearch cluster issues; I don't want to restart elasticsearch on logstash1001 while the cluster is still recovering form that. [production]
00:33 <bd808> restarted logstash on logstash1001; log2udp events not being recorded in elasticsearch [production]
2014-12-11 §
23:44 <bd808> restarted logstash on logstash1001; fatalmonitor report was empty since ~20:30z [production]
16:28 <marktraceur> Synchronized private/PrivateSettings.php: [SWAT] [config] Add password for logstash (duration: 00m 10s) [production]
2014-12-04 §
18:28 <bd808> Rolling restart of the elasticsearch cluster for logstash did not fix corrupted logstash-2014.11.30 index. It was worth a shot. [production]
2014-12-01 §
15:49 <bd808> restarted logstash on logstash1001; log2udp events were not being processed [production]
2014-11-26 §
21:33 <bd808> restarted logstash on logstash1001; log2udp events not being received [production]
20:57 <bd808> All three elasticsaerch nodes in the logstash clsuter think logstash1003 is master but ogstash-2014.11.26 is not allocated on any node [production]
01:23 <bd808> restarted logstash on logstash1001; no events from log2udp relay being recorded [production]
2014-11-18 §
00:53 <bd808> Restarted logstash on logstash1002; lots of errors in the log about GELF input >128 chunks [production]
00:50 <bd808> Restarted hung logstash process on logstash1001 [production]
2014-11-14 §
16:40 <bd808> Increased replica count from 0 to 2 for all logstash elasticsearch indices. Expect icinga warnings as replicas are populated. [production]
07:38 <jgage> logstash hosts: elasticsearch moved to bigger disks [production]
2014-11-13 §
20:32 <bd808> Dropped replica count of all logstash indices except today to 0. Should make rolling restarts faster during hardware upgrade. [production]
16:57 <bd808> dropped replica count to 0 for logstash indices from 2014-10-30 and 2014-10-31. [production]
16:46 <bd808> dropped replica count to 0 for logstash indices from 2014-10-14 through 2014-10-29. See https://phabricator.wikimedia.org/P73 for the commands. [production]
16:16 <bd808> logstash elasticsearch cluster is pretty messed up. logstash1002 has lost shards for all indices except for today, and it's master for that one. [production]
2014-11-06 §
23:22 <bd808> restarted logstash on logstash1002 [production]
23:21 <bd808> restarted logstash on logstash1003 [production]
23:02 <bd808> restarted logstash on logstash1001 for the usual reason (no events making it to elasticsearch) [production]
2014-11-03 §
23:34 <bd808> Changed elasticsearch template for logstash to use "doc_values" for raw fields. http://www.elasticsearch.org/blog/disk-based-field-data-a-k-a-doc-values/ [production]
23:01 <cscott> reconfigured OCG logstash path to use bunyan. The _type field is currently missing (used to be "OfflineContentGenerator"). Will fix tomorrow. [production]
2014-10-30 §
17:11 <bd808> Upgraded kibana to v3.1.1 again. Better testing now that logstash is working. [production]
16:59 <bd808> restarted logstash on logstash1003. No events logged since 00:00Z [production]
16:58 <bd808> restarted logstash on logstash1002. No events logged since 00:00Z [production]
16:58 <bd808> restarted logstash on logstash1001. No events logged since 00:00Z [production]
2014-10-29 §
15:19 <bd808> Restarted logstash on logstash1002 to fix OCG and hadoop log events not being recorded [production]