701-750 of 783 results (10ms)
2015-01-08 §
16:02 <marktraceur> Synchronized wmf-config/logging.php: [SWAT] Honor log sampling and levels for logstash on group0 wikis (duration: 00m 05s) [production]
01:04 <bd808> cleaned up logstash indices dated 2014-01-* and 2015-12-* that look to have been created by some sort of syslog input parsing bug [production]
01:01 <bd808> accidentally deleted 2015-01-07 logstash index when cleaning up rogue indices for 2014-01-* [production]
00:38 <bd808> elasticsearch cluster for logstash is split brain. [production]
00:34 <bd808> restarted logstash on logstash1001 [production]
2015-01-05 §
17:00 <bd808> restarted logstash on logstash1001 to see if that will make syslog events come back [production]
16:58 <bd808> syslog events not being recorded in logstash as expected (apache2, hhvm) [production]
2014-12-31 §
00:42 <bd808> log2udp events still not making it into logstash; possibly related to earlier elasticsearch cluster issues; I don't want to restart elasticsearch on logstash1001 while the cluster is still recovering form that. [production]
00:33 <bd808> restarted logstash on logstash1001; log2udp events not being recorded in elasticsearch [production]
2014-12-11 §
23:44 <bd808> restarted logstash on logstash1001; fatalmonitor report was empty since ~20:30z [production]
16:28 <marktraceur> Synchronized private/PrivateSettings.php: [SWAT] [config] Add password for logstash (duration: 00m 10s) [production]
2014-12-04 §
18:28 <bd808> Rolling restart of the elasticsearch cluster for logstash did not fix corrupted logstash-2014.11.30 index. It was worth a shot. [production]
2014-12-01 §
15:49 <bd808> restarted logstash on logstash1001; log2udp events were not being processed [production]
2014-11-26 §
21:33 <bd808> restarted logstash on logstash1001; log2udp events not being received [production]
20:57 <bd808> All three elasticsaerch nodes in the logstash clsuter think logstash1003 is master but ogstash-2014.11.26 is not allocated on any node [production]
01:23 <bd808> restarted logstash on logstash1001; no events from log2udp relay being recorded [production]
2014-11-18 §
00:53 <bd808> Restarted logstash on logstash1002; lots of errors in the log about GELF input >128 chunks [production]
00:50 <bd808> Restarted hung logstash process on logstash1001 [production]
2014-11-14 §
16:40 <bd808> Increased replica count from 0 to 2 for all logstash elasticsearch indices. Expect icinga warnings as replicas are populated. [production]
07:38 <jgage> logstash hosts: elasticsearch moved to bigger disks [production]
2014-11-13 §
20:32 <bd808> Dropped replica count of all logstash indices except today to 0. Should make rolling restarts faster during hardware upgrade. [production]
16:57 <bd808> dropped replica count to 0 for logstash indices from 2014-10-30 and 2014-10-31. [production]
16:46 <bd808> dropped replica count to 0 for logstash indices from 2014-10-14 through 2014-10-29. See https://phabricator.wikimedia.org/P73 for the commands. [production]
16:16 <bd808> logstash elasticsearch cluster is pretty messed up. logstash1002 has lost shards for all indices except for today, and it's master for that one. [production]
2014-11-06 §
23:22 <bd808> restarted logstash on logstash1002 [production]
23:21 <bd808> restarted logstash on logstash1003 [production]
23:02 <bd808> restarted logstash on logstash1001 for the usual reason (no events making it to elasticsearch) [production]
2014-11-03 §
23:34 <bd808> Changed elasticsearch template for logstash to use "doc_values" for raw fields. http://www.elasticsearch.org/blog/disk-based-field-data-a-k-a-doc-values/ [production]
23:01 <cscott> reconfigured OCG logstash path to use bunyan. The _type field is currently missing (used to be "OfflineContentGenerator"). Will fix tomorrow. [production]
2014-10-30 §
17:11 <bd808> Upgraded kibana to v3.1.1 again. Better testing now that logstash is working. [production]
16:59 <bd808> restarted logstash on logstash1003. No events logged since 00:00Z [production]
16:58 <bd808> restarted logstash on logstash1002. No events logged since 00:00Z [production]
16:58 <bd808> restarted logstash on logstash1001. No events logged since 00:00Z [production]
2014-10-29 §
15:19 <bd808> Restarted logstash on logstash1002 to fix OCG and hadoop log events not being recorded [production]
15:15 <bd808> Restarted logstash on logstash1001. No MW events were being added to the index. [production]
13:11 <manybubbles> lowered redundancy on logstash from 3 way to 2 way [production]
2014-10-28 §
20:29 <manybubbles> removed /etc/elasticsearch/*.dpkg-dist fromg logstash machines - that was breaking logging for some reason. magic. [production]
17:00 <bd808> restarted logstash on logstash1002 to try and get gelf input events into kibana again [production]
16:57 <cscott> no logs for ocg/parsoid on logstash since 2014-10-27T18:50:46.104Z/2014-10-27T18:50:45.977Z (respectively) [production]
2014-10-27 §
23:52 <bd808> Restarted logstash service on logstash1001 because I was not seeing any events from MW make it into kibana [production]
23:27 <Reedy> restarted logstash on logstash1001 [production]
2014-10-25 §
20:41 <bd808> updated logstash-* labs instances to salt minion 2014.1.11 (thanks for the ping apergos) [production]
2014-10-22 §
21:38 <bd808> killed duplicate logstash services running on logstash1001 and restarted [production]
15:57 <marktraceur> Synchronized wmf-config/InitialiseSettings.php: [SWAT] Send collection logs to logstash. (duration: 00m 05s) [production]
2014-10-11 §
23:39 <Reedy> killed both logstash events on logstash100[23]. Started logstash again after [production]
23:33 <Reedy> killed both logstash events on logstash1001. Started logstash again after [production]
23:23 <Reedy> Started logstash on logstash1001 [production]
23:21 <bd808> logstash not showing any udp2log events after 2014-10-10T01:42:22.000Z [production]
2014-10-08 §
19:49 <Reedy> logstash upgraded to 1.4.2-1 on logstash100[1-3] [production]
16:07 <Reedy> elasticsearch upgraded on logstash[ [production]