701-750 of 773 results (10ms)
|
2014-12-11
§
|
| 16:28 |
<marktraceur> |
Synchronized private/PrivateSettings.php: [SWAT] [config] Add password for logstash (duration: 00m 10s) |
[production] |
|
2014-12-04
§
|
| 18:28 |
<bd808> |
Rolling restart of the elasticsearch cluster for logstash did not fix corrupted logstash-2014.11.30 index. It was worth a shot. |
[production] |
|
2014-12-01
§
|
| 15:49 |
<bd808> |
restarted logstash on logstash1001; log2udp events were not being processed |
[production] |
|
2014-11-26
§
|
| 21:33 |
<bd808> |
restarted logstash on logstash1001; log2udp events not being received |
[production] |
| 20:57 |
<bd808> |
All three elasticsaerch nodes in the logstash clsuter think logstash1003 is master but ogstash-2014.11.26 is not allocated on any node |
[production] |
| 01:23 |
<bd808> |
restarted logstash on logstash1001; no events from log2udp relay being recorded |
[production] |
|
2014-11-18
§
|
| 00:53 |
<bd808> |
Restarted logstash on logstash1002; lots of errors in the log about GELF input >128 chunks |
[production] |
| 00:50 |
<bd808> |
Restarted hung logstash process on logstash1001 |
[production] |
|
2014-11-14
§
|
| 16:40 |
<bd808> |
Increased replica count from 0 to 2 for all logstash elasticsearch indices. Expect icinga warnings as replicas are populated. |
[production] |
| 07:38 |
<jgage> |
logstash hosts: elasticsearch moved to bigger disks |
[production] |
|
2014-11-13
§
|
| 20:32 |
<bd808> |
Dropped replica count of all logstash indices except today to 0. Should make rolling restarts faster during hardware upgrade. |
[production] |
| 16:57 |
<bd808> |
dropped replica count to 0 for logstash indices from 2014-10-30 and 2014-10-31. |
[production] |
| 16:46 |
<bd808> |
dropped replica count to 0 for logstash indices from 2014-10-14 through 2014-10-29. See https://phabricator.wikimedia.org/P73 for the commands. |
[production] |
| 16:16 |
<bd808> |
logstash elasticsearch cluster is pretty messed up. logstash1002 has lost shards for all indices except for today, and it's master for that one. |
[production] |
|
2014-11-06
§
|
| 23:22 |
<bd808> |
restarted logstash on logstash1002 |
[production] |
| 23:21 |
<bd808> |
restarted logstash on logstash1003 |
[production] |
| 23:02 |
<bd808> |
restarted logstash on logstash1001 for the usual reason (no events making it to elasticsearch) |
[production] |
|
2014-11-03
§
|
| 23:34 |
<bd808> |
Changed elasticsearch template for logstash to use "doc_values" for raw fields. http://www.elasticsearch.org/blog/disk-based-field-data-a-k-a-doc-values/ |
[production] |
| 23:01 |
<cscott> |
reconfigured OCG logstash path to use bunyan. The _type field is currently missing (used to be "OfflineContentGenerator"). Will fix tomorrow. |
[production] |
|
2014-10-30
§
|
| 17:11 |
<bd808> |
Upgraded kibana to v3.1.1 again. Better testing now that logstash is working. |
[production] |
| 16:59 |
<bd808> |
restarted logstash on logstash1003. No events logged since 00:00Z |
[production] |
| 16:58 |
<bd808> |
restarted logstash on logstash1002. No events logged since 00:00Z |
[production] |
| 16:58 |
<bd808> |
restarted logstash on logstash1001. No events logged since 00:00Z |
[production] |
|
2014-10-29
§
|
| 15:19 |
<bd808> |
Restarted logstash on logstash1002 to fix OCG and hadoop log events not being recorded |
[production] |
| 15:15 |
<bd808> |
Restarted logstash on logstash1001. No MW events were being added to the index. |
[production] |
| 13:11 |
<manybubbles> |
lowered redundancy on logstash from 3 way to 2 way |
[production] |
|
2014-10-28
§
|
| 20:29 |
<manybubbles> |
removed /etc/elasticsearch/*.dpkg-dist fromg logstash machines - that was breaking logging for some reason. magic. |
[production] |
| 17:00 |
<bd808> |
restarted logstash on logstash1002 to try and get gelf input events into kibana again |
[production] |
| 16:57 |
<cscott> |
no logs for ocg/parsoid on logstash since 2014-10-27T18:50:46.104Z/2014-10-27T18:50:45.977Z (respectively) |
[production] |
|
2014-10-27
§
|
| 23:52 |
<bd808> |
Restarted logstash service on logstash1001 because I was not seeing any events from MW make it into kibana |
[production] |
| 23:27 |
<Reedy> |
restarted logstash on logstash1001 |
[production] |
|
2014-10-25
§
|
| 20:41 |
<bd808> |
updated logstash-* labs instances to salt minion 2014.1.11 (thanks for the ping apergos) |
[production] |
|
2014-10-22
§
|
| 21:38 |
<bd808> |
killed duplicate logstash services running on logstash1001 and restarted |
[production] |
| 15:57 |
<marktraceur> |
Synchronized wmf-config/InitialiseSettings.php: [SWAT] Send collection logs to logstash. (duration: 00m 05s) |
[production] |
|
2014-10-11
§
|
| 23:39 |
<Reedy> |
killed both logstash events on logstash100[23]. Started logstash again after |
[production] |
| 23:33 |
<Reedy> |
killed both logstash events on logstash1001. Started logstash again after |
[production] |
| 23:23 |
<Reedy> |
Started logstash on logstash1001 |
[production] |
| 23:21 |
<bd808> |
logstash not showing any udp2log events after 2014-10-10T01:42:22.000Z |
[production] |
|
2014-10-08
§
|
| 19:49 |
<Reedy> |
logstash upgraded to 1.4.2-1 on logstash100[1-3] |
[production] |
| 16:07 |
<Reedy> |
elasticsearch upgraded on logstash[ |
[production] |
|
2014-09-25
§
|
| 19:49 |
<bd808> |
Restarted logstash on logstash1001. udp2log events were not being recorded. |
[production] |
|
2014-09-22
§
|
| 21:04 |
<bd808> |
production-logstash-eqiad healed by restarting elasticsearch on logstash1002 after OOM + split brain |
[production] |
| 20:52 |
<bd808> |
logstash1002 went split brain from rest of logstash elastic search cluster. restarting |
[production] |
|
2014-09-15
§
|
| 17:02 |
<bd808> |
Restarted logstash on logstash1001. I hoped this would fix the dashboards, but it looks like the backing elasticsearch cluster is too sad for them to work at the moment. |
[production] |
|
2014-09-11
§
|
| 16:19 |
<bd808> |
Restarted logstash on logstash1001. Log empty and events not being stored in elasticsearch |
[production] |
|
2014-09-10
§
|
| 17:10 |
<bd808> |
Restarted logstash on logstash1001 |
[production] |
|
2014-08-22
§
|
| 15:08 |
<bd808> |
Still no apache2.log on fluorine or in logstash. Log seems to be available on fenari. |
[production] |
|
2014-08-21
§
|
| 00:07 |
<manybubbles> |
bd808 needs to plan a logstash upgrade soon - let it be logged |
[production] |
|
2014-08-18
§
|
| 14:31 |
<bd808> |
Restarted logstash on logstash1001; event volume was lower than expected |
[production] |
|
2014-07-29
§
|
| 16:10 |
<bd808> |
logstash log event volume up after restart |
[production] |