501-550 of 783 results (9ms)
2016-11-05 §
19:54 <bd808> ELK stack problems are related to Elasticsearch index mapping. Some events are being rejected for not matching the expected mappings and that is filling up the disk on the logstash injestion hosts [production]
19:23 <bd808> Restarted logstash on logstash1001 [production]
19:14 <bd808> Deleted huge logstash1001:/var/log/logstash/logstash.log.1 log file; disk full and difficult to debug with no free space on / [production]
2016-10-27 §
08:24 <moritzm> rolling reboot of logstash cluster for kernel update [production]
2016-10-25 §
14:14 <elukey> removed logstash filter for Apache (https://logstash.wikimedia.org/app/kibana#/dashboard/apache2log) - T144005 [production]
2016-10-20 §
16:44 <gehel@puppetmaster1001> conftool action : set/pooled=yes; selector: dc=eqiad,cluster=logstash,service=kibana [production]
2016-09-14 §
13:33 <gehel> upgrading logstash to elasticsearch 2.3.5 - T145404 [production]
2016-08-24 §
17:27 <gehel> deleting logstash indices from before august 1st (T142357) [production]
13:53 <gehel> rolling restart of logstash nodes to validate fix to T142357 [production]
12:29 <gehel> remove old logstash indices that were not deleted after 31 days [production]
2016-08-23 §
22:04 <bd808> Updated tin:/srv/deployment/logstash/plugins to d18b1c6 (Add output plugin for Sentry) [production]
2016-08-17 §
14:47 <gehel> rolling restart of elasticsearch logstash cluster for elasticsearch upgrade to 2.3.4 [production]
14:29 <gehel> upgrading elasticsearch plugins to 2.3.4 on elasticsearch, relforge and logstash clusters. Rolling restart coming next. [production]
2016-08-05 §
09:17 <moritzm> installing openjdk security updates on logstash cluster, rolling restart to pick up new JRE [production]
2016-08-04 §
18:49 <thcipriani@tin> Synchronized README: test logstash host (duration: 00m 51s) [production]
2016-08-01 §
16:10 <bd808> Restarted logstash on logstash1001; missing hhvm and service logs; no output to /var/log/logstash/logstash.log for days [production]
2016-07-21 §
17:07 <gehel> cleaning leftover crons on logstash* servers - T140973 [production]
2016-07-18 §
21:34 <ebernhardson> changed replica count for logstash-2016.06-(01|02|03|15|16|17) indices back to 2 [production]
21:09 <ebernhardson> changed replica count for logstash-2016.06-(01|02|03|15|16|17) indices to 0 to make room for recovering todays index [production]
19:54 <ebernhardson> re-stopping logstash on logstash1001-3 [production]
19:52 <gehel> disabling puppet on logstash.* nodes for elasticsearch upgrade [production]
19:38 <bd808> Dropped logstash-2016.07.04 through logstash-2016.07.14 indices for backing Elasticsearch upgrade [production]
19:36 <ebernhardson> shutting down logstash and elasticsearch on logstash1001-03 [production]
19:04 <gehel> starting elasticsearch upgrade for logstash (T136001) [production]
2016-07-11 §
21:03 <bd808> Updated default mapping for logstash-* index creation using json generated by https://gerrit.wikimedia.org/r/#/c/298295/. Should take effect starting with the logstash-2016.07.12 index. [production]
2016-07-09 §
19:54 <bd808> restarted logstash on logstash1003 for de-dot plugin update (T136001) [production]
19:52 <bd808> restarted logstash on logstash1002 for de-dot plugin update (T136001) [production]
19:50 <bd808> restarted logstash on logstash1001 for de-dot plugin update (T136001) [production]
19:46 <bd808> Updated logstash/plugins to 18b3f1f (Fix de_dot to process keys with falsey values) [production]
2016-07-07 §
20:15 <gehel> logstash upgrade aborted, rescheduled to Monday July 11th [production]
20:10 <bd808> Restarted logstash on logstash1001; hoping to clear up missing de-dot errors [production]
20:07 <bd808> Restarted logstash on logstash1002; hoping to clear up missing de-dot errors [production]
19:57 <bd808> Restarted logstash on logstash1003; hoping to clear up missing de-dot errors [production]
16:24 <gehel> starting elasticsearch and kibana upgrade on logstash cluster (T136001) [production]
15:44 <bd808> Dropped logstash indices older than logstash-2016.07.01 in preparation for elasticsearch upgrade [production]
2016-06-27 §
17:23 <gehel> deploying new logstash config for transition to elasticsearch 2.x (T138335) [production]
2016-06-23 §
20:31 <ebernhardson> synced out latest logstash-plugins via trebuchet [production]
2016-06-21 §
19:27 <bd808> Restarted dead logstash process on logstash1001. Looks to have stopped itself due to the the Elasticsearch OOM earlier [production]
09:22 <moritzm> rolling reboot of logstash cluster to Linux 4.4 [production]
2016-06-15 §
15:17 <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:294462|Send authentication events to logstash]] (duration: 00m 28s) [production]
2016-06-13 §
13:55 <dcausse> restarting logstash on logstash1001 [production]
08:51 <moritzm> removed /var/log/logstash/logstash.log.1 on logstash1001, depleted disk space on the root partition, fallout of T137400 [production]
2016-06-09 §
07:22 <moritzm> removed /var/log/logstash/logstash.1 on logstash1001, logspam (similar to the what is described in https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/144) depleted the space on the root partition [production]
2016-05-27 §
23:58 <ori> Forcing a Puppet run on logstash* [production]
2016-05-24 §
11:21 <moritzm> rolling restart of elasticsearch in logstash cluster to pick up openjdk security update [production]
2016-05-18 §
22:00 <akosiaris> restart cassandra on maps-test200{2,3,4} to get logstash changing working [production]
2016-05-09 §
20:29 <gehel> restarting logstash server logstash100[26].eqiad.wmnet (T110236) [production]
20:26 <gehel> restarting logstash server logstash1001.eqiad.wmnet (T110236)upgrade [production]
2016-03-31 §
10:24 <moritzm> logstash on logstash100[1-3] is now using systemd [production]
09:50 <moritzm> disabled puppet on logstash1002/1003 (to activate the new logstash systemd unit in steps) [production]