401-450 of 727 results (8ms)
2017-07-25 §
11:16 <moritzm> upgrading/restarting logstash* for openjdk security update [production]
2017-07-19 §
21:26 <mutante> logstash1001/1002 - restarted apache for CA change (logstash/kibana back) [production]
2017-07-17 §
10:02 <moritzm> installing apache updates on logstash [production]
2017-06-22 §
17:24 <gehel> restarting logstash on logstash1001 to validate plugin deplyoment with scap3 [production]
17:23 <gehel@tin> Finished deploy [logstash/plugins@720b648]: (no justification provided) (duration: 00m 02s) [production]
17:23 <gehel@tin> Started deploy [logstash/plugins@720b648]: (no justification provided) [production]
17:14 <gehel> moving to scap for logstash plugin deployment [production]
2017-06-07 §
11:08 <gehel> restarting cron on logstash cluster [production]
09:28 <gehel> upgrading kibana to v5.3.3 on logstash cluster - T167266 [production]
2017-06-02 §
12:11 <hashar> restarting Jenkins to upgrade the logstash plugin [production]
2017-05-12 §
11:35 <gehel> cleaning old elasticsearch and logstash logs on logstash cluster [production]
2017-05-08 §
20:27 <gehel> restarted kibana on logstash cluster - T161908 [production]
20:21 <gehel> upgrading kibana on logstash cluster - T161908 [production]
20:02 <gehel> restarting elasticsearch on logstash cluster after upgrade - T161908 [production]
19:47 <gehel> logstash / elasticsearch downtime coming up - T161908 [production]
19:28 <gehel> starting ELK (logstash) upgrade - T161908 [production]
2017-04-26 §
13:54 <gehel> downtime "ElasticSearch health check for shards" checks for logstash and elasticsearch eqiad - T148506 [production]
2017-04-18 §
11:14 <moritzm> upgrading logstash* to Linux 4.9 [production]
2017-04-10 §
21:33 <gehel> logstash upgrade on all logstash1* nodes completed- T161908 [production]
21:31 <gehel> upgrading logstash on logstash1003 - T161908 [production]
21:22 <gehel> upgrading logstash on logstash1002 - T161908 [production]
21:17 <gehel> logstash upgrade on logstash1001 completed - T161908 [production]
21:13 <gehel> running puppet on logstash1001 to deploy new logstash plugins - T161908 [production]
20:29 <gehel> upgrading logstash on logstash1001 - T161908 [production]
20:27 <ebernhardson> deployed new logstash plugins to logstash100[123] [production]
19:38 <gehel> starting logstash upgrade - some log messages will be lost! - T161908 [production]
2017-03-24 §
02:26 <Krinkle> Reminder to incident doc writer: Logstash was (and is) not responsive serving Kibana-rendered errors about logstash Service unavailable [production]
2017-02-23 §
13:52 <gehel> restart logstash on relforge1001 to test logging configuration - T158664 [production]
2017-02-22 §
00:48 <thcipriani@tin> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:323111|Send "exception" channel to logstash]] [[gerrit:323330|Do not send "exception-json" channel to logstash]] T136849 (duration: 00m 40s) [production]
2017-02-21 §
14:35 <moritzm> upgrading openssl on logstash cluster / various base service restarts [production]
2017-02-09 §
19:14 <bd808> Restarted logstash on logstash1001. Dead since 2017-02-09T06:39:46 with "java.lang.UnsupportedOperationException" crash in worker thread. [production]
2017-02-02 §
09:54 <moritzm> rolling restart of logstash cluster to pick up openjdk/NSS security updates [production]
2017-01-06 §
01:27 <bd808> Restarted logstash on logstash1002 (T154732) [production]
2017-01-05 §
05:44 <mutante> rolling out exim4 upgrades (DSA 3747-1) on db-core-codfw, etcd, graphite, kafka-analytics-canary, kafka-analytics, logstash [production]
2017-01-04 §
23:11 <aaron@tin> Synchronized wmf-config/logging.php: Include DB shard as a logstash column (duration: 00m 41s) [production]
2017-01-03 §
11:52 <akosiaris> reenabling ntpd on logstash eqiad boxes [production]
2017-01-01 §
00:55 <bd808> Restarted logstash on logstash1001 (T154388) [production]
2016-12-01 §
15:06 <gehel> upgrading logstash to Java 8, including rolling restart - T151325 [production]
2016-11-09 §
14:59 <gehel> clear zero sized log files on logstash* (leftover from disk space issues) [production]
2016-11-08 §
07:10 <_joe_> stopped logstash, removed large logfiles that were erroneously non-rotated, started logstash across the logstash cluster [production]
2016-11-07 §
07:02 <_joe_> removing old logfiles on logstash hosts [production]
2016-11-05 §
21:40 <bd808> Deleted huge logstash1003:/var/log/logstash/logstash.log.1 log file; disk full [production]
21:39 <bd808> Deleted huge logstash1002:/var/log/logstash/logstash.log.1 log file; disk full [production]
21:36 <bd808@tin> Synchronized wmf-config/InitialiseSettings.php: logstash: Temporarily disable EventBus channel (T150106) (duration: 00m 50s) [production]
19:54 <bd808> ELK stack problems are related to Elasticsearch index mapping. Some events are being rejected for not matching the expected mappings and that is filling up the disk on the logstash injestion hosts [production]
19:23 <bd808> Restarted logstash on logstash1001 [production]
19:14 <bd808> Deleted huge logstash1001:/var/log/logstash/logstash.log.1 log file; disk full and difficult to debug with no free space on / [production]
2016-10-27 §
08:24 <moritzm> rolling reboot of logstash cluster for kernel update [production]
2016-10-25 §
14:14 <elukey> removed logstash filter for Apache (https://logstash.wikimedia.org/app/kibana#/dashboard/apache2log) - T144005 [production]
2016-10-20 §
16:44 <gehel@puppetmaster1001> conftool action : set/pooled=yes; selector: dc=eqiad,cluster=logstash,service=kibana [production]