351-400 of 799 results (6ms)
2019-02-27 §
11:17 <godog> roll-restart pybal after removing logstash log4j service [production]
2019-02-21 §
15:07 <herron> migrating ES shards away from logstash100[456] with "cluster.routing.allocation.exclude._name" : "logstash1004-production-logstash-eqiad,logstash1005-production-logstash-eqiad,logstash1006-production-logstash-eqiad” T214608 [production]
2019-02-20 §
16:49 <herron> migrating es shards away from logstash100[56] with "cluster.routing.allocation.exclude._name" : "logstash1005-production-logstash-eqiad,logstash1006-production-logstash-eqiad” T214608 [production]
2019-02-19 §
17:55 <herron> temporarily increased eqiad logstash elasticsearch low disk watermark to 87% (will restore to 85% when eqiad expansion hosts are fully online) [production]
2019-02-06 §
00:24 <niharika29@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Demystify Logstash debug level behavior (duration: 00m 51s) [production]
00:23 <niharika29@deploy1001> Synchronized wmf-config/logging.php: Demystify Logstash debug level behavior (duration: 00m 46s) [production]
2019-02-05 §
15:15 <godog> force curator action 'replicas' to set older logstash indices to 1 replica - T213078 [production]
2019-01-22 §
13:55 <godog> bump logstash kafka consumer threads - T214309 [production]
2019-01-15 §
17:26 <godog> roll-restart logstash in eqiad - T213081 [production]
17:21 <godog> depool logstash1007 before restarting logstash - T213081 [production]
2018-12-26 §
16:37 <godog> clear logstash persistent queue /var/lib/logstash on logstash100[789] [production]
15:47 <godog> roll-restart logstash on logstash100[789] [production]
2018-12-17 §
18:34 <herron@puppetmaster1001> conftool action : set/pooled=yes; selector: dc=codfw,cluster=logstash,service=kibana,name=logstash2006.codfw.wmnet [production]
18:34 <herron@puppetmaster1001> conftool action : set/pooled=yes; selector: dc=codfw,cluster=logstash,service=kibana,name=logstash2005.codfw.wmnet [production]
18:04 <herron@puppetmaster1001> conftool action : set/pooled=yes; selector: dc=codfw,cluster=logstash,service=kibana,name=logstash2004.codfw.wmnet [production]
2018-12-14 §
08:47 <elukey> disabled kafkatee-webrequest logstash output on oxygen (prep step before weblog1001) [production]
2018-11-08 §
09:37 <godog> keep 2x not 3x copies of older (>15d) logstash elasticsearch indices [production]
09:29 <godog> temporarily set elasticsearch logstash watermark to low:0.85 and high:0.9 [production]
2018-10-31 §
20:27 <ladsgroup@deploy1001> Finished deploy [ores/deploy@70ba14b]: Upgrade to celery4 and flask 0.12.4, logstash fixes: T181546 T181630 T168921 T205256 T169586 T208258 T178441 (duration: 21m 29s) [production]
20:06 <ladsgroup@deploy1001> Started deploy [ores/deploy@70ba14b]: Upgrade to celery4 and flask 0.12.4, logstash fixes: T181546 T181630 T168921 T205256 T169586 T208258 T178441 [production]
2018-10-24 §
14:23 <herron> scheduled icinga downtime and disabling puppet on logstash hosts. deploying role::kafka::logging to logstash elasticserach data hosts [production]
2018-10-18 §
17:33 <ladsgroup@deploy1001> Finished deploy [ores/deploy@4ac4c8b]: Logstash support for ores: T181546 T169586 T168921 T181630 T205256 (duration: 23m 48s) [production]
17:19 <herron> aborted enabling kafka on logstash elasticsearch cluster due to puppet errors. reverted change T206454 [production]
17:09 <ladsgroup@deploy1001> Started deploy [ores/deploy@4ac4c8b]: Logstash support for ores: T181546 T169586 T168921 T181630 T205256 [production]
16:57 <herron> enabling kafka on logstash elasticsearch cluster T206454 [production]
2018-09-25 §
16:59 <gehel> forcing puppet run on all elastic nodes (including logstash) to recover prometheus metric exporter [production]
2018-09-19 §
16:52 <gehel> rolling restart of elasticsearch / logstash for new systemd unit completed - new systemd unit is elasticsearch_5@production-logstash-eqiad [production]
12:18 <gehel> rolling restart of elasticsearch / logstash for new systemd unit [production]
2018-08-31 §
08:57 <gehel> elasticsearch data directory migration on all logstash nodes [production]
2018-08-30 §
16:15 <gehel> restart of logstash to move data directory - T198351 [production]
2018-08-27 §
15:12 <herron> decreasing logstash elasticsearch index replica count to 1 on indices older than 1 day [production]
15:12 <godog> set transient low watermark to 80% for elasticsearch logstash cluster to allow shard replica allocation - T201971 [production]
2018-08-09 §
19:02 <herron> logstash1008:~# systemctl restart logstash T200960 [production]
2018-08-08 §
17:55 <herron> performing rolling restart of logstash instances T200960 [production]
2018-08-07 §
19:45 <herron> upload prometheus-logstash-exporter_0.1.2-1 to apt.wikimedia.org/jessie-wikimedia/main [production]
19:45 <herron> upload prometheus-logstash-exporter_0.1.2-1 to apt.wikimedia.org/stretch-wikimedia/main [production]
14:18 <herron> double again logstash jvm heap size to 1g and rolling restart logstash instances [production]
10:33 <godog> bounce logstash on logstash1007 for tests [production]
2018-08-06 §
18:04 <herron> double logstash jvm heap size from 256m to 512m and rolling restart of logstash instances T200960 [production]
08:23 <_joe_> restarting logstash on logstash1007, losing packets as well [production]
08:03 <_joe_> restarting logstash on logstash1009, losing packets again [production]
07:51 <_joe_> restarting logstash on logstash1008, losing packets again [production]
2018-08-04 §
14:50 <godog> roll-restart logstash because of increasing packet loss [production]
2018-08-02 §
11:17 <godog> temporarily remove multiline filter from logstash to allow using multiple workers - T200960 [production]
07:20 <TimStarling> restarting logstash on logstash1008 and logstash1009 [production]
07:06 <TimStarling> on logstash1007 increased net.core.rmem_default from 212992 to 851968 in soft state and restarted logstash [production]
06:59 <TimStarling> on logstash1007 restarting logstash [production]
2018-07-27 §
19:04 <mutante> terbium is being removed from ferm rules on elasticsearch/relforge, logstash/collector, mariadb/labtestwikitech and mw-maintenance itself (T192092) [production]
2018-06-25 §
23:30 <twentyafterfour> After monitoring logstash, everything appears stable. Evening SWAT is complete. [production]
2018-06-06 §
20:47 <ebernhardson> sighup logstash on logstash100[789] to reload config for gerrit.wikimedia.org/r/437657 [production]