<p dir="auto">java version "1.8.0_92"<br>
ES 2.3.4</p>
<p dir="auto">So, I have 4 nodes, each with 32GB dedicated to the heap space. After about 2 days the heap makes it's way up to over 90% and I start to get rolling outage. Checking the stats I notice that ALL of the heap stays in the old generation pool and never seems to clear out. I've tried switching to G1GC which seemed to help a little bit, I've tried messing with the NewGen Ratio and all sorts of other things, but in the end the old generation never clears and the cluster is unstable until I start restarting nodes.</p>
<p dir="auto">Current Run Options:</p>
<p dir="auto">/platform/jdk/bin/java -Xms32g -Xmx32g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintClassHistogram -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/var/log/elasticsearch/gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-UseParNewGC -XX:-UseConcMarkSweepGC -XX:+UseG1GC -XX:+PrintGCDetails -XX:NewRatio=4 -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-2.3.4.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch start -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.conf=/etc/elasticsearch</p>
<p dir="auto">elasticsearch.yaml</p>
<h1 dir="auto">Cluster Settings</h1>
<p dir="auto">cluster.name: elasticsearch</p>
<p dir="auto">node.rack: "East"<br>
node.name: "es-1"<br>
node.max_local_storage_nodes: 1</p>
<h1 dir="auto">Path Settings</h1>
<p dir="auto">path.conf: /etc/elasticsearch<br>
path.data: /storage/elasticsearch</p>
<h1 dir="auto">Lock memory for ES</h1>
<p dir="auto">bootstrap.mlockall: true</p>
<h1 dir="auto">Networking</h1>
<p dir="auto">network.host: [ <em>local</em>, <em>bond0</em> ]<br>
discovery.zen.minimum_master_nodes: 3<br>
discovery.zen.ping.unicast.hosts: ["es-1", "es-2", "es-3", "es-4"]</p>
<h1 dir="auto">CLUSTER</h1>
<p dir="auto">cluster.routing.allocation.disk.watermark.high: "95%"<br>
cluster.routing.allocation.node_concurrent_recoveries: 30<br>
cluster.routing.allocation.cluster_concurrent_rebalance: 6</p>
<h1 dir="auto">INDICES</h1>
<p dir="auto">indices.store.throttle.max_bytes_per_sec: 400mb</p>
<h1 dir="auto">Recovery</h1>
<p dir="auto">indices.recovery.max_bytes_per_sec: 1000mb<br>
indices.recovery.concurrent_streams: 10<br>
gateway.recover_after_data_nodes: 2<br>
gateway.recover_after_time: 1m<br>
gateway.expected_data_nodes: 2</p>
<h1 dir="auto">Field Data tweaks</h1>
<h1 dir="auto">indices.fielddata.cache.size:  20% #Evictions a large amount of garbage in memory, which must be garbage collected later on</h1>
<h1 dir="auto">Circuit Breaker</h1>
<p dir="auto">indices.breaker.fielddata.limit: 25%<br>
indices.breaker.request.limit: 25%<br>
indices.breaker.total.limit: 50%</p>
<h1 dir="auto">SLOW LOG</h1>
<p dir="auto">index.search.slowlog.threshold.query.warn: 10s<br>
index.search.slowlog.threshold.fetch.warn: 1s<br>
index.indexing.slowlog.threshold.index.warn: 10s</p>
<h1 dir="auto">GC LOGGING</h1>
<p dir="auto">monitor.jvm.gc.young.warn: 1000ms<br>
monitor.jvm.gc.young.info: 700ms<br>
monitor.jvm.gc.young.debug: 400ms<br>
monitor.jvm.gc.old.warn: 10s<br>
monitor.jvm.gc.old.info: 5s<br>
monitor.jvm.gc.old.debug: 2s</p>
<p dir="auto">We run all queries with <code class="notranslate">timeout=160s</code>, but I understand this only really bounds the collection phase of the search?</p>
<p dir="auto">We had someone run a query using aggregations today over billions of records, which brought the cluster down.<br>
The cluster never OOM'd, but it did run into constant GC as the heap got full.<br>
Another query was run again over potentially huge amounts of data. The cluster had indexing disabled as it was recovering from the previous event, and since the query was run it's been at 100% CPU for about 40 minutes now. The query, afaict, is still running. But we have no way of knowing what it is, and no way to kill it other than restarting the entire cluster.</p>
<p dir="auto">I'd like to request a feature that lists all queries that are currently executing on every node, as well a way to kill them while they're in progress.</p>