<p dir="auto"><strong>Elasticsearch version</strong>:</p>
<p dir="auto">5.0.0~alpha2</p>
<p dir="auto"><strong>JVM version</strong>:</p>
<p dir="auto">java version "1.8.0_91"<br>
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)<br>
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)</p>
<p dir="auto"><strong>OS version</strong>:</p>
<p dir="auto">Linux ip-10-10-155-12 3.13.0-74-generic <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="165835" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/118" data-hovercard-type="issue" data-hovercard-url="/elastic/elasticsearch/issues/118/hovercard" href="https://github.com/elastic/elasticsearch/issues/118">#118</a>-Ubuntu SMP Thu Dec 17 22:52:10 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux</p>
<p dir="auto"><strong>Description of the problem including expected versus actual behavior</strong>:</p>
<p dir="auto"><strong>Steps to reproduce</strong>:</p>
<ol dir="auto">
<li>Create a large number of documents</li>
<li>Attempt to delete the documents using something like a match_all query and the delete-by-query plugin</li>
<li>Boom!</li>
</ol>
<p dir="auto"><strong>Provide logs (if relevant)</strong>:</p>
<p dir="auto">You may see any combination of these:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[2016-05-15 00:38:29,675][ERROR][action.deletebyquery     ] [client] scroll request [...] failed, scrolling document(s) is stopped
Failed to execute phase [query], all shards failed; shardFailures {RemoteTransportException[[worker][10.10.155.231:9300][indices:data/read/search[phase/query/scroll]]]; nested: SearchContextMissingException[No search context found for id [3894]]; }{RemoteTransportException[[worker][10.10.155.184:9300][indices:data/read/search[phase/query/scroll]]]; nested: SearchContextMissingException[No search context found for id [3894]]; }{RemoteTransportException[[worker][10.10.155.83:9300][indices:data/read/search[phase/query/scroll]]]; nested: SearchContextMissingException[No search context found for id [3897]]; }{RemoteTransportException[[worker][10.10.155.181:9300][indices:data/read/search[phase/query/scroll]]]; nested: SearchContextMissingException[No search context found for id [3894]]; }{RemoteTransportException[[worker][10.10.155.192:9300][indices:data/read/search[phase/query/scroll]]]; nested: SearchContextMissingException[No search context found for id [3897]]; }
        at org.elasticsearch.action.search.SearchScrollQueryThenFetchAsyncAction.onQueryPhaseFailure(SearchScrollQueryThenFetchAsyncAction.java:155)
        at org.elasticsearch.action.search.SearchScrollQueryThenFetchAsyncAction$1.onFailure(SearchScrollQueryThenFetchAsyncAction.java:142)
        at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51)
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:795)
        at org.elasticsearch.transport.netty.MessageChannelHandler.handleException(MessageChannelHandler.java:204)
        at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:194)
        at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:141)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[ERROR][action.deletebyquery     ] [client] scroll request [...] failed, scrolling document(s) is stopped"><pre class="notranslate"><code class="notranslate">[2016-05-15 00:38:29,675][ERROR][action.deletebyquery     ] [client] scroll request [...] failed, scrolling document(s) is stopped
Failed to execute phase [query], all shards failed; shardFailures {RemoteTransportException[[worker][10.10.155.231:9300][indices:data/read/search[phase/query/scroll]]]; nested: SearchContextMissingException[No search context found for id [3894]]; }{RemoteTransportException[[worker][10.10.155.184:9300][indices:data/read/search[phase/query/scroll]]]; nested: SearchContextMissingException[No search context found for id [3894]]; }{RemoteTransportException[[worker][10.10.155.83:9300][indices:data/read/search[phase/query/scroll]]]; nested: SearchContextMissingException[No search context found for id [3897]]; }{RemoteTransportException[[worker][10.10.155.181:9300][indices:data/read/search[phase/query/scroll]]]; nested: SearchContextMissingException[No search context found for id [3894]]; }{RemoteTransportException[[worker][10.10.155.192:9300][indices:data/read/search[phase/query/scroll]]]; nested: SearchContextMissingException[No search context found for id [3897]]; }
        at org.elasticsearch.action.search.SearchScrollQueryThenFetchAsyncAction.onQueryPhaseFailure(SearchScrollQueryThenFetchAsyncAction.java:155)
        at org.elasticsearch.action.search.SearchScrollQueryThenFetchAsyncAction$1.onFailure(SearchScrollQueryThenFetchAsyncAction.java:142)
        at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51)
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:795)
        at org.elasticsearch.transport.netty.MessageChannelHandler.handleException(MessageChannelHandler.java:204)
        at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:194)
        at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:141)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[ERROR][action.deletebyquery     ] [client] scroll request [...] failed, scrolling document(s) is stopped
</code></pre></div>
<p dir="auto">or:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="2016-05-15 23:38:55,644WARNrest.suppressed /_bulk Params: {}
CircuitBreakingException[parent Data too large, data for &lt;http_request&gt; would be larger than limit of 2994274304/2.7gb]
at org.elasticsearch.indices.breaker.HierarchyCircuitBreakerService.checkParentLimit(HierarchyCircuitBreakerService.java:211)
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:128)
..."><pre class="notranslate"><code class="notranslate">2016-05-15 23:38:55,644WARNrest.suppressed /_bulk Params: {}
CircuitBreakingException[parent Data too large, data for &lt;http_request&gt; would be larger than limit of 2994274304/2.7gb]
at org.elasticsearch.indices.breaker.HierarchyCircuitBreakerService.checkParentLimit(HierarchyCircuitBreakerService.java:211)
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:128)
...
</code></pre></div>
<p dir="auto">Yup, 2.7 gigs!</p>
<p dir="auto">Since I had quite a few of these queries going in tandem, you can guess what happened next. Several of the worker nodes pegged at 200% CPU and started to use up all swap and ram (44gigs for the elasticsearch process in most cases). Searches would take 15 minutes.</p>
<p dir="auto">Looking at the task list I see a big backlog of scroll requests and delete-by-query tasks, none of them cancellable. Not sure what I can do here except kill the index and start again.</p>
<p dir="auto">Looking quickly at <a href="https://github.com/elastic/elasticsearch/blob/master/plugins/delete-by-query/src/main/java/org/elasticsearch/action/deletebyquery/TransportDeleteByQueryAction.java">https://github.com/elastic/elasticsearch/blob/master/plugins/delete-by-query/src/main/java/org/elasticsearch/action/deletebyquery/TransportDeleteByQueryAction.java</a> I'm guessing that the plugin just builds a bulk delete request without considering how large it might get.</p>
<p dir="auto">I think some first steps might be:</p>
<ol dir="auto">
<li>Make this plugin cancellable, not sure what this involves but there seems plenty of opportunity in between pages of the scroll request to check a cancellation flag.</li>
<li>At least error out before attempting to build queries of such a large size, the search loop could probably check the <code class="notranslate">BulkRequest#estimatedSizeInBytes</code> against the cluster's max request size limits.</li>
<li>Chunk bulk requests into smaller components and deliver them piecemeal</li>
</ol>
<p dir="auto"><strong>Elasticsearch version</strong>: alpha2</p>
<p dir="auto"><strong>JVM version</strong>: build 1.8.0_74-b02</p>
<p dir="auto"><strong>OS version</strong>: OS X El Capitan 10.11.3</p>
<p dir="auto"><strong>Description of the problem including expected versus actual behavior</strong>:</p>
<p dir="auto"><strong>Steps to reproduce</strong>:</p>
<ol dir="auto">
<li>
<p dir="auto">Install and run elasticsearch-alpha2, topbeat-alpha2, and kibana-alpha2 (Topbeat is only monitoring the node process on a 20 second interval.)</p>
</li>
<li>
<p dir="auto">I am using Kibana to monitor a node process. Here is the query Kibana is using to generate the visualization:</p>
<div class="highlight highlight-source-js notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
   &quot;size&quot;:0,
   &quot;aggs&quot;:{
      &quot;2&quot;:{
         &quot;date_histogram&quot;:{
            &quot;field&quot;:&quot;@timestamp&quot;,
            &quot;interval&quot;:&quot;30s&quot;,
            &quot;time_zone&quot;:&quot;America/Los_Angeles&quot;,
            &quot;min_doc_count&quot;:1
         },
         &quot;aggs&quot;:{
            &quot;1&quot;:{
               &quot;max&quot;:{
                  &quot;field&quot;:&quot;proc.mem.rss&quot;
               }
            }
         }
      }
   },
   &quot;highlight&quot;:{
      &quot;pre_tags&quot;:[
         &quot;@kibana-highlighted-field@&quot;
      ],
      &quot;post_tags&quot;:[
         &quot;@/kibana-highlighted-field@&quot;
      ],
      &quot;fields&quot;:{
         &quot;*&quot;:{

         }
      },
      &quot;require_field_match&quot;:false,
      &quot;fragment_size&quot;:2147483647
   },
   &quot;query&quot;:{
      &quot;bool&quot;:{
         &quot;must&quot;:[
            {
               &quot;query_string&quot;:{
                  &quot;query&quot;:&quot;*&quot;,
                  &quot;analyze_wildcard&quot;:true
               }
            },
            {
               &quot;match&quot;:{
                  &quot;proc.cmdline&quot;:{
                     &quot;query&quot;:&quot;/Users/tyler/.nvm/versions/node/v4.4.3/bin/node ./bin/../src/cli&quot;,
                     &quot;type&quot;:&quot;phrase&quot;
                  }
               }
            },
            {
               &quot;range&quot;:{
                  &quot;@timestamp&quot;:{
                     &quot;gte&quot;:1462389453519,
                     &quot;lte&quot;:1462390353519,
                     &quot;format&quot;:&quot;epoch_millis&quot;
                  }
               }
            }
         ],
         &quot;must_not&quot;:[

         ]
      }
   }
}"><pre class="notranslate"><span class="pl-kos">{</span>
   <span class="pl-s">"size"</span>:<span class="pl-c1">0</span><span class="pl-kos">,</span>
   <span class="pl-s">"aggs"</span>:<span class="pl-kos">{</span>
      <span class="pl-s">"2"</span>:<span class="pl-kos">{</span>
         <span class="pl-s">"date_histogram"</span>:<span class="pl-kos">{</span>
            <span class="pl-s">"field"</span>:<span class="pl-s">"@timestamp"</span><span class="pl-kos">,</span>
            <span class="pl-s">"interval"</span>:<span class="pl-s">"30s"</span><span class="pl-kos">,</span>
            <span class="pl-s">"time_zone"</span>:<span class="pl-s">"America/Los_Angeles"</span><span class="pl-kos">,</span>
            <span class="pl-s">"min_doc_count"</span>:<span class="pl-c1">1</span>
         <span class="pl-kos">}</span><span class="pl-kos">,</span>
         <span class="pl-s">"aggs"</span>:<span class="pl-kos">{</span>
            <span class="pl-s">"1"</span>:<span class="pl-kos">{</span>
               <span class="pl-s">"max"</span>:<span class="pl-kos">{</span>
                  <span class="pl-s">"field"</span>:<span class="pl-s">"proc.mem.rss"</span>
               <span class="pl-kos">}</span>
            <span class="pl-kos">}</span>
         <span class="pl-kos">}</span>
      <span class="pl-kos">}</span>
   <span class="pl-kos">}</span><span class="pl-kos">,</span>
   <span class="pl-s">"highlight"</span>:<span class="pl-kos">{</span>
      <span class="pl-s">"pre_tags"</span>:<span class="pl-kos">[</span>
         <span class="pl-s">"@kibana-highlighted-field@"</span>
      <span class="pl-kos">]</span><span class="pl-kos">,</span>
      <span class="pl-s">"post_tags"</span>:<span class="pl-kos">[</span>
         <span class="pl-s">"@/kibana-highlighted-field@"</span>
      <span class="pl-kos">]</span><span class="pl-kos">,</span>
      <span class="pl-s">"fields"</span>:<span class="pl-kos">{</span>
         <span class="pl-s">"*"</span>:<span class="pl-kos">{</span>

         <span class="pl-kos">}</span>
      <span class="pl-kos">}</span><span class="pl-kos">,</span>
      <span class="pl-s">"require_field_match"</span>:<span class="pl-c1">false</span><span class="pl-kos">,</span>
      <span class="pl-s">"fragment_size"</span>:<span class="pl-c1">2147483647</span>
   <span class="pl-kos">}</span><span class="pl-kos">,</span>
   <span class="pl-s">"query"</span>:<span class="pl-kos">{</span>
      <span class="pl-s">"bool"</span>:<span class="pl-kos">{</span>
         <span class="pl-s">"must"</span>:<span class="pl-kos">[</span>
            <span class="pl-kos">{</span>
               <span class="pl-s">"query_string"</span>:<span class="pl-kos">{</span>
                  <span class="pl-s">"query"</span>:<span class="pl-s">"*"</span><span class="pl-kos">,</span>
                  <span class="pl-s">"analyze_wildcard"</span>:<span class="pl-c1">true</span>
               <span class="pl-kos">}</span>
            <span class="pl-kos">}</span><span class="pl-kos">,</span>
            <span class="pl-kos">{</span>
               <span class="pl-s">"match"</span>:<span class="pl-kos">{</span>
                  <span class="pl-s">"proc.cmdline"</span>:<span class="pl-kos">{</span>
                     <span class="pl-s">"query"</span>:<span class="pl-s">"/Users/tyler/.nvm/versions/node/v4.4.3/bin/node ./bin/../src/cli"</span><span class="pl-kos">,</span>
                     <span class="pl-s">"type"</span>:<span class="pl-s">"phrase"</span>
                  <span class="pl-kos">}</span>
               <span class="pl-kos">}</span>
            <span class="pl-kos">}</span><span class="pl-kos">,</span>
            <span class="pl-kos">{</span>
               <span class="pl-s">"range"</span>:<span class="pl-kos">{</span>
                  <span class="pl-s">"@timestamp"</span>:<span class="pl-kos">{</span>
                     <span class="pl-s">"gte"</span>:<span class="pl-c1">1462389453519</span><span class="pl-kos">,</span>
                     <span class="pl-s">"lte"</span>:<span class="pl-c1">1462390353519</span><span class="pl-kos">,</span>
                     <span class="pl-s">"format"</span>:<span class="pl-s">"epoch_millis"</span>
                  <span class="pl-kos">}</span>
               <span class="pl-kos">}</span>
            <span class="pl-kos">}</span>
         <span class="pl-kos">]</span><span class="pl-kos">,</span>
         <span class="pl-s">"must_not"</span>:<span class="pl-kos">[</span>

         <span class="pl-kos">]</span>
      <span class="pl-kos">}</span>
   <span class="pl-kos">}</span>
<span class="pl-kos">}</span></pre></div>
</li>
<li>
<p dir="auto">Run for about 30 minutes.</p>
</li>
</ol>
<p dir="auto">I have 1027 documents, and the total size is 1.6MB.</p>
<p dir="auto"><strong>Provide logs (if relevant)</strong>:</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="CircuitBreakingException[[parent] Data too large, data for [&lt;http_request&gt;] would be larger than limit of [726571417/692.9mb]]
    at org.elasticsearch.indices.breaker.HierarchyCircuitBreakerService.checkParentLimit(HierarchyCircuitBreakerService.java:211)
    at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:128)
    at org.elasticsearch.http.HttpServer.dispatchRequest(HttpServer.java:109)
    at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:489)
    at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:65)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.messageReceived(HttpPipeliningHandler.java:85)
    at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:83)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)"><pre class="notranslate">CircuitBreakingException[[parent] Data too large, data <span class="pl-k">for</span> [<span class="pl-k">&lt;</span>http_request<span class="pl-k">&gt;</span>] would be larger than limit of [726571417/692.9mb]]
    at org.elasticsearch.indices.breaker.HierarchyCircuitBreakerService.checkParentLimit(HierarchyCircuitBreakerService.java:211)
    at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:128)
    at org.elasticsearch.http.HttpServer.dispatchRequest(HttpServer.java:109)
    at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:489)
    at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:65)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline<span class="pl-smi">$DefaultChannelHandlerContext</span>.sendUpstream(DefaultChannelPipeline.java:791)
    at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.messageReceived(HttpPipeliningHandler.java:85)
    at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline<span class="pl-smi">$DefaultChannelHandlerContext</span>.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline<span class="pl-smi">$DefaultChannelHandlerContext</span>.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline<span class="pl-smi">$DefaultChannelHandlerContext</span>.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline<span class="pl-smi">$DefaultChannelHandlerContext</span>.sendUpstream(DefaultChannelPipeline.java:791)
    at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:83)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.jboss.netty.util.internal.DeadLockProofWorker<span class="pl-smi">$1</span>.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor<span class="pl-smi">$Worker</span>.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)</pre></div>
<p dir="auto">Eventually, all requests to ES will fail with this exception including _stats.</p>