<p dir="auto"><strong>Elasticsearch version</strong>: 2.2.0</p>
<p dir="auto"><strong>JVM version</strong>: 1.8.0_65</p>
<p dir="auto"><strong>OS version</strong>: Ubuntu 15.04</p>
<p dir="auto"><strong>Description of the problem including expected versus actual behavior</strong>: When hitting any Cat APIs that deal with the list of nodes, the ordering of the nodes is non-deterministic, so depending on which node handles the request, the ordering will be different.</p>
<p dir="auto">The following APIs are affected:</p>
<ul dir="auto">
<li>_cat/nodes</li>
<li>_cat/allocation</li>
<li>_cat/nodeattrs</li>
</ul>
<p dir="auto">I'm inclined to raise this as a bug, since the ordering used to be deterministic in 1.7.</p>
<p dir="auto"><strong>Steps to reproduce</strong>:</p>
<ol dir="auto">
<li>Setup a cluster with at least two nodes</li>
<li>Hit _cat/nodes and note the ordering of entries</li>
<li>Hit _cat/nodes on a different node in the cluster and note the ordering is different</li>
</ol>
<p dir="auto">Opening a new issue as requested by <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/brwe/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/brwe">@brwe</a> in the comments to <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="51073197" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/8788" data-hovercard-type="issue" data-hovercard-url="/elastic/elasticsearch/issues/8788/hovercard" href="https://github.com/elastic/elasticsearch/issues/8788">#8788</a>.</p>
<p dir="auto">We recently saw an unexpectedly high count on an index we're feeding. It's the second time we've seen this; the first time was only off by 1, but this time it was off by over 16,000 in an index that should contain less than 300k docs. Since the dataset is so small I pulled down everything (<code class="notranslate">_search?pretty=true&amp;fields=&amp;size=400000</code>), scraped out the ids and ran them through <code class="notranslate">uniq</code>, which revealed a ton of duplicates.</p>
<p dir="auto">Notes/observations:</p>
<ul dir="auto">
<li>We're using 1.3.4, doing bulk indexing with the Java API's <code class="notranslate">TransportClient</code> and <code class="notranslate">BulkProcessor</code>.</li>
<li>We're <strong>not</strong> using autogenerated ids, so it doesn't look as if the fix for <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="51073197" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/8788" data-hovercard-type="issue" data-hovercard-url="/elastic/elasticsearch/issues/8788/hovercard" href="https://github.com/elastic/elasticsearch/issues/8788">#8788</a> is likely to help us.</li>
<li>No id appeared more than twice.</li>
<li>The index has not been migrated from an earlier version. In fact, the index is deleted and recreated irregularly but fairly frequently, from every few days to every couple of weeks.</li>
<li>Around the time the duplicates appeared, we saw problems in other (non-Elastic) parts of the system. I can't see any way that they could directly cause the duplication, but it's possible that network issues were the common cause of both.</li>
</ul>
<p dir="auto">Running</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl -s 'http://HOST:9200/std_physicalasset_alpha_a/_search?pretty&amp;q=_id:77309419847&amp;explain&amp;fields=_source,_routing' -d '{&quot;version&quot;:true}'"><pre class="notranslate"><code class="notranslate">curl -s 'http://HOST:9200/std_physicalasset_alpha_a/_search?pretty&amp;q=_id:77309419847&amp;explain&amp;fields=_source,_routing' -d '{"version":true}'
</code></pre></div>
<p dir="auto">produced</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
  &quot;took&quot; : 247,
  &quot;timed_out&quot; : false,
  &quot;_shards&quot; : {
    &quot;total&quot; : 5,
    &quot;successful&quot; : 5,
    &quot;failed&quot; : 0
  },
  &quot;hits&quot; : {
    &quot;total&quot; : 2,
    &quot;max_score&quot; : 1.0,
    &quot;hits&quot; : [ {
      &quot;_shard&quot; : 3,
      &quot;_node&quot; : &quot;HRLe1R1iTzqzZ44ocSb4Sw&quot;,
      &quot;_index&quot; : &quot;std_physicalasset_alpha_a&quot;,
      &quot;_type&quot; : &quot;other&quot;,
      &quot;_id&quot; : &quot;77309419847&quot;,
      &quot;_version&quot; : 2,
      &quot;_score&quot; : 1.0,
      &quot;_source&quot;:{&quot;DEBUGRUN&quot;:&quot;2015-04-03T20:48:55.327Z&quot;},
      &quot;_explanation&quot; : {
        &quot;value&quot; : 1.0,
        &quot;description&quot; : &quot;ConstantScore(_uid:_default_#77309419847 _uid:generic#77309419847 _uid:other#77309419847 _uid:plant#77309419847 _uid:vessel#77309419847), product of:&quot;,
        &quot;details&quot; : [ {
          &quot;value&quot; : 1.0,
          &quot;description&quot; : &quot;boost&quot;
        }, {
          &quot;value&quot; : 1.0,
          &quot;description&quot; : &quot;queryNorm&quot;
        } ]
      }
    }, {
      &quot;_shard&quot; : 3,
      &quot;_node&quot; : &quot;HRLe1R1iTzqzZ44ocSb4Sw&quot;,
      &quot;_index&quot; : &quot;std_physicalasset_alpha_a&quot;,
      &quot;_type&quot; : &quot;vessel&quot;,
      &quot;_id&quot; : &quot;77309419847&quot;,
      &quot;_version&quot; : 456,
      &quot;_score&quot; : 1.0,
      &quot;_source&quot;:{&quot;DEBUGRUN&quot;:&quot;2015-04-08T16:12:52.903Z&quot;},
      &quot;_explanation&quot; : {
        &quot;value&quot; : 1.0,
        &quot;description&quot; : &quot;ConstantScore(_uid:_default_#77309419847 _uid:generic#77309419847 _uid:other#77309419847 _uid:plant#77309419847 _uid:vessel#77309419847), product of:&quot;,
        &quot;details&quot; : [ {
          &quot;value&quot; : 1.0,
          &quot;description&quot; : &quot;boost&quot;
        }, {
          &quot;value&quot; : 1.0,
          &quot;description&quot; : &quot;queryNorm&quot;
        } ]
      }
    } ]
  }
}"><pre class="notranslate"><code class="notranslate">{
  "took" : 247,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 2,
    "max_score" : 1.0,
    "hits" : [ {
      "_shard" : 3,
      "_node" : "HRLe1R1iTzqzZ44ocSb4Sw",
      "_index" : "std_physicalasset_alpha_a",
      "_type" : "other",
      "_id" : "77309419847",
      "_version" : 2,
      "_score" : 1.0,
      "_source":{"DEBUGRUN":"2015-04-03T20:48:55.327Z"},
      "_explanation" : {
        "value" : 1.0,
        "description" : "ConstantScore(_uid:_default_#77309419847 _uid:generic#77309419847 _uid:other#77309419847 _uid:plant#77309419847 _uid:vessel#77309419847), product of:",
        "details" : [ {
          "value" : 1.0,
          "description" : "boost"
        }, {
          "value" : 1.0,
          "description" : "queryNorm"
        } ]
      }
    }, {
      "_shard" : 3,
      "_node" : "HRLe1R1iTzqzZ44ocSb4Sw",
      "_index" : "std_physicalasset_alpha_a",
      "_type" : "vessel",
      "_id" : "77309419847",
      "_version" : 456,
      "_score" : 1.0,
      "_source":{"DEBUGRUN":"2015-04-08T16:12:52.903Z"},
      "_explanation" : {
        "value" : 1.0,
        "description" : "ConstantScore(_uid:_default_#77309419847 _uid:generic#77309419847 _uid:other#77309419847 _uid:plant#77309419847 _uid:vessel#77309419847), product of:",
        "details" : [ {
          "value" : 1.0,
          "description" : "boost"
        }, {
          "value" : 1.0,
          "description" : "queryNorm"
        } ]
      }
    } ]
  }
}
</code></pre></div>
<p dir="auto">I've removed most of <code class="notranslate">_source</code> as verbose and not relevant, but <code class="notranslate">DEBUGBATCH</code> is interesting; it's a manually-set human-readable equivalent of <code class="notranslate">_timestamp</code>, and shows that the two documents with this id were loaded almost 5 days apart. The second has been updated again since I captured that, although the feeder process has now been stopped.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl -s 'http://HOST:9200/std_physicalasset_alpha_a/_segments?pretty'"><pre class="notranslate"><code class="notranslate">curl -s 'http://HOST:9200/std_physicalasset_alpha_a/_segments?pretty'
</code></pre></div>
<p dir="auto">produces around 100k of JSON, which seems a bit much to post in full. (That may be telling in itself; the full response contained a suspiciously-round 255 segment objects.) The following is just the portion for the shard and (non-primary) node containing the sample duplicate id above:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
  &quot;_shards&quot; : {
    &quot;total&quot; : 10,
    &quot;successful&quot; : 10,
    &quot;failed&quot; : 0
  },
  &quot;indices&quot; : {
    &quot;std_physicalasset_alpha_a&quot; : {
      &quot;shards&quot; : {
====================
# Shards 0-2 omitted
====================
        &quot;3&quot; : [ {
          &quot;routing&quot; : {
            &quot;state&quot; : &quot;STARTED&quot;,
            &quot;primary&quot; : false,
            &quot;node&quot; : &quot;HRLe1R1iTzqzZ44ocSb4Sw&quot;
          },
          &quot;num_committed_segments&quot; : 26,
          &quot;num_search_segments&quot; : 16,
          &quot;segments&quot; : {
            &quot;_2a5z&quot; : {
              &quot;generation&quot; : 106487,
              &quot;num_docs&quot; : 43857,
              &quot;deleted_docs&quot; : 15215,
              &quot;size_in_bytes&quot; : 341737818,
              &quot;memory_in_bytes&quot; : 2035824,
              &quot;committed&quot; : true,
              &quot;search&quot; : true,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : false
            },
            &quot;_2a76&quot; : {
              &quot;generation&quot; : 106530,
              &quot;num_docs&quot; : 1197,
              &quot;deleted_docs&quot; : 31,
              &quot;size_in_bytes&quot; : 9592426,
              &quot;memory_in_bytes&quot; : 418896,
              &quot;committed&quot; : true,
              &quot;search&quot; : true,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : false
            },
            &quot;_2a7t&quot; : {
              &quot;generation&quot; : 106553,
              &quot;num_docs&quot; : 1382,
              &quot;deleted_docs&quot; : 35,
              &quot;size_in_bytes&quot; : 11067533,
              &quot;memory_in_bytes&quot; : 434208,
              &quot;committed&quot; : true,
              &quot;search&quot; : true,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : false
            },
            &quot;_2a82&quot; : {
              &quot;generation&quot; : 106562,
              &quot;num_docs&quot; : 1629,
              &quot;deleted_docs&quot; : 48,
              &quot;size_in_bytes&quot; : 12533722,
              &quot;memory_in_bytes&quot; : 443952,
              &quot;committed&quot; : true,
              &quot;search&quot; : true,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : false
            },
            &quot;_2a8n&quot; : {
              &quot;generation&quot; : 106583,
              &quot;num_docs&quot; : 2070,
              &quot;deleted_docs&quot; : 61,
              &quot;size_in_bytes&quot; : 15764916,
              &quot;memory_in_bytes&quot; : 447088,
              &quot;committed&quot; : true,
              &quot;search&quot; : true,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : false
            },
            &quot;_2a9e&quot; : {
              &quot;generation&quot; : 106610,
              &quot;num_docs&quot; : 1480,
              &quot;deleted_docs&quot; : 28,
              &quot;size_in_bytes&quot; : 11471539,
              &quot;memory_in_bytes&quot; : 435432,
              &quot;committed&quot; : true,
              &quot;search&quot; : true,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : false
            },
            &quot;_2a9m&quot; : {
              &quot;generation&quot; : 106618,
              &quot;num_docs&quot; : 151,
              &quot;deleted_docs&quot; : 3,
              &quot;size_in_bytes&quot; : 1529856,
              &quot;memory_in_bytes&quot; : 0,
              &quot;committed&quot; : true,
              &quot;search&quot; : false,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : true
            },
            &quot;_2a9n&quot; : {
              &quot;generation&quot; : 106619,
              &quot;num_docs&quot; : 6528,
              &quot;deleted_docs&quot; : 145,
              &quot;size_in_bytes&quot; : 45294050,
              &quot;memory_in_bytes&quot; : 632040,
              &quot;committed&quot; : true,
              &quot;search&quot; : true,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : false
            },
            &quot;_2a9q&quot; : {
              &quot;generation&quot; : 106622,
              &quot;num_docs&quot; : 186,
              &quot;deleted_docs&quot; : 4,
              &quot;size_in_bytes&quot; : 1872124,
              &quot;memory_in_bytes&quot; : 0,
              &quot;committed&quot; : true,
              &quot;search&quot; : false,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : true
            },
            &quot;_2a9t&quot; : {
              &quot;generation&quot; : 106625,
              &quot;num_docs&quot; : 220,
              &quot;deleted_docs&quot; : 9,
              &quot;size_in_bytes&quot; : 2276799,
              &quot;memory_in_bytes&quot; : 0,
              &quot;committed&quot; : true,
              &quot;search&quot; : false,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : true
            },
            &quot;_2a9w&quot; : {
              &quot;generation&quot; : 106628,
              &quot;num_docs&quot; : 654,
              &quot;deleted_docs&quot; : 18,
              &quot;size_in_bytes&quot; : 5614691,
              &quot;memory_in_bytes&quot; : 368520,
              &quot;committed&quot; : true,
              &quot;search&quot; : true,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : false
            },
            &quot;_2aaf&quot; : {
              &quot;generation&quot; : 106647,
              &quot;num_docs&quot; : 101,
              &quot;deleted_docs&quot; : 80,
              &quot;size_in_bytes&quot; : 1858120,
              &quot;memory_in_bytes&quot; : 0,
              &quot;committed&quot; : true,
              &quot;search&quot; : false,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : true
            },
            &quot;_2aaj&quot; : {
              &quot;generation&quot; : 106651,
              &quot;num_docs&quot; : 499,
              &quot;deleted_docs&quot; : 269,
              &quot;size_in_bytes&quot; : 6309352,
              &quot;memory_in_bytes&quot; : 0,
              &quot;committed&quot; : true,
              &quot;search&quot; : false,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : false
            },
            &quot;_2aaq&quot; : {
              &quot;generation&quot; : 106658,
              &quot;num_docs&quot; : 609,
              &quot;deleted_docs&quot; : 207,
              &quot;size_in_bytes&quot; : 6785525,
              &quot;memory_in_bytes&quot; : 0,
              &quot;committed&quot; : true,
              &quot;search&quot; : false,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : false
            },
            &quot;_2aas&quot; : {
              &quot;generation&quot; : 106660,
              &quot;num_docs&quot; : 663,
              &quot;deleted_docs&quot; : 225,
              &quot;size_in_bytes&quot; : 7485334,
              &quot;memory_in_bytes&quot; : 0,
              &quot;committed&quot; : true,
              &quot;search&quot; : false,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : false
            },
            &quot;_2aat&quot; : {
              &quot;generation&quot; : 106661,
              &quot;num_docs&quot; : 106,
              &quot;deleted_docs&quot; : 0,
              &quot;size_in_bytes&quot; : 1942039,
              &quot;memory_in_bytes&quot; : 0,
              &quot;committed&quot; : true,
              &quot;search&quot; : false,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : true
            },
            &quot;_2aau&quot; : {
              &quot;generation&quot; : 106662,
              &quot;num_docs&quot; : 123,
              &quot;deleted_docs&quot; : 0,
              &quot;size_in_bytes&quot; : 2226032,
              &quot;memory_in_bytes&quot; : 0,
              &quot;committed&quot; : true,
              &quot;search&quot; : false,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : true
            },
            &quot;_2aaw&quot; : {
              &quot;generation&quot; : 106664,
              &quot;num_docs&quot; : 75,
              &quot;deleted_docs&quot; : 0,
              &quot;size_in_bytes&quot; : 1489187,
              &quot;memory_in_bytes&quot; : 0,
              &quot;committed&quot; : true,
              &quot;search&quot; : false,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : true
            },
            &quot;_2aaz&quot; : {
              &quot;generation&quot; : 106667,
              &quot;num_docs&quot; : 93,
              &quot;deleted_docs&quot; : 0,
              &quot;size_in_bytes&quot; : 1683929,
              &quot;memory_in_bytes&quot; : 0,
              &quot;committed&quot; : true,
              &quot;search&quot; : false,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : true
            },
            &quot;_2ab0&quot; : {
              &quot;generation&quot; : 106668,
              &quot;num_docs&quot; : 74,
              &quot;deleted_docs&quot; : 0,
              &quot;size_in_bytes&quot; : 1368605,
              &quot;memory_in_bytes&quot; : 0,
              &quot;committed&quot; : true,
              &quot;search&quot; : false,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : true
            },
            &quot;_2ab5&quot; : {
              &quot;generation&quot; : 106673,
              &quot;num_docs&quot; : 68,
              &quot;deleted_docs&quot; : 0,
              &quot;size_in_bytes&quot; : 1346587,
              &quot;memory_in_bytes&quot; : 0,
              &quot;committed&quot; : true,
              &quot;search&quot; : false,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : true
            },
            &quot;_2ab8&quot; : {
              &quot;generation&quot; : 106676,
              &quot;num_docs&quot; : 31,
              &quot;deleted_docs&quot; : 0,
              &quot;size_in_bytes&quot; : 690748,
              &quot;memory_in_bytes&quot; : 0,
              &quot;committed&quot; : true,
              &quot;search&quot; : false,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : true
            },
            &quot;_2ab9&quot; : {
              &quot;generation&quot; : 106677,
              &quot;num_docs&quot; : 54,
              &quot;deleted_docs&quot; : 0,
              &quot;size_in_bytes&quot; : 1086051,
              &quot;memory_in_bytes&quot; : 0,
              &quot;committed&quot; : true,
              &quot;search&quot; : false,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : true
            },
            &quot;_2aba&quot; : {
              &quot;generation&quot; : 106678,
              &quot;num_docs&quot; : 121,
              &quot;deleted_docs&quot; : 0,
              &quot;size_in_bytes&quot; : 2119135,
              &quot;memory_in_bytes&quot; : 0,
              &quot;committed&quot; : true,
              &quot;search&quot; : false,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : true
            },
            &quot;_2abb&quot; : {
              &quot;generation&quot; : 106679,
              &quot;num_docs&quot; : 473,
              &quot;deleted_docs&quot; : 0,
              &quot;size_in_bytes&quot; : 6822800,
              &quot;memory_in_bytes&quot; : 0,
              &quot;committed&quot; : true,
              &quot;search&quot; : false,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : false
            },
            &quot;_2abc&quot; : {
              &quot;generation&quot; : 106680,
              &quot;num_docs&quot; : 714,
              &quot;deleted_docs&quot; : 388,
              &quot;size_in_bytes&quot; : 12332118,
              &quot;memory_in_bytes&quot; : 417680,
              &quot;committed&quot; : false,
              &quot;search&quot; : true,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : false
            },
            &quot;_2abd&quot; : {
              &quot;generation&quot; : 106681,
              &quot;num_docs&quot; : 1,
              &quot;deleted_docs&quot; : 0,
              &quot;size_in_bytes&quot; : 49613,
              &quot;memory_in_bytes&quot; : 0,
              &quot;committed&quot; : true,
              &quot;search&quot; : false,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : true
            },
            &quot;_2acb&quot; : {
              &quot;generation&quot; : 106715,
              &quot;num_docs&quot; : 17,
              &quot;deleted_docs&quot; : 0,
              &quot;size_in_bytes&quot; : 398023,
              &quot;memory_in_bytes&quot; : 195496,
              &quot;committed&quot; : false,
              &quot;search&quot; : true,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : true
            },
            &quot;_2acc&quot; : {
              &quot;generation&quot; : 106716,
              &quot;num_docs&quot; : 36,
              &quot;deleted_docs&quot; : 0,
              &quot;size_in_bytes&quot; : 756378,
              &quot;memory_in_bytes&quot; : 228552,
              &quot;committed&quot; : false,
              &quot;search&quot; : true,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : true
            },
            &quot;_2acd&quot; : {
              &quot;generation&quot; : 106717,
              &quot;num_docs&quot; : 13,
              &quot;deleted_docs&quot; : 0,
              &quot;size_in_bytes&quot; : 316552,
              &quot;memory_in_bytes&quot; : 196488,
              &quot;committed&quot; : false,
              &quot;search&quot; : true,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : true
            },
            &quot;_2ace&quot; : {
              &quot;generation&quot; : 106718,
              &quot;num_docs&quot; : 17,
              &quot;deleted_docs&quot; : 0,
              &quot;size_in_bytes&quot; : 354373,
              &quot;memory_in_bytes&quot; : 189920,
              &quot;committed&quot; : false,
              &quot;search&quot; : true,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : true
            },
            &quot;_2acf&quot; : {
              &quot;generation&quot; : 106719,
              &quot;num_docs&quot; : 2118,
              &quot;deleted_docs&quot; : 96,
              &quot;size_in_bytes&quot; : 23788950,
              &quot;memory_in_bytes&quot; : 490448,
              &quot;committed&quot; : false,
              &quot;search&quot; : true,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : false
            },
            &quot;_2acg&quot; : {
              &quot;generation&quot; : 106720,
              &quot;num_docs&quot; : 24,
              &quot;deleted_docs&quot; : 0,
              &quot;size_in_bytes&quot; : 516210,
              &quot;memory_in_bytes&quot; : 219592,
              &quot;committed&quot; : false,
              &quot;search&quot; : true,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : true
            },
            &quot;_2ach&quot; : {
              &quot;generation&quot; : 106721,
              &quot;num_docs&quot; : 67,
              &quot;deleted_docs&quot; : 0,
              &quot;size_in_bytes&quot; : 1265169,
              &quot;memory_in_bytes&quot; : 236688,
              &quot;committed&quot; : false,
              &quot;search&quot; : true,
              &quot;version&quot; : &quot;4.9&quot;,
              &quot;compound&quot; : true
            }
          }
        }, {
          &quot;routing&quot; : {
            &quot;state&quot; : &quot;STARTED&quot;,
            &quot;primary&quot; : true,
            &quot;node&quot; : &quot;oyFoYqV1SHyLoJmOzRQrDQ&quot;
          },
          &quot;num_committed_segments&quot; : 14,
          &quot;num_search_segments&quot; : 13,
          &quot;segments&quot; : {
==========================
20 segment objects omitted
==========================
        } ],
===============
Shard 4 omitted
===============
      }
    }
  }
}"><pre class="notranslate"><code class="notranslate">{
  "_shards" : {
    "total" : 10,
    "successful" : 10,
    "failed" : 0
  },
  "indices" : {
    "std_physicalasset_alpha_a" : {
      "shards" : {
====================
# Shards 0-2 omitted
====================
        "3" : [ {
          "routing" : {
            "state" : "STARTED",
            "primary" : false,
            "node" : "HRLe1R1iTzqzZ44ocSb4Sw"
          },
          "num_committed_segments" : 26,
          "num_search_segments" : 16,
          "segments" : {
            "_2a5z" : {
              "generation" : 106487,
              "num_docs" : 43857,
              "deleted_docs" : 15215,
              "size_in_bytes" : 341737818,
              "memory_in_bytes" : 2035824,
              "committed" : true,
              "search" : true,
              "version" : "4.9",
              "compound" : false
            },
            "_2a76" : {
              "generation" : 106530,
              "num_docs" : 1197,
              "deleted_docs" : 31,
              "size_in_bytes" : 9592426,
              "memory_in_bytes" : 418896,
              "committed" : true,
              "search" : true,
              "version" : "4.9",
              "compound" : false
            },
            "_2a7t" : {
              "generation" : 106553,
              "num_docs" : 1382,
              "deleted_docs" : 35,
              "size_in_bytes" : 11067533,
              "memory_in_bytes" : 434208,
              "committed" : true,
              "search" : true,
              "version" : "4.9",
              "compound" : false
            },
            "_2a82" : {
              "generation" : 106562,
              "num_docs" : 1629,
              "deleted_docs" : 48,
              "size_in_bytes" : 12533722,
              "memory_in_bytes" : 443952,
              "committed" : true,
              "search" : true,
              "version" : "4.9",
              "compound" : false
            },
            "_2a8n" : {
              "generation" : 106583,
              "num_docs" : 2070,
              "deleted_docs" : 61,
              "size_in_bytes" : 15764916,
              "memory_in_bytes" : 447088,
              "committed" : true,
              "search" : true,
              "version" : "4.9",
              "compound" : false
            },
            "_2a9e" : {
              "generation" : 106610,
              "num_docs" : 1480,
              "deleted_docs" : 28,
              "size_in_bytes" : 11471539,
              "memory_in_bytes" : 435432,
              "committed" : true,
              "search" : true,
              "version" : "4.9",
              "compound" : false
            },
            "_2a9m" : {
              "generation" : 106618,
              "num_docs" : 151,
              "deleted_docs" : 3,
              "size_in_bytes" : 1529856,
              "memory_in_bytes" : 0,
              "committed" : true,
              "search" : false,
              "version" : "4.9",
              "compound" : true
            },
            "_2a9n" : {
              "generation" : 106619,
              "num_docs" : 6528,
              "deleted_docs" : 145,
              "size_in_bytes" : 45294050,
              "memory_in_bytes" : 632040,
              "committed" : true,
              "search" : true,
              "version" : "4.9",
              "compound" : false
            },
            "_2a9q" : {
              "generation" : 106622,
              "num_docs" : 186,
              "deleted_docs" : 4,
              "size_in_bytes" : 1872124,
              "memory_in_bytes" : 0,
              "committed" : true,
              "search" : false,
              "version" : "4.9",
              "compound" : true
            },
            "_2a9t" : {
              "generation" : 106625,
              "num_docs" : 220,
              "deleted_docs" : 9,
              "size_in_bytes" : 2276799,
              "memory_in_bytes" : 0,
              "committed" : true,
              "search" : false,
              "version" : "4.9",
              "compound" : true
            },
            "_2a9w" : {
              "generation" : 106628,
              "num_docs" : 654,
              "deleted_docs" : 18,
              "size_in_bytes" : 5614691,
              "memory_in_bytes" : 368520,
              "committed" : true,
              "search" : true,
              "version" : "4.9",
              "compound" : false
            },
            "_2aaf" : {
              "generation" : 106647,
              "num_docs" : 101,
              "deleted_docs" : 80,
              "size_in_bytes" : 1858120,
              "memory_in_bytes" : 0,
              "committed" : true,
              "search" : false,
              "version" : "4.9",
              "compound" : true
            },
            "_2aaj" : {
              "generation" : 106651,
              "num_docs" : 499,
              "deleted_docs" : 269,
              "size_in_bytes" : 6309352,
              "memory_in_bytes" : 0,
              "committed" : true,
              "search" : false,
              "version" : "4.9",
              "compound" : false
            },
            "_2aaq" : {
              "generation" : 106658,
              "num_docs" : 609,
              "deleted_docs" : 207,
              "size_in_bytes" : 6785525,
              "memory_in_bytes" : 0,
              "committed" : true,
              "search" : false,
              "version" : "4.9",
              "compound" : false
            },
            "_2aas" : {
              "generation" : 106660,
              "num_docs" : 663,
              "deleted_docs" : 225,
              "size_in_bytes" : 7485334,
              "memory_in_bytes" : 0,
              "committed" : true,
              "search" : false,
              "version" : "4.9",
              "compound" : false
            },
            "_2aat" : {
              "generation" : 106661,
              "num_docs" : 106,
              "deleted_docs" : 0,
              "size_in_bytes" : 1942039,
              "memory_in_bytes" : 0,
              "committed" : true,
              "search" : false,
              "version" : "4.9",
              "compound" : true
            },
            "_2aau" : {
              "generation" : 106662,
              "num_docs" : 123,
              "deleted_docs" : 0,
              "size_in_bytes" : 2226032,
              "memory_in_bytes" : 0,
              "committed" : true,
              "search" : false,
              "version" : "4.9",
              "compound" : true
            },
            "_2aaw" : {
              "generation" : 106664,
              "num_docs" : 75,
              "deleted_docs" : 0,
              "size_in_bytes" : 1489187,
              "memory_in_bytes" : 0,
              "committed" : true,
              "search" : false,
              "version" : "4.9",
              "compound" : true
            },
            "_2aaz" : {
              "generation" : 106667,
              "num_docs" : 93,
              "deleted_docs" : 0,
              "size_in_bytes" : 1683929,
              "memory_in_bytes" : 0,
              "committed" : true,
              "search" : false,
              "version" : "4.9",
              "compound" : true
            },
            "_2ab0" : {
              "generation" : 106668,
              "num_docs" : 74,
              "deleted_docs" : 0,
              "size_in_bytes" : 1368605,
              "memory_in_bytes" : 0,
              "committed" : true,
              "search" : false,
              "version" : "4.9",
              "compound" : true
            },
            "_2ab5" : {
              "generation" : 106673,
              "num_docs" : 68,
              "deleted_docs" : 0,
              "size_in_bytes" : 1346587,
              "memory_in_bytes" : 0,
              "committed" : true,
              "search" : false,
              "version" : "4.9",
              "compound" : true
            },
            "_2ab8" : {
              "generation" : 106676,
              "num_docs" : 31,
              "deleted_docs" : 0,
              "size_in_bytes" : 690748,
              "memory_in_bytes" : 0,
              "committed" : true,
              "search" : false,
              "version" : "4.9",
              "compound" : true
            },
            "_2ab9" : {
              "generation" : 106677,
              "num_docs" : 54,
              "deleted_docs" : 0,
              "size_in_bytes" : 1086051,
              "memory_in_bytes" : 0,
              "committed" : true,
              "search" : false,
              "version" : "4.9",
              "compound" : true
            },
            "_2aba" : {
              "generation" : 106678,
              "num_docs" : 121,
              "deleted_docs" : 0,
              "size_in_bytes" : 2119135,
              "memory_in_bytes" : 0,
              "committed" : true,
              "search" : false,
              "version" : "4.9",
              "compound" : true
            },
            "_2abb" : {
              "generation" : 106679,
              "num_docs" : 473,
              "deleted_docs" : 0,
              "size_in_bytes" : 6822800,
              "memory_in_bytes" : 0,
              "committed" : true,
              "search" : false,
              "version" : "4.9",
              "compound" : false
            },
            "_2abc" : {
              "generation" : 106680,
              "num_docs" : 714,
              "deleted_docs" : 388,
              "size_in_bytes" : 12332118,
              "memory_in_bytes" : 417680,
              "committed" : false,
              "search" : true,
              "version" : "4.9",
              "compound" : false
            },
            "_2abd" : {
              "generation" : 106681,
              "num_docs" : 1,
              "deleted_docs" : 0,
              "size_in_bytes" : 49613,
              "memory_in_bytes" : 0,
              "committed" : true,
              "search" : false,
              "version" : "4.9",
              "compound" : true
            },
            "_2acb" : {
              "generation" : 106715,
              "num_docs" : 17,
              "deleted_docs" : 0,
              "size_in_bytes" : 398023,
              "memory_in_bytes" : 195496,
              "committed" : false,
              "search" : true,
              "version" : "4.9",
              "compound" : true
            },
            "_2acc" : {
              "generation" : 106716,
              "num_docs" : 36,
              "deleted_docs" : 0,
              "size_in_bytes" : 756378,
              "memory_in_bytes" : 228552,
              "committed" : false,
              "search" : true,
              "version" : "4.9",
              "compound" : true
            },
            "_2acd" : {
              "generation" : 106717,
              "num_docs" : 13,
              "deleted_docs" : 0,
              "size_in_bytes" : 316552,
              "memory_in_bytes" : 196488,
              "committed" : false,
              "search" : true,
              "version" : "4.9",
              "compound" : true
            },
            "_2ace" : {
              "generation" : 106718,
              "num_docs" : 17,
              "deleted_docs" : 0,
              "size_in_bytes" : 354373,
              "memory_in_bytes" : 189920,
              "committed" : false,
              "search" : true,
              "version" : "4.9",
              "compound" : true
            },
            "_2acf" : {
              "generation" : 106719,
              "num_docs" : 2118,
              "deleted_docs" : 96,
              "size_in_bytes" : 23788950,
              "memory_in_bytes" : 490448,
              "committed" : false,
              "search" : true,
              "version" : "4.9",
              "compound" : false
            },
            "_2acg" : {
              "generation" : 106720,
              "num_docs" : 24,
              "deleted_docs" : 0,
              "size_in_bytes" : 516210,
              "memory_in_bytes" : 219592,
              "committed" : false,
              "search" : true,
              "version" : "4.9",
              "compound" : true
            },
            "_2ach" : {
              "generation" : 106721,
              "num_docs" : 67,
              "deleted_docs" : 0,
              "size_in_bytes" : 1265169,
              "memory_in_bytes" : 236688,
              "committed" : false,
              "search" : true,
              "version" : "4.9",
              "compound" : true
            }
          }
        }, {
          "routing" : {
            "state" : "STARTED",
            "primary" : true,
            "node" : "oyFoYqV1SHyLoJmOzRQrDQ"
          },
          "num_committed_segments" : 14,
          "num_search_segments" : 13,
          "segments" : {
==========================
20 segment objects omitted
==========================
        } ],
===============
Shard 4 omitted
===============
      }
    }
  }
}
</code></pre></div>
<p dir="auto">It's highly unlikely that I'll be able to make the ES server logs available in full, but I can grep them if there's anything specific that might help. I can't find any mention of the <code class="notranslate">std_physicalasset_alpha_a</code> index in them except for a bunch of entries of the form</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[2015-04-05 22:20:58,598][INFO ][index.engine.internal    ] [c115knppaes01-01] [std_physicalasset_alpha_a][2] now throttling indexing: numMergesInFlight=6, maxNumMerges=5
[2015-04-05 22:20:59,216][INFO ][index.engine.internal    ] [c115knppaes01-01] [std_physicalasset_alpha_a][2] stop throttling indexing: numMergesInFlight=4, maxNumMerges=5"><pre class="notranslate"><code class="notranslate">[2015-04-05 22:20:58,598][INFO ][index.engine.internal    ] [c115knppaes01-01] [std_physicalasset_alpha_a][2] now throttling indexing: numMergesInFlight=6, maxNumMerges=5
[2015-04-05 22:20:59,216][INFO ][index.engine.internal    ] [c115knppaes01-01] [std_physicalasset_alpha_a][2] stop throttling indexing: numMergesInFlight=4, maxNumMerges=5
</code></pre></div>