<p dir="auto">We have seen this issue in a production environment (Running Elastic Search 0.19.12) on more than one occasion - the most recent occasion being after one of the servers in the cluster was unexpectedly rebooted by Amazon.</p>
<p dir="auto">We have been able to reproduce this locally using multiple Vagrant boxes - although not every time (2/3 attempts will reproduce this)</p>
<p dir="auto">This bug seems very similar / the same as <a href="https://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/percolator/elasticsearch/VZjslNupBNY/Q3VFfXDqt6sJ" rel="nofollow">https://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/percolator/elasticsearch/VZjslNupBNY/Q3VFfXDqt6sJ</a></p>
<h2 dir="auto">Steps to reproduce:</h2>
<ul dir="auto">
<li>Have a cluster of Elastic Search servers</li>
<li>Index some Percolator queries</li>
<li>Test that each node returns the same response to a percolation request</li>
<li>Forcefully reboot one of the servers (e.g. sudo reboot -f)</li>
<li>Wait until the rebooted node rejoins the cluster</li>
<li>Test each node returns the same response</li>
</ul>
<h2 dir="auto">Expected Result:</h2>
<ul dir="auto">
<li>Every node returns the same (correct) response</li>
</ul>
<h2 dir="auto">Actual Result:</h2>
<ul dir="auto">
<li>The restarted node returns no matched percolation queries, despite the cluster being in the healthy (green) state and the expected number of docs showing in the _percolator index.</li>
<li>Sometimes the correct result is returned by the restarted node for a short time, and <em>then</em>, after a few seconds, starts returning no matches.</li>
<li>Restarting Elastic Search on the node seems to fix this.</li>
<li>Nothing unusual appears in the log files.</li>
</ul>
<p dir="auto">There are two different definitions of the <code class="notranslate">timeout</code> parameter in <a href="https://github.com/elastic/elasticsearch/blob/5.0/rest-api-spec/src/main/resources/rest-api-spec/api/update_by_query.json"><code class="notranslate">update_by_query.json</code></a>:</p>
<div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="...
&quot;timeout&quot;: {
  &quot;type&quot; : &quot;time&quot;,
  &quot;description&quot; : &quot;Explicit operation timeout&quot;
},
...
&quot;timeout&quot;: {
  &quot;type&quot; : &quot;time&quot;,
  &quot;default&quot;: &quot;1m&quot;,
  &quot;description&quot; : &quot;Time each individual bulk request should wait for shards that are unavailable.&quot;
},
..."><pre class="notranslate"><span class="pl-ii">...</span>
<span class="pl-ent">"timeout"</span>: {
  <span class="pl-ent">"type"</span> : <span class="pl-s"><span class="pl-pds">"</span>time<span class="pl-pds">"</span></span>,
  <span class="pl-ent">"description"</span> : <span class="pl-s"><span class="pl-pds">"</span>Explicit operation timeout<span class="pl-pds">"</span></span>
},
<span class="pl-ii">...</span>
<span class="pl-ent">"timeout"</span>: {
  <span class="pl-ent">"type"</span> : <span class="pl-s"><span class="pl-pds">"</span>time<span class="pl-pds">"</span></span>,
  <span class="pl-ent">"default"</span>: <span class="pl-s"><span class="pl-pds">"</span>1m<span class="pl-pds">"</span></span>,
  <span class="pl-ent">"description"</span> : <span class="pl-s"><span class="pl-pds">"</span>Time each individual bulk request should wait for shards that are unavailable.<span class="pl-pds">"</span></span>
},
<span class="pl-ii">...</span></pre></div>