<p dir="auto">Using</p>
<ol dir="auto">
<li>Elasticsearch: 2.3.1</li>
<li>JVM version: 1.8.0_60-b27</li>
<li>OS version: Centos 7</li>
</ol>
<p dir="auto">With</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=".
.
.
node:
  data: true
  master: false
  name: esd02-data4
  tier: hot
os:
  available_processors: 8
path:
  data:
      - /storage/sdb1
      - /storage/sdx1
.
.
."><pre class="notranslate"><code class="notranslate">.
.
.
node:
  data: true
  master: false
  name: esd02-data4
  tier: hot
os:
  available_processors: 8
path:
  data:
      - /storage/sdb1
      - /storage/sdx1
.
.
.
</code></pre></div>
<p dir="auto">in <code class="notranslate">elasticsearch.yml</code> we have observed twice now where one of the disks (say, <code class="notranslate">/storage/sdx1</code>) will fill up to 100% while the other <code class="notranslate">/storage/sdb1</code> still has space. The error logs fill up with errors like:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[logstash-syslog-2016.07.21][[logstash-syslog-2016.07.21][1]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: FileSystemException[/storage/sdx1/stage/nodes/0/indices/logstash-syslog-2016.07.21/1/translog/translog.ckp -&gt; /storage/sdx1/stage/nodes/0/indices/logstash-syslog-2016.07.21/1/translog/translog-7395039383191700052.tlog: No space left on device];
Caused by: [logstash-syslog-2016.07.21][[logstash-syslog-2016.07.21][1]] EngineCreationFailureException[failed to create engine]; nested: FileSystemException[/storage/sdx1/stage/nodes/0/indices/logstash-syslog-2016.07.21/1/translog/translog.ckp -&gt; /storage/sdx1/stage/nodes/0/indices/logstash-syslog-2016.07.21/1/translog/translog-7395039383191700052.tlog: No space left on device];
Caused by: java.nio.file.FileSystemException: /storage/sdx1/stage/nodes/0/indices/logstash-syslog-2016.07.21/1/translog/translog.ckp -&gt; /storage/sdx1/stage/nodes/0/indices/logstash-syslog-2016.07.21/1/translog/translog-7395039383191700052.tlog: No space left on device
java.nio.file.FileSystemException: /storage/sdb1/stage/nodes/0/indices/logstash-syslog-2016.07.21/_state/state-13787.st -&gt; /storage/sdx1/stage/nodes/0/indices/logstash-syslog-2016.07.21/_state/state-13787.st.tmp: No space left on device"><pre class="notranslate"><code class="notranslate">[logstash-syslog-2016.07.21][[logstash-syslog-2016.07.21][1]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: FileSystemException[/storage/sdx1/stage/nodes/0/indices/logstash-syslog-2016.07.21/1/translog/translog.ckp -&gt; /storage/sdx1/stage/nodes/0/indices/logstash-syslog-2016.07.21/1/translog/translog-7395039383191700052.tlog: No space left on device];
Caused by: [logstash-syslog-2016.07.21][[logstash-syslog-2016.07.21][1]] EngineCreationFailureException[failed to create engine]; nested: FileSystemException[/storage/sdx1/stage/nodes/0/indices/logstash-syslog-2016.07.21/1/translog/translog.ckp -&gt; /storage/sdx1/stage/nodes/0/indices/logstash-syslog-2016.07.21/1/translog/translog-7395039383191700052.tlog: No space left on device];
Caused by: java.nio.file.FileSystemException: /storage/sdx1/stage/nodes/0/indices/logstash-syslog-2016.07.21/1/translog/translog.ckp -&gt; /storage/sdx1/stage/nodes/0/indices/logstash-syslog-2016.07.21/1/translog/translog-7395039383191700052.tlog: No space left on device
java.nio.file.FileSystemException: /storage/sdb1/stage/nodes/0/indices/logstash-syslog-2016.07.21/_state/state-13787.st -&gt; /storage/sdx1/stage/nodes/0/indices/logstash-syslog-2016.07.21/_state/state-13787.st.tmp: No space left on device
</code></pre></div>
<p dir="auto">It's pretty obvious what's happening... <code class="notranslate">storage/sdx1</code> owns a shard of <code class="notranslate">logstash-syslog-2016.07.21</code> and ES is refusing to "break" the shard across multiple disk paths....</p>
<p dir="auto">but it's also failing to recover from this on its own, or to stop sending data to the shard whose disk is full.  I would expect either (A) in the event of a full disk, ES would nominate the other shards to accept 100% of the writes to the index or (B) ES would loosen its requirement that shards are not allowed to span multiple write paths.</p>
<p dir="auto"><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="155765302" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/18467" data-hovercard-type="pull_request" data-hovercard-url="/elastic/elasticsearch/pull/18467/hovercard" href="https://github.com/elastic/elasticsearch/pull/18467">#18467</a> was intended to limit how many times shard allocation was attempted before giving up, instead of constantly retrying. eg if an analyzer requires a local synonyms file, which is missing, then we should stop trying until the situation is resolved.</p>
<p dir="auto">Unfortunately it doesn't work for unassigned primary shards because of the way allocation works today. For instance, a primary shard might be sitting on a disk which is over the high watermark, or it may have allocation filtering which prevents it from being allocated to the node where it already exists.  Today, we just go ahead and try to assign the primary regardless of the allocation deciders (which means that we also don't limit retries to 5).</p>
<p dir="auto">Instead, we could add extra logic to the appropriate deciders to say "always return <code class="notranslate">YES</code> if the shard in question is a primary which already exists on this node".  The decision returned would be <code class="notranslate">YES</code>, but the explanation provided by the reroute or cluster allocation explain could include the reason this decider was ignored.</p>
<p dir="auto">/cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ywelsch/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ywelsch">@ywelsch</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/dakrone/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/dakrone">@dakrone</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/s1monw/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/s1monw">@s1monw</a></p>
<p dir="auto">Related to <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="154668066" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/18321" data-hovercard-type="issue" data-hovercard-url="/elastic/elasticsearch/issues/18321/hovercard" href="https://github.com/elastic/elasticsearch/issues/18321">#18321</a></p>