<p dir="auto">We started upgrading our ES two nodes cluster from 1.1.1 to 1.4.2, using the rolling upgrade.<br>
After restarting one of the upgraded node, we found one of the shards couldn't get itself back. The log out put give this error:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=x56z8s actual=1h6zri0 resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@393b946e)
    at org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
    at org.elasticsearch.index.store.Store.verify(Store.java:365)
    at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
    at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
"><pre class="notranslate"><code class="notranslate">Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=x56z8s actual=1h6zri0 resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@393b946e)
    at org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
    at org.elasticsearch.index.store.Store.verify(Store.java:365)
    at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
    at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

</code></pre></div>
<p dir="auto">Also, the expected size of that shard should be around 170G, however, the recovering directory grew to more than 650+G.</p>
<p dir="auto">I checked the hardware, which has no issues at all. But to make sure it wasn't the hard ware issue, I added a brand new node into the cluster, and it seems that particular shard has the reproducible issue on new machine too.</p>
<p dir="auto">I deleted the directory manually as mentioned in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="54391925" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/9302" data-hovercard-type="issue" data-hovercard-url="/elastic/elasticsearch/issues/9302/hovercard" href="https://github.com/elastic/elasticsearch/issues/9302">#9302</a>, the cluster didn't automatically create the replica.<br>
So I use the reroute API, to try to move that primary shard from the old version node to new version node. It seemed to be promising, because when the move finishes, the size of the directory was correct. However, it turned out that after moving, the old shard on the old version node didn't get removed, and the newly created one on new version node became just a copy of the old shard, not even a replica, because cluster didn't allocate it.</p>
<p dir="auto">Any idea about how to fix this issue?</p>
<p dir="auto">After updating form ES 0.20.6 to 1.4.2 the cluster remains in a RED state and CorruptIndexExceptions are generated:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[2015-01-05 12:57:20,933][WARN ][cluster.action.shard     ] [Bob Diamond] [index1][2] sending failed shard for [index1][2], node[gkXgdCNvSACjIFiPAVTxXA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[index1][2] failed to fetch index version after copying it over]; nested: CorruptIndexException[[index1][2] Preexisting corrupted index [corrupted_jbVtki8fRwulkaA6A3HZ4Q] caused by: CorruptIndexException[checksum failed (hardware problem?) : expected=3uya22 actual=1rkh7qi resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@49c64a7f)]
org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=3uya22 actual=1rkh7qi resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@49c64a7f)
    at org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
    at org.elasticsearch.index.store.Store.verify(Store.java:365)
    at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
    at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
    Suppressed: org.elasticsearch.transport.RemoteTransportException: [Phoenix][inet[/127.0.0.1:9302]][internal:index/shard/recovery/file_chunk]
    Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=1vw8erc actual=1ckq9x4 resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@3f592170)
        at org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
        at org.elasticsearch.index.store.Store.verify(Store.java:365)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
        at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
    Suppressed: org.elasticsearch.transport.RemoteTransportException: [Phoenix][inet[/127.0.0.1:9302]][internal:index/shard/recovery/file_chunk]
    Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=1vw8erc actual=1ckq9x4 resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@2a0df6a)
        at org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
        at org.elasticsearch.index.store.Store.verify(Store.java:365)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
        at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
    Suppressed: org.elasticsearch.transport.RemoteTransportException: [Phoenix][inet[/127.0.0.1:9302]][internal:index/shard/recovery/file_chunk]
    Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=ze9sfb actual=ne6jhj resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@7992625b)
        at org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
        at org.elasticsearch.index.store.Store.verify(Store.java:365)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
        at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
    Suppressed: org.elasticsearch.transport.RemoteTransportException: [Phoenix][inet[/127.0.0.1:9302]][internal:index/shard/recovery/file_chunk]
    Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=1om7x0v actual=1urrlkf resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@1633862f)
        at org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
        at org.elasticsearch.index.store.Store.verify(Store.java:365)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
        at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)"><pre class="notranslate"><code class="notranslate">[2015-01-05 12:57:20,933][WARN ][cluster.action.shard     ] [Bob Diamond] [index1][2] sending failed shard for [index1][2], node[gkXgdCNvSACjIFiPAVTxXA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[index1][2] failed to fetch index version after copying it over]; nested: CorruptIndexException[[index1][2] Preexisting corrupted index [corrupted_jbVtki8fRwulkaA6A3HZ4Q] caused by: CorruptIndexException[checksum failed (hardware problem?) : expected=3uya22 actual=1rkh7qi resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@49c64a7f)]
org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=3uya22 actual=1rkh7qi resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@49c64a7f)
    at org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
    at org.elasticsearch.index.store.Store.verify(Store.java:365)
    at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
    at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
    Suppressed: org.elasticsearch.transport.RemoteTransportException: [Phoenix][inet[/127.0.0.1:9302]][internal:index/shard/recovery/file_chunk]
    Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=1vw8erc actual=1ckq9x4 resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@3f592170)
        at org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
        at org.elasticsearch.index.store.Store.verify(Store.java:365)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
        at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
    Suppressed: org.elasticsearch.transport.RemoteTransportException: [Phoenix][inet[/127.0.0.1:9302]][internal:index/shard/recovery/file_chunk]
    Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=1vw8erc actual=1ckq9x4 resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@2a0df6a)
        at org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
        at org.elasticsearch.index.store.Store.verify(Store.java:365)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
        at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
    Suppressed: org.elasticsearch.transport.RemoteTransportException: [Phoenix][inet[/127.0.0.1:9302]][internal:index/shard/recovery/file_chunk]
    Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=ze9sfb actual=ne6jhj resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@7992625b)
        at org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
        at org.elasticsearch.index.store.Store.verify(Store.java:365)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
        at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
    Suppressed: org.elasticsearch.transport.RemoteTransportException: [Phoenix][inet[/127.0.0.1:9302]][internal:index/shard/recovery/file_chunk]
    Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=1om7x0v actual=1urrlkf resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@1633862f)
        at org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
        at org.elasticsearch.index.store.Store.verify(Store.java:365)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
        at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
</code></pre></div>
<p dir="auto">I reproduced the error with a vanilla ES, these are the steps:</p>
<ol dir="auto">
<li>
<p dir="auto">Start cluster ES 0.20.6 nodes</p>
</li>
<li>
<p dir="auto">Index data (one index ~100MB)</p>
</li>
<li>
<p dir="auto">Cluster health</p>
</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl -XGET http://localhost:9200/_cluster/health\?pretty\=true
{
  &quot;cluster_name&quot; : &quot;elasticsearch&quot;,
  &quot;status&quot; : &quot;green&quot;,
  &quot;timed_out&quot; : false,
  &quot;number_of_nodes&quot; : 4,
  &quot;number_of_data_nodes&quot; : 4,
  &quot;active_primary_shards&quot; : 5,
  &quot;active_shards&quot; : 10,
  &quot;relocating_shards&quot; : 0,
  &quot;initializing_shards&quot; : 0,
  &quot;unassigned_shards&quot; : 0
}"><pre class="notranslate"><code class="notranslate">curl -XGET http://localhost:9200/_cluster/health\?pretty\=true
{
  "cluster_name" : "elasticsearch",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 4,
  "number_of_data_nodes" : 4,
  "active_primary_shards" : 5,
  "active_shards" : 10,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0
}
</code></pre></div>
<ol start="4" dir="auto">
<li>Disable shard allocation</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl -XPUT localhost:9200/_cluster/settings -d '{&quot;persistent&quot;:{&quot;cluster.routing.allocation.disable_allocation”:true}}’

curl -XGET localhost:9200/_cluster/settings

{&quot;persistent&quot;:{&quot;cluster.routing.allocation.disable_allocation&quot;:&quot;true&quot;},&quot;transient&quot;:{}}"><pre class="notranslate"><code class="notranslate">curl -XPUT localhost:9200/_cluster/settings -d '{"persistent":{"cluster.routing.allocation.disable_allocation”:true}}’

curl -XGET localhost:9200/_cluster/settings

{"persistent":{"cluster.routing.allocation.disable_allocation":"true"},"transient":{}}
</code></pre></div>
<ol start="5" dir="auto">
<li>Shutdown cluster</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl -XPOST http://localhost:9200/_shutdown

{&quot;cluster_name&quot;:&quot;elasticsearch&quot;,&quot;nodes&quot;:{&quot;hxt59kUgTBCN8NAB6K2KwQ&quot;:{&quot;name&quot;:&quot;Whitman, Debra&quot;},&quot;h7JQP60BShWadDNFTB94CA&quot;:{&quot;name&quot;:&quot;Corbo, Adrian&quot;},&quot;T_Qzz66MQ9uTJeEwe7Q4Ig&quot;:{&quot;name&quot;:&quot;Surtur&quot;},&quot;l8Xex8xuRnSIM1cMJgvhvA&quot;:{&quot;name&quot;:&quot;Shriker&quot;}}}"><pre class="notranslate"><code class="notranslate">curl -XPOST http://localhost:9200/_shutdown

{"cluster_name":"elasticsearch","nodes":{"hxt59kUgTBCN8NAB6K2KwQ":{"name":"Whitman, Debra"},"h7JQP60BShWadDNFTB94CA":{"name":"Corbo, Adrian"},"T_Qzz66MQ9uTJeEwe7Q4Ig":{"name":"Surtur"},"l8Xex8xuRnSIM1cMJgvhvA":{"name":"Shriker"}}}
</code></pre></div>
<ol start="6" dir="auto">
<li>
<p dir="auto">Update</p>
</li>
<li>
<p dir="auto">Start cluster ES 1.4.2 nodes</p>
</li>
<li>
<p dir="auto">Cluster health</p>
</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl -XGET http://localhost:9200/_cluster/health\?pretty\=true
{
  &quot;cluster_name&quot; : &quot;elasticsearch&quot;,
  &quot;status&quot; : &quot;yellow&quot;,
  &quot;timed_out&quot; : false,
  &quot;number_of_nodes&quot; : 4,
  &quot;number_of_data_nodes&quot; : 4,
  &quot;active_primary_shards&quot; : 5,
  &quot;active_shards&quot; : 5,
  &quot;relocating_shards&quot; : 0,
  &quot;initializing_shards&quot; : 0,
  &quot;unassigned_shards&quot; : 5
}"><pre class="notranslate"><code class="notranslate">curl -XGET http://localhost:9200/_cluster/health\?pretty\=true
{
  "cluster_name" : "elasticsearch",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 4,
  "number_of_data_nodes" : 4,
  "active_primary_shards" : 5,
  "active_shards" : 5,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 5
}
</code></pre></div>
<ol start="9" dir="auto">
<li>Enable shard allocation</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl -XPUT localhost:9200/_cluster/settings -d '{&quot;persistent&quot;:{&quot;cluster.routing.allocation.disable_allocation&quot;:false}}'
{&quot;acknowledged&quot;:true,&quot;persistent&quot;:{&quot;cluster&quot;:{&quot;routing&quot;:{&quot;allocation&quot;:{&quot;disable_allocation&quot;:&quot;false&quot;}}}},&quot;transient&quot;:{}}"><pre class="notranslate"><code class="notranslate">curl -XPUT localhost:9200/_cluster/settings -d '{"persistent":{"cluster.routing.allocation.disable_allocation":false}}'
{"acknowledged":true,"persistent":{"cluster":{"routing":{"allocation":{"disable_allocation":"false"}}}},"transient":{}}
</code></pre></div>
<ol start="10" dir="auto">
<li>Cluster health</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="curl -XGET http://localhost:9200/_cluster/health\?pretty\=true
{
  &quot;cluster_name&quot; : &quot;elasticsearch&quot;,
  &quot;status&quot; : &quot;red&quot;,
  &quot;timed_out&quot; : false,
  &quot;number_of_nodes&quot; : 4,
  &quot;number_of_data_nodes&quot; : 4,
  &quot;active_primary_shards&quot; : 0,
  &quot;active_shards&quot; : 0,
  &quot;relocating_shards&quot; : 0,
  &quot;initializing_shards&quot; : 5,
  &quot;unassigned_shards&quot; : 5
}"><pre class="notranslate"><code class="notranslate">curl -XGET http://localhost:9200/_cluster/health\?pretty\=true
{
  "cluster_name" : "elasticsearch",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 4,
  "number_of_data_nodes" : 4,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 5,
  "unassigned_shards" : 5
}
</code></pre></div>
<p dir="auto">Some logs from the log files:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[2015-01-05 12:42:12,579][INFO ][node                     ] [Roma] version[1.4.2], pid[65313], build[927caff/2014-12-16T14:11:12Z]
[2015-01-05 12:42:12,580][INFO ][node                     ] [Roma] initializing ...
[2015-01-05 12:42:12,584][INFO ][plugins                  ] [Roma] loaded [], sites []
[2015-01-05 12:42:15,173][INFO ][node                     ] [Roma] initialized
[2015-01-05 12:42:15,173][INFO ][node                     ] [Roma] starting ...
[2015-01-05 12:42:15,241][INFO ][transport                ] [Roma] bound_address {inet[/127.0.0.1:9300]}, publish_address {inet[/127.0.0.1:9300]}
[2015-01-05 12:42:15,260][INFO ][discovery                ] [Roma] elasticsearch/BvPgLtdCS8eIEFUATx2lVw
[2015-01-05 12:42:19,035][INFO ][cluster.service          ] [Roma] new_master [Roma][BvPgLtdCS8eIEFUATx2lVw][dw1949demum.int.demandware.com][inet[/127.0.0.1:9300]], reason: zen-disco-join (elected_as_master)
[2015-01-05 12:42:19,051][INFO ][http                     ] [Roma] bound_address {inet[/127.0.0.1:9200]}, publish_address {inet[/127.0.0.1:9200]}
[2015-01-05 12:42:19,052][INFO ][node                     ] [Roma] started
[2015-01-05 12:42:19,505][INFO ][cluster.routing.allocation.decider] [Roma] updating [cluster.routing.allocation.disable_allocation] from [false] to [true]
[2015-01-05 12:42:19,511][INFO ][gateway                  ] [Roma] recovered [1] indices into cluster_state
[2015-01-05 12:42:25,403][INFO ][cluster.service          ] [Roma] added {[Bob Diamond][gkXgdCNvSACjIFiPAVTxXA][dw1949demum.int.demandware.com][inet[/127.0.0.1:9301]],}, reason: zen-disco-receive(join from node[[Bob Diamond][gkXgdCNvSACjIFiPAVTxXA][dw1949demum.int.demandware.com][inet[/127.0.0.1:9301]]])
[2015-01-05 12:42:32,863][INFO ][cluster.service          ] [Roma] added {[Phoenix][LTdV5wDIQtuJUPTl9NuuJw][dw1949demum.int.demandware.com][inet[/127.0.0.1:9302]],}, reason: zen-disco-receive(join from node[[Phoenix][LTdV5wDIQtuJUPTl9NuuJw][dw1949demum.int.demandware.com][inet[/127.0.0.1:9302]]])
[2015-01-05 12:42:40,505][INFO ][cluster.service          ] [Roma] added {[Druid][ftzmRGfARAuhjPfw9CVE5Q][dw1949demum.int.demandware.com][inet[/127.0.0.1:9303]],}, reason: zen-disco-receive(join from node[[Druid][ftzmRGfARAuhjPfw9CVE5Q][dw1949demum.int.demandware.com][inet[/127.0.0.1:9303]]])
[2015-01-05 12:56:58,560][INFO ][cluster.routing.allocation.decider] [Roma] updating [cluster.routing.allocation.disable_allocation] from [true] to [false]
[2015-01-05 12:56:59,184][WARN ][cluster.action.shard     ] [Roma] [index1][2] received shard failed for [index1][2], node[gkXgdCNvSACjIFiPAVTxXA], [P], s[STARTED], indexUUID [_na_], reason [engine failure, message [corrupt file detected source: [recovery phase 1]][RecoverFilesRecoveryException[[index1][2] Failed to transfer [54] files with total size of [4.3mb]]; nested: CorruptIndexException[checksum failed (hardware problem?) : expected=3uya22 actual=1rkh7qi resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@49c64a7f)]; ]]
[2015-01-05 12:56:59,740][WARN ][cluster.action.shard     ] [Roma] [index1][1] received shard failed for [index1][1], node[gkXgdCNvSACjIFiPAVTxXA], [P], s[STARTED], indexUUID [_na_], reason [engine failure, message [corrupt file detected source: [recovery phase 1]][RecoverFilesRecoveryException[[index1][1] Failed to transfer [94] files with total size of [4.3mb]]; nested: CorruptIndexException[checksum failed (hardware problem?) : expected=1vw8erc actual=1ckq9x4 resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@47778e4f)]; ]]
[2015-01-05 12:57:09,641][WARN ][cluster.action.shard     ] [Roma] [index1][3] received shard failed for [index1][3], node[LTdV5wDIQtuJUPTl9NuuJw], [P], s[STARTED], indexUUID [_na_], reason [engine failure, message [corrupt file detected source: [recovery phase 1]][RecoverFilesRecoveryException[[index1][3] Failed to transfer [114] files with total size of [4.3mb]]; nested: CorruptIndexException[checksum failed (hardware problem?) : expected=1vw8erc actual=1ckq9x4 resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@7e6256a6)]; ]]
[2015-01-05 12:57:09,663][WARN ][cluster.action.shard     ] [Roma] [index1][2] received shard failed for [index1][2], node[LTdV5wDIQtuJUPTl9NuuJw], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[index1][2] failed recovery]; nested: EngineCreationFailureException[[index1][2] failed to open reader on writer]; nested: FileNotFoundException[No such file [_in.tis]]; ]]
[2015-01-05 12:57:09,695][WARN ][cluster.action.shard     ] [Roma] [index1][2] received shard failed for [index1][2], node[gkXgdCNvSACjIFiPAVTxXA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[index1][2] failed to fetch index version after copying it over]; nested: CorruptIndexException[[index1][2] Preexisting corrupted index [corrupted_jbVtki8fRwulkaA6A3HZ4Q] caused by: CorruptIndexException[checksum failed (hardware problem?) : expected=3uya22 actual=1rkh7qi resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@49c64a7f)]
org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=3uya22 actual=1rkh7qi resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@49c64a7f)
    at org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
    at org.elasticsearch.index.store.Store.verify(Store.java:365)
    at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
    at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
    Suppressed: org.elasticsearch.transport.RemoteTransportException: [Phoenix][inet[/127.0.0.1:9302]][internal:index/shard/recovery/file_chunk]
    Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=1vw8erc actual=1ckq9x4 resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@3f592170)
        at org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
        at org.elasticsearch.index.store.Store.verify(Store.java:365)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
        at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
    Suppressed: org.elasticsearch.transport.RemoteTransportException: [Phoenix][inet[/127.0.0.1:9302]][internal:index/shard/recovery/file_chunk]
    Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=1vw8erc actual=1ckq9x4 resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@2a0df6a)
        at org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
        at org.elasticsearch.index.store.Store.verify(Store.java:365)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
        at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
    Suppressed: org.elasticsearch.transport.RemoteTransportException: [Phoenix][inet[/127.0.0.1:9302]][internal:index/shard/recovery/file_chunk]
    Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=ze9sfb actual=ne6jhj resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@7992625b)
        at org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
        at org.elasticsearch.index.store.Store.verify(Store.java:365)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
        at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745
..."><pre class="notranslate"><code class="notranslate">[2015-01-05 12:42:12,579][INFO ][node                     ] [Roma] version[1.4.2], pid[65313], build[927caff/2014-12-16T14:11:12Z]
[2015-01-05 12:42:12,580][INFO ][node                     ] [Roma] initializing ...
[2015-01-05 12:42:12,584][INFO ][plugins                  ] [Roma] loaded [], sites []
[2015-01-05 12:42:15,173][INFO ][node                     ] [Roma] initialized
[2015-01-05 12:42:15,173][INFO ][node                     ] [Roma] starting ...
[2015-01-05 12:42:15,241][INFO ][transport                ] [Roma] bound_address {inet[/127.0.0.1:9300]}, publish_address {inet[/127.0.0.1:9300]}
[2015-01-05 12:42:15,260][INFO ][discovery                ] [Roma] elasticsearch/BvPgLtdCS8eIEFUATx2lVw
[2015-01-05 12:42:19,035][INFO ][cluster.service          ] [Roma] new_master [Roma][BvPgLtdCS8eIEFUATx2lVw][dw1949demum.int.demandware.com][inet[/127.0.0.1:9300]], reason: zen-disco-join (elected_as_master)
[2015-01-05 12:42:19,051][INFO ][http                     ] [Roma] bound_address {inet[/127.0.0.1:9200]}, publish_address {inet[/127.0.0.1:9200]}
[2015-01-05 12:42:19,052][INFO ][node                     ] [Roma] started
[2015-01-05 12:42:19,505][INFO ][cluster.routing.allocation.decider] [Roma] updating [cluster.routing.allocation.disable_allocation] from [false] to [true]
[2015-01-05 12:42:19,511][INFO ][gateway                  ] [Roma] recovered [1] indices into cluster_state
[2015-01-05 12:42:25,403][INFO ][cluster.service          ] [Roma] added {[Bob Diamond][gkXgdCNvSACjIFiPAVTxXA][dw1949demum.int.demandware.com][inet[/127.0.0.1:9301]],}, reason: zen-disco-receive(join from node[[Bob Diamond][gkXgdCNvSACjIFiPAVTxXA][dw1949demum.int.demandware.com][inet[/127.0.0.1:9301]]])
[2015-01-05 12:42:32,863][INFO ][cluster.service          ] [Roma] added {[Phoenix][LTdV5wDIQtuJUPTl9NuuJw][dw1949demum.int.demandware.com][inet[/127.0.0.1:9302]],}, reason: zen-disco-receive(join from node[[Phoenix][LTdV5wDIQtuJUPTl9NuuJw][dw1949demum.int.demandware.com][inet[/127.0.0.1:9302]]])
[2015-01-05 12:42:40,505][INFO ][cluster.service          ] [Roma] added {[Druid][ftzmRGfARAuhjPfw9CVE5Q][dw1949demum.int.demandware.com][inet[/127.0.0.1:9303]],}, reason: zen-disco-receive(join from node[[Druid][ftzmRGfARAuhjPfw9CVE5Q][dw1949demum.int.demandware.com][inet[/127.0.0.1:9303]]])
[2015-01-05 12:56:58,560][INFO ][cluster.routing.allocation.decider] [Roma] updating [cluster.routing.allocation.disable_allocation] from [true] to [false]
[2015-01-05 12:56:59,184][WARN ][cluster.action.shard     ] [Roma] [index1][2] received shard failed for [index1][2], node[gkXgdCNvSACjIFiPAVTxXA], [P], s[STARTED], indexUUID [_na_], reason [engine failure, message [corrupt file detected source: [recovery phase 1]][RecoverFilesRecoveryException[[index1][2] Failed to transfer [54] files with total size of [4.3mb]]; nested: CorruptIndexException[checksum failed (hardware problem?) : expected=3uya22 actual=1rkh7qi resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@49c64a7f)]; ]]
[2015-01-05 12:56:59,740][WARN ][cluster.action.shard     ] [Roma] [index1][1] received shard failed for [index1][1], node[gkXgdCNvSACjIFiPAVTxXA], [P], s[STARTED], indexUUID [_na_], reason [engine failure, message [corrupt file detected source: [recovery phase 1]][RecoverFilesRecoveryException[[index1][1] Failed to transfer [94] files with total size of [4.3mb]]; nested: CorruptIndexException[checksum failed (hardware problem?) : expected=1vw8erc actual=1ckq9x4 resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@47778e4f)]; ]]
[2015-01-05 12:57:09,641][WARN ][cluster.action.shard     ] [Roma] [index1][3] received shard failed for [index1][3], node[LTdV5wDIQtuJUPTl9NuuJw], [P], s[STARTED], indexUUID [_na_], reason [engine failure, message [corrupt file detected source: [recovery phase 1]][RecoverFilesRecoveryException[[index1][3] Failed to transfer [114] files with total size of [4.3mb]]; nested: CorruptIndexException[checksum failed (hardware problem?) : expected=1vw8erc actual=1ckq9x4 resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@7e6256a6)]; ]]
[2015-01-05 12:57:09,663][WARN ][cluster.action.shard     ] [Roma] [index1][2] received shard failed for [index1][2], node[LTdV5wDIQtuJUPTl9NuuJw], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[index1][2] failed recovery]; nested: EngineCreationFailureException[[index1][2] failed to open reader on writer]; nested: FileNotFoundException[No such file [_in.tis]]; ]]
[2015-01-05 12:57:09,695][WARN ][cluster.action.shard     ] [Roma] [index1][2] received shard failed for [index1][2], node[gkXgdCNvSACjIFiPAVTxXA], [P], s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[index1][2] failed to fetch index version after copying it over]; nested: CorruptIndexException[[index1][2] Preexisting corrupted index [corrupted_jbVtki8fRwulkaA6A3HZ4Q] caused by: CorruptIndexException[checksum failed (hardware problem?) : expected=3uya22 actual=1rkh7qi resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@49c64a7f)]
org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=3uya22 actual=1rkh7qi resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@49c64a7f)
    at org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
    at org.elasticsearch.index.store.Store.verify(Store.java:365)
    at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
    at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
    Suppressed: org.elasticsearch.transport.RemoteTransportException: [Phoenix][inet[/127.0.0.1:9302]][internal:index/shard/recovery/file_chunk]
    Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=1vw8erc actual=1ckq9x4 resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@3f592170)
        at org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
        at org.elasticsearch.index.store.Store.verify(Store.java:365)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
        at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
    Suppressed: org.elasticsearch.transport.RemoteTransportException: [Phoenix][inet[/127.0.0.1:9302]][internal:index/shard/recovery/file_chunk]
    Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=1vw8erc actual=1ckq9x4 resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@2a0df6a)
        at org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
        at org.elasticsearch.index.store.Store.verify(Store.java:365)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
        at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
    Suppressed: org.elasticsearch.transport.RemoteTransportException: [Phoenix][inet[/127.0.0.1:9302]][internal:index/shard/recovery/file_chunk]
    Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=ze9sfb actual=ne6jhj resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@7992625b)
        at org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
        at org.elasticsearch.index.store.Store.verify(Store.java:365)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
        at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745
...
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[2015-01-05 12:42:19,707][INFO ][node                     ] [Bob Diamond] version[1.4.2], pid[65324], build[927caff/2014-12-16T14:11:12Z]
[2015-01-05 12:42:19,707][INFO ][node                     ] [Bob Diamond] initializing ...
[2015-01-05 12:42:19,712][INFO ][plugins                  ] [Bob Diamond] loaded [], sites []
[2015-01-05 12:42:22,303][INFO ][node                     ] [Bob Diamond] initialized
[2015-01-05 12:42:22,304][INFO ][node                     ] [Bob Diamond] starting ...
[2015-01-05 12:42:22,365][INFO ][transport                ] [Bob Diamond] bound_address {inet[/127.0.0.1:9301]}, publish_address {inet[/127.0.0.1:9301]}
[2015-01-05 12:42:22,378][INFO ][discovery                ] [Bob Diamond] elasticsearch/gkXgdCNvSACjIFiPAVTxXA
[2015-01-05 12:42:25,417][INFO ][cluster.service          ] [Bob Diamond] detected_master [Roma][BvPgLtdCS8eIEFUATx2lVw][dw1949demum.int.demandware.com][inet[/127.0.0.1:9300]], added {[Roma][BvPgLtdCS8eIEFUATx2lVw][dw1949demum.int.demandware.com][inet[/127.0.0.1:9300]],}, reason: zen-disco-receive(from master [[Roma][BvPgLtdCS8eIEFUATx2lVw][dw1949demum.int.demandware.com][inet[/127.0.0.1:9300]]])
[2015-01-05 12:42:25,423][INFO ][cluster.routing.allocation.decider] [Bob Diamond] updating [cluster.routing.allocation.disable_allocation] from [false] to [true]
[2015-01-05 12:42:25,436][INFO ][http                     ] [Bob Diamond] bound_address {inet[/127.0.0.1:9201]}, publish_address {inet[/127.0.0.1:9201]}
[2015-01-05 12:42:25,436][INFO ][node                     ] [Bob Diamond] started
[2015-01-05 12:42:32,864][INFO ][cluster.service          ] [Bob Diamond] added {[Phoenix][LTdV5wDIQtuJUPTl9NuuJw][dw1949demum.int.demandware.com][inet[/127.0.0.1:9302]],}, reason: zen-disco-receive(from master [[Roma][BvPgLtdCS8eIEFUATx2lVw][dw1949demum.int.demandware.com][inet[/127.0.0.1:9300]]])
[2015-01-05 12:42:40,507][INFO ][cluster.service          ] [Bob Diamond] added {[Druid][ftzmRGfARAuhjPfw9CVE5Q][dw1949demum.int.demandware.com][inet[/127.0.0.1:9303]],}, reason: zen-disco-receive(from master [[Roma][BvPgLtdCS8eIEFUATx2lVw][dw1949demum.int.demandware.com][inet[/127.0.0.1:9300]]])
[2015-01-05 12:56:58,557][INFO ][cluster.routing.allocation.decider] [Bob Diamond] updating [cluster.routing.allocation.disable_allocation] from [true] to [false]
[2015-01-05 12:56:58,801][WARN ][indices.recovery         ] [Bob Diamond] [index1][2] Corrupted file detected name [_im.tis], length [530], checksum [3uya22], writtenBy [null] checksum mismatch
[2015-01-05 12:56:58,801][WARN ][indices.recovery         ] [Bob Diamond] [index1][2] Corrupted file detected name [_im.tii], length [35], checksum [1vw8erc], writtenBy [null] checksum mismatch
[2015-01-05 12:56:58,807][WARN ][indices.recovery         ] [Bob Diamond] [index1][2] Corrupted file detected name [_io.tii], length [35], checksum [1vw8erc], writtenBy [null] checksum mismatch
[2015-01-05 12:56:58,814][WARN ][indices.recovery         ] [Bob Diamond] [index1][2] Corrupted file detected name [_io.tis], length [537], checksum [ze9sfb], writtenBy [null] checksum mismatch
[2015-01-05 12:56:58,841][WARN ][indices.recovery         ] [Bob Diamond] [index1][2] Corrupted file detected name [_in.tii], length [1124], checksum [1om7x0v], writtenBy [null] checksum mismatch
[2015-01-05 12:56:58,906][WARN ][indices.recovery         ] [Bob Diamond] [index1][2] Corrupted file detected name [_in.tis], length [75185], checksum [elttvz], writtenBy [null] checksum mismatch
[2015-01-05 12:56:59,075][WARN ][indices.recovery         ] [Bob Diamond] [index1][2] Corrupted file detected name [_iq.tis], length [438], checksum [1wqfspo], writtenBy [null] checksum mismatch
[2015-01-05 12:56:59,081][WARN ][indices.recovery         ] [Bob Diamond] [index1][2] Corrupted file detected name [_iq.tii], length [35], checksum [1vw8erc], writtenBy [null] checksum mismatch
[2015-01-05 12:56:59,089][WARN ][indices.recovery         ] [Bob Diamond] [index1][2] Corrupted file detected name [_ip.tis], length [522], checksum [gn93en], writtenBy [null] checksum mismatch
[2015-01-05 12:56:59,093][WARN ][indices.recovery         ] [Bob Diamond] [index1][2] Corrupted file detected name [_ip.tii], length [35], checksum [1vw8erc], writtenBy [null] checksum mismatch
[2015-01-05 12:56:59,099][WARN ][index.engine.internal    ] [Bob Diamond] [index1][2] failed engine [corrupt file detected source: [recovery phase 1]]
org.elasticsearch.indices.recovery.RecoverFilesRecoveryException: [index1][2] Failed to transfer [54] files with total size of [4.3mb]
    at org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:276)
    at org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1116)
    at org.elasticsearch.index.shard.service.InternalIndexShard.recover(InternalIndexShard.java:654)
    at org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:137)
    at org.elasticsearch.indices.recovery.RecoverySource.access$2600(RecoverySource.java:74)
    at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:464)
    at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:450)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=3uya22 actual=1rkh7qi resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@49c64a7f)
    at org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
    at org.elasticsearch.index.store.Store.verify(Store.java:365)
    at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
    at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
    ... 4 more
    Suppressed: org.elasticsearch.transport.RemoteTransportException: [Phoenix][inet[/127.0.0.1:9302]][internal:index/shard/recovery/file_chunk]
    Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=1vw8erc actual=1ckq9x4 resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@3f592170)
        at org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
        at org.elasticsearch.index.store.Store.verify(Store.java:365)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
        at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
    Suppressed: org.elasticsearch.transport.RemoteTransportException: [Phoenix][inet[/127.0.0.1:9302]][internal:index/shard/recovery/file_chunk]
    Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=1vw8erc actual=1ckq9x4 resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@2a0df6a)
        at org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
        at org.elasticsearch.index.store.Store.verify(Store.java:365)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
        at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
    Suppressed: org.elasticsearch.transport.RemoteTransportException: [Phoenix][inet[/127.0.0.1:9302]][internal:index/shard/recovery/file_chunk]
    Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=ze9sfb actual=ne6jhj resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@7992625b)
        at org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
        at org.elasticsearch.index.store.Store.verify(Store.java:365)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
        at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
..."><pre class="notranslate"><code class="notranslate">[2015-01-05 12:42:19,707][INFO ][node                     ] [Bob Diamond] version[1.4.2], pid[65324], build[927caff/2014-12-16T14:11:12Z]
[2015-01-05 12:42:19,707][INFO ][node                     ] [Bob Diamond] initializing ...
[2015-01-05 12:42:19,712][INFO ][plugins                  ] [Bob Diamond] loaded [], sites []
[2015-01-05 12:42:22,303][INFO ][node                     ] [Bob Diamond] initialized
[2015-01-05 12:42:22,304][INFO ][node                     ] [Bob Diamond] starting ...
[2015-01-05 12:42:22,365][INFO ][transport                ] [Bob Diamond] bound_address {inet[/127.0.0.1:9301]}, publish_address {inet[/127.0.0.1:9301]}
[2015-01-05 12:42:22,378][INFO ][discovery                ] [Bob Diamond] elasticsearch/gkXgdCNvSACjIFiPAVTxXA
[2015-01-05 12:42:25,417][INFO ][cluster.service          ] [Bob Diamond] detected_master [Roma][BvPgLtdCS8eIEFUATx2lVw][dw1949demum.int.demandware.com][inet[/127.0.0.1:9300]], added {[Roma][BvPgLtdCS8eIEFUATx2lVw][dw1949demum.int.demandware.com][inet[/127.0.0.1:9300]],}, reason: zen-disco-receive(from master [[Roma][BvPgLtdCS8eIEFUATx2lVw][dw1949demum.int.demandware.com][inet[/127.0.0.1:9300]]])
[2015-01-05 12:42:25,423][INFO ][cluster.routing.allocation.decider] [Bob Diamond] updating [cluster.routing.allocation.disable_allocation] from [false] to [true]
[2015-01-05 12:42:25,436][INFO ][http                     ] [Bob Diamond] bound_address {inet[/127.0.0.1:9201]}, publish_address {inet[/127.0.0.1:9201]}
[2015-01-05 12:42:25,436][INFO ][node                     ] [Bob Diamond] started
[2015-01-05 12:42:32,864][INFO ][cluster.service          ] [Bob Diamond] added {[Phoenix][LTdV5wDIQtuJUPTl9NuuJw][dw1949demum.int.demandware.com][inet[/127.0.0.1:9302]],}, reason: zen-disco-receive(from master [[Roma][BvPgLtdCS8eIEFUATx2lVw][dw1949demum.int.demandware.com][inet[/127.0.0.1:9300]]])
[2015-01-05 12:42:40,507][INFO ][cluster.service          ] [Bob Diamond] added {[Druid][ftzmRGfARAuhjPfw9CVE5Q][dw1949demum.int.demandware.com][inet[/127.0.0.1:9303]],}, reason: zen-disco-receive(from master [[Roma][BvPgLtdCS8eIEFUATx2lVw][dw1949demum.int.demandware.com][inet[/127.0.0.1:9300]]])
[2015-01-05 12:56:58,557][INFO ][cluster.routing.allocation.decider] [Bob Diamond] updating [cluster.routing.allocation.disable_allocation] from [true] to [false]
[2015-01-05 12:56:58,801][WARN ][indices.recovery         ] [Bob Diamond] [index1][2] Corrupted file detected name [_im.tis], length [530], checksum [3uya22], writtenBy [null] checksum mismatch
[2015-01-05 12:56:58,801][WARN ][indices.recovery         ] [Bob Diamond] [index1][2] Corrupted file detected name [_im.tii], length [35], checksum [1vw8erc], writtenBy [null] checksum mismatch
[2015-01-05 12:56:58,807][WARN ][indices.recovery         ] [Bob Diamond] [index1][2] Corrupted file detected name [_io.tii], length [35], checksum [1vw8erc], writtenBy [null] checksum mismatch
[2015-01-05 12:56:58,814][WARN ][indices.recovery         ] [Bob Diamond] [index1][2] Corrupted file detected name [_io.tis], length [537], checksum [ze9sfb], writtenBy [null] checksum mismatch
[2015-01-05 12:56:58,841][WARN ][indices.recovery         ] [Bob Diamond] [index1][2] Corrupted file detected name [_in.tii], length [1124], checksum [1om7x0v], writtenBy [null] checksum mismatch
[2015-01-05 12:56:58,906][WARN ][indices.recovery         ] [Bob Diamond] [index1][2] Corrupted file detected name [_in.tis], length [75185], checksum [elttvz], writtenBy [null] checksum mismatch
[2015-01-05 12:56:59,075][WARN ][indices.recovery         ] [Bob Diamond] [index1][2] Corrupted file detected name [_iq.tis], length [438], checksum [1wqfspo], writtenBy [null] checksum mismatch
[2015-01-05 12:56:59,081][WARN ][indices.recovery         ] [Bob Diamond] [index1][2] Corrupted file detected name [_iq.tii], length [35], checksum [1vw8erc], writtenBy [null] checksum mismatch
[2015-01-05 12:56:59,089][WARN ][indices.recovery         ] [Bob Diamond] [index1][2] Corrupted file detected name [_ip.tis], length [522], checksum [gn93en], writtenBy [null] checksum mismatch
[2015-01-05 12:56:59,093][WARN ][indices.recovery         ] [Bob Diamond] [index1][2] Corrupted file detected name [_ip.tii], length [35], checksum [1vw8erc], writtenBy [null] checksum mismatch
[2015-01-05 12:56:59,099][WARN ][index.engine.internal    ] [Bob Diamond] [index1][2] failed engine [corrupt file detected source: [recovery phase 1]]
org.elasticsearch.indices.recovery.RecoverFilesRecoveryException: [index1][2] Failed to transfer [54] files with total size of [4.3mb]
    at org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:276)
    at org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:1116)
    at org.elasticsearch.index.shard.service.InternalIndexShard.recover(InternalIndexShard.java:654)
    at org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:137)
    at org.elasticsearch.indices.recovery.RecoverySource.access$2600(RecoverySource.java:74)
    at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:464)
    at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:450)
    at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=3uya22 actual=1rkh7qi resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@49c64a7f)
    at org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
    at org.elasticsearch.index.store.Store.verify(Store.java:365)
    at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
    at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
    ... 4 more
    Suppressed: org.elasticsearch.transport.RemoteTransportException: [Phoenix][inet[/127.0.0.1:9302]][internal:index/shard/recovery/file_chunk]
    Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=1vw8erc actual=1ckq9x4 resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@3f592170)
        at org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
        at org.elasticsearch.index.store.Store.verify(Store.java:365)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
        at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
    Suppressed: org.elasticsearch.transport.RemoteTransportException: [Phoenix][inet[/127.0.0.1:9302]][internal:index/shard/recovery/file_chunk]
    Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=1vw8erc actual=1ckq9x4 resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@2a0df6a)
        at org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
        at org.elasticsearch.index.store.Store.verify(Store.java:365)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
        at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
    Suppressed: org.elasticsearch.transport.RemoteTransportException: [Phoenix][inet[/127.0.0.1:9302]][internal:index/shard/recovery/file_chunk]
    Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=ze9sfb actual=ne6jhj resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@7992625b)
        at org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
        at org.elasticsearch.index.store.Store.verify(Store.java:365)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
        at org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
        at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
...
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[2015-01-05 12:42:34,825][INFO ][node                     ] [Druid] version[1.4.2], pid[65351], build[927caff/2014-12-16T14:11:12Z]
[2015-01-05 12:42:34,826][INFO ][node                     ] [Druid] initializing ...
[2015-01-05 12:42:34,830][INFO ][plugins                  ] [Druid] loaded [], sites []
[2015-01-05 12:42:37,395][INFO ][node                     ] [Druid] initialized
[2015-01-05 12:42:37,395][INFO ][node                     ] [Druid] starting ...
[2015-01-05 12:42:37,466][INFO ][transport                ] [Druid] bound_address {inet[/127.0.0.1:9303]}, publish_address {inet[/127.0.0.1:9303]}
[2015-01-05 12:42:37,480][INFO ][discovery                ] [Druid] elasticsearch/ftzmRGfARAuhjPfw9CVE5Q
[2015-01-05 12:42:40,516][INFO ][cluster.service          ] [Druid] detected_master [Roma][BvPgLtdCS8eIEFUATx2lVw][dw1949demum.int.demandware.com][inet[/127.0.0.1:9300]], added {[Roma][BvPgLtdCS8eIEFUATx2lVw][dw1949demum.int.demandware.com][inet[/127.0.0.1:9300]],[Bob Diamond][gkXgdCNvSACjIFiPAVTxXA][dw1949demum.int.demandware.com][inet[/127.0.0.1:9301]],[Phoenix][LTdV5wDIQtuJUPTl9NuuJw][dw1949demum.int.demandware.com][inet[/127.0.0.1:9302]],}, reason: zen-disco-receive(from master [[Roma][BvPgLtdCS8eIEFUATx2lVw][dw1949demum.int.demandware.com][inet[/127.0.0.1:9300]]])
[2015-01-05 12:42:40,526][INFO ][cluster.routing.allocation.decider] [Druid] updating [cluster.routing.allocation.disable_allocation] from [false] to [true]
[2015-01-05 12:42:40,540][INFO ][http                     ] [Druid] bound_address {inet[/127.0.0.1:9203]}, publish_address {inet[/127.0.0.1:9203]}
[2015-01-05 12:42:40,540][INFO ][node                     ] [Druid] started
[2015-01-05 12:56:58,557][INFO ][cluster.routing.allocation.decider] [Druid] updating [cluster.routing.allocation.disable_allocation] from [true] to [false]
[2015-01-05 12:57:09,746][WARN ][indices.cluster          ] [Druid] [index1][1] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [index1][1] failed recovery
    at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:185)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [index1][1] failed to open reader on writer
    at org.elasticsearch.index.engine.internal.InternalEngine.start(InternalEngine.java:321)
    at org.elasticsearch.index.shard.service.InternalIndexShard.postRecovery(InternalIndexShard.java:710)
    at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:223)
    at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
    ... 3 more
Caused by: java.io.FileNotFoundException: No such file [_id.tis]
    at org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
    at org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
    at org.elasticsearch.index.store.DistributorDirectory.openInput(DistributorDirectory.java:130)
    at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
    at org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:487)
    at org.apache.lucene.codecs.lucene3x.TermInfosReader.&lt;init&gt;(TermInfosReader.java:115)
    at org.apache.lucene.codecs.lucene3x.Lucene3xFields.newTermInfosReader(Lucene3xFields.java:142)
    at org.apache.lucene.codecs.lucene3x.Lucene3xFields.&lt;init&gt;(Lucene3xFields.java:88)
    at org.apache.lucene.codecs.lucene3x.Lucene3xPostingsFormat.fieldsProducer(Lucene3xPostingsFormat.java:62)
    at org.apache.lucene.index.SegmentCoreReaders.&lt;init&gt;(SegmentCoreReaders.java:120)
    at org.apache.lucene.index.SegmentReader.&lt;init&gt;(SegmentReader.java:108)
    at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:144)
    at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:238)
    at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:104)
    at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:422)
    at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:112)
    at org.apache.lucene.search.SearcherManager.&lt;init&gt;(SearcherManager.java:89)
    at org.elasticsearch.index.engine.internal.InternalEngine.buildSearchManager(InternalEngine.java:1527)
    at org.elasticsearch.index.engine.internal.InternalEngine.start(InternalEngine.java:309)
    ... 6 more
..."><pre class="notranslate"><code class="notranslate">[2015-01-05 12:42:34,825][INFO ][node                     ] [Druid] version[1.4.2], pid[65351], build[927caff/2014-12-16T14:11:12Z]
[2015-01-05 12:42:34,826][INFO ][node                     ] [Druid] initializing ...
[2015-01-05 12:42:34,830][INFO ][plugins                  ] [Druid] loaded [], sites []
[2015-01-05 12:42:37,395][INFO ][node                     ] [Druid] initialized
[2015-01-05 12:42:37,395][INFO ][node                     ] [Druid] starting ...
[2015-01-05 12:42:37,466][INFO ][transport                ] [Druid] bound_address {inet[/127.0.0.1:9303]}, publish_address {inet[/127.0.0.1:9303]}
[2015-01-05 12:42:37,480][INFO ][discovery                ] [Druid] elasticsearch/ftzmRGfARAuhjPfw9CVE5Q
[2015-01-05 12:42:40,516][INFO ][cluster.service          ] [Druid] detected_master [Roma][BvPgLtdCS8eIEFUATx2lVw][dw1949demum.int.demandware.com][inet[/127.0.0.1:9300]], added {[Roma][BvPgLtdCS8eIEFUATx2lVw][dw1949demum.int.demandware.com][inet[/127.0.0.1:9300]],[Bob Diamond][gkXgdCNvSACjIFiPAVTxXA][dw1949demum.int.demandware.com][inet[/127.0.0.1:9301]],[Phoenix][LTdV5wDIQtuJUPTl9NuuJw][dw1949demum.int.demandware.com][inet[/127.0.0.1:9302]],}, reason: zen-disco-receive(from master [[Roma][BvPgLtdCS8eIEFUATx2lVw][dw1949demum.int.demandware.com][inet[/127.0.0.1:9300]]])
[2015-01-05 12:42:40,526][INFO ][cluster.routing.allocation.decider] [Druid] updating [cluster.routing.allocation.disable_allocation] from [false] to [true]
[2015-01-05 12:42:40,540][INFO ][http                     ] [Druid] bound_address {inet[/127.0.0.1:9203]}, publish_address {inet[/127.0.0.1:9203]}
[2015-01-05 12:42:40,540][INFO ][node                     ] [Druid] started
[2015-01-05 12:56:58,557][INFO ][cluster.routing.allocation.decider] [Druid] updating [cluster.routing.allocation.disable_allocation] from [true] to [false]
[2015-01-05 12:57:09,746][WARN ][indices.cluster          ] [Druid] [index1][1] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [index1][1] failed recovery
    at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:185)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [index1][1] failed to open reader on writer
    at org.elasticsearch.index.engine.internal.InternalEngine.start(InternalEngine.java:321)
    at org.elasticsearch.index.shard.service.InternalIndexShard.postRecovery(InternalIndexShard.java:710)
    at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:223)
    at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
    ... 3 more
Caused by: java.io.FileNotFoundException: No such file [_id.tis]
    at org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
    at org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
    at org.elasticsearch.index.store.DistributorDirectory.openInput(DistributorDirectory.java:130)
    at org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
    at org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:487)
    at org.apache.lucene.codecs.lucene3x.TermInfosReader.&lt;init&gt;(TermInfosReader.java:115)
    at org.apache.lucene.codecs.lucene3x.Lucene3xFields.newTermInfosReader(Lucene3xFields.java:142)
    at org.apache.lucene.codecs.lucene3x.Lucene3xFields.&lt;init&gt;(Lucene3xFields.java:88)
    at org.apache.lucene.codecs.lucene3x.Lucene3xPostingsFormat.fieldsProducer(Lucene3xPostingsFormat.java:62)
    at org.apache.lucene.index.SegmentCoreReaders.&lt;init&gt;(SegmentCoreReaders.java:120)
    at org.apache.lucene.index.SegmentReader.&lt;init&gt;(SegmentReader.java:108)
    at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:144)
    at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:238)
    at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:104)
    at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:422)
    at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:112)
    at org.apache.lucene.search.SearcherManager.&lt;init&gt;(SearcherManager.java:89)
    at org.elasticsearch.index.engine.internal.InternalEngine.buildSearchManager(InternalEngine.java:1527)
    at org.elasticsearch.index.engine.internal.InternalEngine.start(InternalEngine.java:309)
    ... 6 more
...
</code></pre></div>