title: java.io.StreamCorruptedException: invalid internal transport message format
body: <p>I am working on <strong>elasticsearch 0.90.5</strong> on <strong>ubuntu 13.04</strong> in LAN environment. When I start it I get <code>StreamCorruptedException</code>. There are no other instances running and I haven't configured anything else. <a href="http://prayag-waves.blogspot.com/2013/10/first-night-hack-on-elasticsearch.html">Things were good</a> when I did the same thing on my laptop with ubuntu-12.04.</p>
  
  <h3>Full stacktrace is as below : </h3>
  
  <pre><code>prayag@prayag:~$ JVM/elasticsearch-0.90.5/bin/elasticsearch -f
  [2013-10-22 02:01:40,400][INFO ][node                     ] [The Blank] version[0.90.5], pid[10767], build[c8714e8/2013-09-17T12:50:20Z]
  [2013-10-22 02:01:40,400][INFO ][node                     ] [The Blank] initializing ...
  [2013-10-22 02:01:40,414][INFO ][plugins                  ] [The Blank] loaded [], sites []
  [2013-10-22 02:01:43,631][INFO ][node                     ] [The Blank] initialized
  [2013-10-22 02:01:43,632][INFO ][node                     ] [The Blank] starting ...
  [2013-10-22 02:01:43,744][INFO ][transport                ] [The Blank] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.3.87:9300]}
  [2013-10-22 02:01:43,814][WARN ][transport.netty          ] [The Blank] exception caught on transport layer [[id: 0x377d3d6b, /192.168.1.178:53828 =&gt; /192.168.3.87:9300]], closing connection
  java.io.StreamCorruptedException: invalid internal transport message format
      at org.elasticsearch.transport.netty.SizeHeaderFrameDecoder.decode(SizeHeaderFrameDecoder.java:27)
      at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
      at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
      at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
      at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
      at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
      at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
      at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
      at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
      at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
      at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
      at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
      at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
      at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:722)
  [2013-10-22 02:01:43,817][WARN ][transport.netty          ] [The Blank] exception caught on transport layer [[id: 0x377d3d6b, /192.168.1.178:53828 :&gt; /192.168.3.87:9300]], closing connection
  java.io.StreamCorruptedException: invalid internal transport message format
      at org.elasticsearch.transport.netty.SizeHeaderFrameDecoder.decode(SizeHeaderFrameDecoder.java:27)
      at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
      at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:482)
      at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:365)
      at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
      at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
      at org.elasticsearch.common.netty.channel.Channels.fireChannelDisconnected(Channels.java:396)
      at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:361)
      at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:81)
      at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:36)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
      at org.elasticsearch.common.netty.channel.Channels.close(Channels.java:812)
      at org.elasticsearch.common.netty.channel.AbstractChannel.close(AbstractChannel.java:197)
      at org.elasticsearch.transport.netty.NettyTransport.exceptionCaught(NettyTransport.java:501)
      at org.elasticsearch.transport.netty.MessageChannelHandler.exceptionCaught(MessageChannelHandler.java:228)
      at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
      at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377)
      at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
      at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
      at org.elasticsearch.common.netty.channel.Channels.fireExceptionCaught(Channels.java:525)
      at org.elasticsearch.common.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:48)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:566)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
      at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
      at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
      at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
      at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
      at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
      at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
      at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
      at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
      at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
      at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:722)
  [2013-10-22 02:01:45,285][WARN ][transport.netty          ] [The Blank] exception caught on transport layer [[id: 0x7da77a28, /192.168.1.178:53837 =&gt; /192.168.3.87:9300]], closing connection
  java.io.StreamCorruptedException: invalid internal transport message format
      at org.elasticsearch.transport.netty.SizeHeaderFrameDecoder.decode(SizeHeaderFrameDecoder.java:27)
      at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
      at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
      at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
      at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
      at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
      at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
      at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
      at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
      at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
      at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
      at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
      at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
      at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:722)
  [2013-10-22 02:01:45,287][WARN ][transport.netty          ] [The Blank] exception caught on transport layer [[id: 0x7da77a28, /192.168.1.178:53837 :&gt; /192.168.3.87:9300]], closing connection
  java.io.StreamCorruptedException: invalid internal transport message format
      at org.elasticsearch.transport.netty.SizeHeaderFrameDecoder.decode(SizeHeaderFrameDecoder.java:27)
      at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
      at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.cleanup(FrameDecoder.java:482)
      at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.channelDisconnected(FrameDecoder.java:365)
      at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:102)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
      at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
      at org.elasticsearch.common.netty.channel.Channels.fireChannelDisconnected(Channels.java:396)
      at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:361)
      at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:81)
      at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:36)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:574)
      at org.elasticsearch.common.netty.channel.Channels.close(Channels.java:812)
      at org.elasticsearch.common.netty.channel.AbstractChannel.close(AbstractChannel.java:197)
      at org.elasticsearch.transport.netty.NettyTransport.exceptionCaught(NettyTransport.java:501)
      at org.elasticsearch.transport.netty.MessageChannelHandler.exceptionCaught(MessageChannelHandler.java:228)
      at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
      at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:377)
      at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:112)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
      at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
      at org.elasticsearch.common.netty.channel.Channels.fireExceptionCaught(Channels.java:525)
      at org.elasticsearch.common.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:48)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:566)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
      at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
      at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
      at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
      at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
      at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
      at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
      at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
      at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
      at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
      at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
      at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:722)
  [2013-10-22 02:01:46,790][INFO ][cluster.service          ] [The Blank] new_master [The Blank][rQTIOt33TiWyJC5t87_qNw][inet[/192.168.3.87:9300]], reason: zen-disco-join (elected_as_master)
  [2013-10-22 02:01:46,839][INFO ][discovery                ] [The Blank] elasticsearch/rQTIOt33TiWyJC5t87_qNw
  [2013-10-22 02:01:46,895][INFO ][http                     ] [The Blank] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.3.87:9200]}
  [2013-10-22 02:01:46,895][INFO ][node                     ] [The Blank] started
  [2013-10-22 02:01:47,705][INFO ][gateway                  ] [The Blank] recovered [40] indices into cluster_state
  </code></pre>
  
  <p><strong>When I check the cluster status, I find it RED, which means the <a href="http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-health.html">specific shard is not allocated in the cluster.</a></strong></p>
  
  <pre><code>prayag@prayag:~$ curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
  {
    "cluster_name" : "elasticsearch",
    "status" : "red",
    "timed_out" : false,
    "number_of_nodes" : 1,
    "number_of_data_nodes" : 1,
    "active_primary_shards" : 34,
    "active_shards" : 34,
    "relocating_shards" : 0,
    "initializing_shards" : 0,
    "unassigned_shards" : 366
  }
  </code></pre>
  
  <p><strong>I can hit <code>-XGET</code>s.</strong></p>
  
  <pre><code>prayag@prayag:~$ curl -XGET 'http://localhost:9200/?pretty=true'
  {
    "ok" : true,
    "status" : 200,
    "name" : "The Blank",
    "version" : {
      "number" : "0.90.5",
      "build_hash" : "c8714e8e0620b62638f660f6144831792b9dedee",
      "build_timestamp" : "2013-09-17T12:50:20Z",
      "build_snapshot" : false,
      "lucene_version" : "4.4"
    },
    "tagline" : "You Know, for Search"
  }
  </code></pre>
  
  <p>But, with <code>-XPUT</code>s, it battles for about a minute and then throws <code>UnavailableShardsException</code>.</p>
  
  <pre><code>prayag@prayag:~$ curl -XPUT 'http://localhost:9200/movies/movie/1' -d '
  &gt; {
  &gt;     "title": "The Godfather",
  &gt;     "director": "Francis Ford Coppola",
  &gt;     "year": 1972
  &gt; }'
  {"error":"UnavailableShardsException[[movies][2] [2] shardIt, [0] active : Timeout waiting for [1m], request: index {[movies][movie][3], source[\n{\n    \"title\": \"The Godfather\",\n    \"director\": \"Francis Ford Coppola\",\n    \"year\": 1972\n}]}]","status":503}
  </code></pre>
  
  <p>I found similar issue at google-group, <a href="https://groups.google.com/d/msg/elasticsearch/k-TYhXM8dXQ/yzN81x1db4IJ">how to resolve elasticsearch status red</a>, where David Pilato  suggests</p>
  
  <blockquote>
    <p>My opinion is that you created some indexes and some of them with no
    replica. You started more than one node in your LAN. Then, you
    shutdown one node.</p>
    
    <p>ES can not give you a green or yellow health as some of your documents
    (index with no replica) can not be seen by ES.</p>
  </blockquote>
  
  <p>But, I don't think thats true in my case.</p>