<p dir="auto">I have a 3 node ES cluster. I just upgraded from 0.90.11 to 1.0.1 and started experiencing these exceptions. When I try to access <code class="notranslate">curl 'http://server:9200/_nodes?pretty=true'</code> on any of my nodes I get this exception in ES logs:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[2014-03-06 03:52:23,848][DEBUG][action.admin.cluster.node.info] [logserver3-la] failed to execute on node [iPvGOBIQTuOV_YhNAmLAUg]
org.elasticsearch.transport.RemoteTransportException: Failed to deserialize response of type [org.elasticsearch.action.admin.cluster.node.info.NodeInfo]
Caused by: org.elasticsearch.transport.TransportSerializationException: Failed to deserialize response of type [org.elasticsearch.action.admin.cluster.node.info.NodeInfo]
    at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:148)
    at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:125)
    at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
    at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
    at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
    at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
    at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
    at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)
Caused by: java.lang.IndexOutOfBoundsException: Readable byte limit exceeded: 7711
    at org.elasticsearch.common.netty.buffer.AbstractChannelBuffer.readByte(AbstractChannelBuffer.java:236)
    at org.elasticsearch.transport.netty.ChannelBufferStreamInput.readByte(ChannelBufferStreamInput.java:132)
    at org.elasticsearch.common.io.stream.StreamInput.readString(StreamInput.java:276)
    at org.elasticsearch.common.io.stream.HandlesStreamInput.readString(HandlesStreamInput.java:61)
    at org.elasticsearch.threadpool.ThreadPool$Info.readFrom(ThreadPool.java:597)
    at org.elasticsearch.threadpool.ThreadPoolInfo.readFrom(ThreadPoolInfo.java:65)
    at org.elasticsearch.threadpool.ThreadPoolInfo.readThreadPoolInfo(ThreadPoolInfo.java:55)
    at org.elasticsearch.action.admin.cluster.node.info.NodeInfo.readFrom(NodeInfo.java:224)
    at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:146)
    ... 23 more"><pre class="notranslate"><code class="notranslate">[2014-03-06 03:52:23,848][DEBUG][action.admin.cluster.node.info] [logserver3-la] failed to execute on node [iPvGOBIQTuOV_YhNAmLAUg]
org.elasticsearch.transport.RemoteTransportException: Failed to deserialize response of type [org.elasticsearch.action.admin.cluster.node.info.NodeInfo]
Caused by: org.elasticsearch.transport.TransportSerializationException: Failed to deserialize response of type [org.elasticsearch.action.admin.cluster.node.info.NodeInfo]
    at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:148)
    at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:125)
    at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
    at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
    at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
    at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
    at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
    at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
    at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)
Caused by: java.lang.IndexOutOfBoundsException: Readable byte limit exceeded: 7711
    at org.elasticsearch.common.netty.buffer.AbstractChannelBuffer.readByte(AbstractChannelBuffer.java:236)
    at org.elasticsearch.transport.netty.ChannelBufferStreamInput.readByte(ChannelBufferStreamInput.java:132)
    at org.elasticsearch.common.io.stream.StreamInput.readString(StreamInput.java:276)
    at org.elasticsearch.common.io.stream.HandlesStreamInput.readString(HandlesStreamInput.java:61)
    at org.elasticsearch.threadpool.ThreadPool$Info.readFrom(ThreadPool.java:597)
    at org.elasticsearch.threadpool.ThreadPoolInfo.readFrom(ThreadPoolInfo.java:65)
    at org.elasticsearch.threadpool.ThreadPoolInfo.readThreadPoolInfo(ThreadPoolInfo.java:55)
    at org.elasticsearch.action.admin.cluster.node.info.NodeInfo.readFrom(NodeInfo.java:224)
    at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:146)
    ... 23 more
</code></pre></div>
<p dir="auto">I created a gist of the output of _nodes that I do get here:<br>
<a href="https://gist.github.com/daledude/c6c0fb018d06d1e45a62">https://gist.github.com/daledude/c6c0fb018d06d1e45a62</a></p>
<p dir="auto">The exception in the logs is the same for all nodes. Using ES 1.0.1 and Java HotSpot(TM) 64-Bit Server VM 1.7.0_25 on all nodes.</p>
<p dir="auto">This is my config which is the same for all nodes except the hosts, rack, zone:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="cluster.name: mycluster
node.name: &quot;logserver1-chi&quot;
node.rack: chi1
node.zone: chi
node.master: true
node.data: true

index.number_of_replicas: 0

# cluster discovery
discovery.zen.fd.ping_interval: 15s
discovery.zen.fd.ping_timeout: 60s
discovery.zen.fd.ping_retries: 5
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: [&quot;logserver3-la.domain.com&quot;, &quot;logserver2.domain.com&quot;]
cluster.routing.allocation.awareness.attributes: zone

indices.memory.index_buffer_size: 20%
index.translog.flush_threshold_ops: 50000
indices.fielddata.cache.size: 30%
bootstrap.mlockall: true

threadpool.search.type: fixed
threadpool.search.size: 20
threadpool.search.queue_size: -1

threadpool.index.type: fixed
threadpool.index.size: 60
threadpool.index.queue_size: -1

action.disable_delete_all_indices: false"><pre class="notranslate"><code class="notranslate">cluster.name: mycluster
node.name: "logserver1-chi"
node.rack: chi1
node.zone: chi
node.master: true
node.data: true

index.number_of_replicas: 0

# cluster discovery
discovery.zen.fd.ping_interval: 15s
discovery.zen.fd.ping_timeout: 60s
discovery.zen.fd.ping_retries: 5
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["logserver3-la.domain.com", "logserver2.domain.com"]
cluster.routing.allocation.awareness.attributes: zone

indices.memory.index_buffer_size: 20%
index.translog.flush_threshold_ops: 50000
indices.fielddata.cache.size: 30%
bootstrap.mlockall: true

threadpool.search.type: fixed
threadpool.search.size: 20
threadpool.search.queue_size: -1

threadpool.index.type: fixed
threadpool.index.size: 60
threadpool.index.queue_size: -1

action.disable_delete_all_indices: false
</code></pre></div>
<p dir="auto">This issue exists in both in the 1.1 branch and head.</p>
<p dir="auto">I have set the threadpool.get.queue_size to -1 using the REST API.<br>
I am running the elasticsearch server with assertions enable (-ea).<br>
If then a TransportClient with "sniff=false" connects and tries to send a query it gets a org.elasticsearch.client.transport.NoNodeAvailableException<br>
The underlying exception is:<br>
Caused by: java.lang.AssertionError<br>
at org.elasticsearch.common.io.stream.StreamOutput.writeVLong(StreamOutput.java:176)<br>
at org.elasticsearch.common.io.stream.AdapterStreamOutput.writeVLong(AdapterStreamOutput.java:126)<br>
at org.elasticsearch.common.unit.SizeValue.writeTo(SizeValue.java:211)<br>
at org.elasticsearch.threadpool.ThreadPool$Info.writeTo(ThreadPool.java:643)<br>
at org.elasticsearch.threadpool.ThreadPoolInfo.writeTo(ThreadPoolInfo.java:74)<br>
at org.elasticsearch.action.admin.cluster.node.info.NodeInfo.writeTo(NodeInfo.java:291)<br>
at org.elasticsearch.action.admin.cluster.node.info.NodesInfoResponse.writeTo(NodesInfoResponse.java:68)<br>
at org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:83)<br>
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$TransportHandler$1.onResponse(TransportNodesOperationAction.java:246)<br>
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$TransportHandler$1.onResponse(TransportNodesOperationAction.java:241)<br>
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.finishHim(TransportNodesOperationAction.java:227)<br>
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.onOperation(TransportNodesOperationAction.java:202)<br>
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction.access$900(TransportNodesOperationAction.java:102)<br>
at org.elasticsearch.action.support.nodes.TransportNodesOperationAction$AsyncAction$2.run(TransportNodesOperationAction.java:146)<br>
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)<br>
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)<br>
at java.lang.Thread.run(Thread.java:724)</p>
<p dir="auto">That is the server tries to use OutputStream.writeVLong on a negative number.</p>
<p dir="auto">Please let me know if you need more information.</p>