<p dir="auto"><strong>Elasticsearch version</strong>: 2.3.5 (using <a href="https://hub.docker.com/_/elasticsearch/" rel="nofollow">https://hub.docker.com/_/elasticsearch/</a>)</p>
<p dir="auto"><strong>Plugins installed</strong>: head, license, cloud-aws, marvel-agent (marvel is disabled on tribe and master-only nodes)</p>
<p dir="auto"><strong>JVM version</strong>: <a href="https://hub.docker.com/_/elasticsearch/" rel="nofollow">https://hub.docker.com/_/elasticsearch/</a></p>
<p dir="auto"><strong>OS version</strong>: <a href="https://hub.docker.com/_/elasticsearch/" rel="nofollow">https://hub.docker.com/_/elasticsearch/</a></p>
<p dir="auto"><strong>Description of the problem including expected versus actual behavior</strong>:<br>
Configure tribe for all possible remote clusters, but only some may be available at a single time.  This configuration uses service discovery and DNS because the IPs will be dynamic. If a cluster is not yet up or DNS fails to resolve a remote entry, elasticsearch exits.  If i configure only the DNS cluster entries that are up at the time of startup, then it appears to work as intended.</p>
<p dir="auto">My assumption is that it would continue retrying at a backoff interval and serving requests for the clusters it can successfully connect to.  I'm not sure if this behavior would also affect an extended WAN outage, but would prefer a similar robust behavior.</p>
<p dir="auto"><strong>Steps to reproduce</strong>:</p>
<ol dir="auto">
<li>Configure tribe nodes with DNS with at least one being unknown host at startup:</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="     &quot;-Des.node.master=false&quot;,
     &quot;-Des.marvel.enabled=false&quot;,
     &quot;-Des.tribe.dc1.cluster.name=dc1-elasticsearch&quot;,
     &quot;-Des.tribe.dc1.discovery.zen.ping.unicast.hosts=master-elasticsearch-dc1&quot;,
     &quot;-Des.tribe.dc2.cluster.name=dc2-elasticsearch&quot;,
     &quot;-Des.tribe.dc2.discovery.zen.ping.unicast.hosts=master-elasticsearch-dc2&quot;,"><pre lang="&quot;-Des.node.data=false&quot;," class="notranslate"><code class="notranslate">     "-Des.node.master=false",
     "-Des.marvel.enabled=false",
     "-Des.tribe.dc1.cluster.name=dc1-elasticsearch",
     "-Des.tribe.dc1.discovery.zen.ping.unicast.hosts=master-elasticsearch-dc1",
     "-Des.tribe.dc2.cluster.name=dc2-elasticsearch",
     "-Des.tribe.dc2.discovery.zen.ping.unicast.hosts=master-elasticsearch-dc2",
</code></pre></div>
<ol dir="auto">
<li>elasticsearch will exit with exception below.</li>
</ol>
<p dir="auto"><strong>Provide logs (if relevant)</strong>:<br>
Exception in thread "main" java.lang.IllegalArgumentException: Failed to resolve address for [master-elasticsearch-dc2]<br>
Likely root cause: java.net.UnknownHostException: master-elasticsearch-dc2: unknown error<br>
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)<br>
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)<br>
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)<br>
at java.net.InetAddress.getAllByName0(InetAddress.java:1276)<br>
at java.net.InetAddress.getAllByName(InetAddress.java:1192)<br>
at java.net.InetAddress.getAllByName(InetAddress.java:1126)<br>
at org.elasticsearch.transport.netty.NettyTransport.parse(NettyTransport.java:733)<br>
at org.elasticsearch.transport.netty.NettyTransport.addressesFromString(NettyTransport.java:685)<br>
at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:424)<br>
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.(UnicastZenPing.java:160)<br>
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)<br>
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)<br>
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)<br>
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)<br>
at &lt;&lt;&gt;&gt;<br>
at org.elasticsearch.node.Node.(Node.java:213)<br>
at org.elasticsearch.tribe.TribeClientNode.(TribeClientNode.java:35)<br>
at org.elasticsearch.tribe.TribeService.(TribeService.java:146)<br>
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)<br>
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)<br>
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)<br>
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)<br>
at &lt;&lt;&gt;&gt;<br>
at org.elasticsearch.node.Node.(Node.java:213)<br>
at org.elasticsearch.node.Node.(Node.java:140)<br>
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:143)<br>
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:178)<br>
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270)<br>
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)</p>
<p dir="auto">Hi folks,</p>
<p dir="auto">there seems to be a regression from <strong>1.7.3</strong> to <strong>2.0.0</strong> regarding the <code class="notranslate">gateway.recover_after_time</code> setting.</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="docker run --name=&quot;elasticsearch&quot; -p 9200:9200 -p 9300:9300 -e &quot;SERVICE_ID=$hostname&quot; -e &quot;SERVICE_NAME=elasticsearch&quot; elasticsearch:2.0.0 -Des.index.number_of_shards=3 -Des.index.number_of_replicas=2 -Des.gateway.recover_after_nodes=2 -Des.gateway.expected_nodes=3 -Des.gateway.recover_after_time=5m -Des.discovery.zen.ping.multicast.enabled=false -Des.discovery.zen.ping.unicast.hosts=&quot;01.elasticsearch,02.elasticsearch,03.elasticsearch&quot; -Des.cluster.name=&quot;logging-test&quot; -Des.network.publish_host=&quot;${PUBLIC_IP}&quot;"><pre class="notranslate">docker run --name=<span class="pl-s"><span class="pl-pds">"</span>elasticsearch<span class="pl-pds">"</span></span> -p 9200:9200 -p 9300:9300 -e <span class="pl-s"><span class="pl-pds">"</span>SERVICE_ID=<span class="pl-smi">$hostname</span><span class="pl-pds">"</span></span> -e <span class="pl-s"><span class="pl-pds">"</span>SERVICE_NAME=elasticsearch<span class="pl-pds">"</span></span> elasticsearch:2.0.0 -Des.index.number_of_shards=3 -Des.index.number_of_replicas=2 -Des.gateway.recover_after_nodes=2 -Des.gateway.expected_nodes=3 -Des.gateway.recover_after_time=5m -Des.discovery.zen.ping.multicast.enabled=false -Des.discovery.zen.ping.unicast.hosts=<span class="pl-s"><span class="pl-pds">"</span>01.elasticsearch,02.elasticsearch,03.elasticsearch<span class="pl-pds">"</span></span> -Des.cluster.name=<span class="pl-s"><span class="pl-pds">"</span>logging-test<span class="pl-pds">"</span></span> -Des.network.publish_host=<span class="pl-s"><span class="pl-pds">"</span><span class="pl-smi">${PUBLIC_IP}</span><span class="pl-pds">"</span></span></pre></div>
<p dir="auto">I run Elasticsearch on three nodes (Docker hosts). The DNS names are not immediatly available. In <strong>1.7.3</strong> the cluster stabilizes after a couple of minutes. In <strong>2.0.0</strong> the container crashes directly.</p>
<div class="highlight highlight-source-java notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="Exception in thread &quot;main&quot; java.lang.IllegalArgumentException: Failed to resolve address for [02.elasticsearch]
Likely root cause: java.net.UnknownHostException: 02.elasticsearch: unknown error
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at org.elasticsearch.transport.netty.NettyTransport.parse(NettyTransport.java:668)
at org.elasticsearch.transport.netty.NettyTransport.addressesFromString(NettyTransport.java:620)
at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:398)
at org.elasticsearch.discovery.zen.ping.unicast.UnicastZenPing.&lt;init&gt;(UnicastZenPing.java:141)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at &lt;&lt;&lt;guice&gt;&gt;&gt;
at org.elasticsearch.node.Node.&lt;init&gt;(Node.java:198)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:145)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:170)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:270)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)"><pre class="notranslate"><span class="pl-smi">Exception</span> <span class="pl-s1">in</span> <span class="pl-s1">thread</span> <span class="pl-s">"main"</span> <span class="pl-smi">java</span>.<span class="pl-smi">lang</span>.<span class="pl-smi">IllegalArgumentException</span>: <span class="pl-s1">Failed</span> <span class="pl-s1">to</span> <span class="pl-s1">resolve</span> <span class="pl-smi">address</span> <span class="pl-s1">for</span> [<span class="pl-c1">02.</span><span class="pl-s1">elasticsearch</span>]
<span class="pl-smi">Likely</span> <span class="pl-s1">root</span> <span class="pl-s1">cause</span>: <span class="pl-smi">java</span>.<span class="pl-smi">net</span>.<span class="pl-smi">UnknownHostException</span>: <span class="pl-c1">02.</span><span class="pl-s1">elasticsearch</span>: <span class="pl-smi">unknown</span> <span class="pl-s1">error</span>
<span class="pl-smi">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">net</span>.<span class="pl-s1">Inet6AddressImpl</span>.<span class="pl-s1">lookupAllHostAddr</span>(<span class="pl-smi">Native</span> <span class="pl-s1">Method</span>)
<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">net</span>.<span class="pl-s1">InetAddress$2</span>.<span class="pl-en">lookupAllHostAddr</span>(<span class="pl-smi">InetAddress</span>.<span class="pl-smi">java</span>:<span class="pl-c1">928</span>)
<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">net</span>.<span class="pl-s1">InetAddress</span>.<span class="pl-en">getAddressesFromNameService</span>(<span class="pl-smi">InetAddress</span>.<span class="pl-smi">java</span>:<span class="pl-c1">1323</span>)
<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">net</span>.<span class="pl-s1">InetAddress</span>.<span class="pl-en">getAllByName0</span>(<span class="pl-smi">InetAddress</span>.<span class="pl-smi">java</span>:<span class="pl-c1">1276</span>)
<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">net</span>.<span class="pl-s1">InetAddress</span>.<span class="pl-en">getAllByName</span>(<span class="pl-smi">InetAddress</span>.<span class="pl-smi">java</span>:<span class="pl-c1">1192</span>)
<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">net</span>.<span class="pl-s1">InetAddress</span>.<span class="pl-en">getAllByName</span>(<span class="pl-smi">InetAddress</span>.<span class="pl-smi">java</span>:<span class="pl-c1">1126</span>)
<span class="pl-s1">at</span> <span class="pl-s1">org</span>.<span class="pl-s1">elasticsearch</span>.<span class="pl-s1">transport</span>.<span class="pl-s1">netty</span>.<span class="pl-s1">NettyTransport</span>.<span class="pl-en">parse</span>(<span class="pl-smi">NettyTransport</span>.<span class="pl-smi">java</span>:<span class="pl-c1">668</span>)
<span class="pl-s1">at</span> <span class="pl-s1">org</span>.<span class="pl-s1">elasticsearch</span>.<span class="pl-s1">transport</span>.<span class="pl-s1">netty</span>.<span class="pl-s1">NettyTransport</span>.<span class="pl-en">addressesFromString</span>(<span class="pl-smi">NettyTransport</span>.<span class="pl-smi">java</span>:<span class="pl-c1">620</span>)
<span class="pl-s1">at</span> <span class="pl-s1">org</span>.<span class="pl-s1">elasticsearch</span>.<span class="pl-s1">transport</span>.<span class="pl-s1">TransportService</span>.<span class="pl-en">addressesFromString</span>(<span class="pl-smi">TransportService</span>.<span class="pl-smi">java</span>:<span class="pl-c1">398</span>)
<span class="pl-s1">at</span> <span class="pl-s1">org</span>.<span class="pl-s1">elasticsearch</span>.<span class="pl-s1">discovery</span>.<span class="pl-s1">zen</span>.<span class="pl-s1">ping</span>.<span class="pl-s1">unicast</span>.<span class="pl-s1">UnicastZenPing</span>.&lt;<span class="pl-smi">init</span>&gt;(<span class="pl-smi">UnicastZenPing</span>.<span class="pl-smi">java</span>:<span class="pl-c1">141</span>)
<span class="pl-smi">at</span> <span class="pl-s1">sun</span>.<span class="pl-s1">reflect</span>.<span class="pl-s1">NativeConstructorAccessorImpl</span>.<span class="pl-en">newInstance0</span>(<span class="pl-smi">Native</span> <span class="pl-s1">Method</span>)
<span class="pl-s1">at</span> <span class="pl-s1">sun</span>.<span class="pl-s1">reflect</span>.<span class="pl-s1">NativeConstructorAccessorImpl</span>.<span class="pl-en">newInstance</span>(<span class="pl-smi">NativeConstructorAccessorImpl</span>.<span class="pl-smi">java</span>:<span class="pl-c1">62</span>)
<span class="pl-s1">at</span> <span class="pl-s1">sun</span>.<span class="pl-s1">reflect</span>.<span class="pl-s1">DelegatingConstructorAccessorImpl</span>.<span class="pl-en">newInstance</span>(<span class="pl-smi">DelegatingConstructorAccessorImpl</span>.<span class="pl-smi">java</span>:<span class="pl-c1">45</span>)
<span class="pl-s1">at</span> <span class="pl-s1">java</span>.<span class="pl-s1">lang</span>.<span class="pl-s1">reflect</span>.<span class="pl-s1">Constructor</span>.<span class="pl-en">newInstance</span>(<span class="pl-smi">Constructor</span>.<span class="pl-smi">java</span>:<span class="pl-c1">422</span>)
<span class="pl-s1">at</span> &lt;&lt;&lt;<span class="pl-s1">guice</span>&gt;&gt;&gt;
<span class="pl-smi">at</span> <span class="pl-s1">org</span>.<span class="pl-s1">elasticsearch</span>.<span class="pl-s1">node</span>.<span class="pl-s1">Node</span>.&lt;<span class="pl-smi">init</span>&gt;(<span class="pl-smi">Node</span>.<span class="pl-smi">java</span>:<span class="pl-c1">198</span>)
<span class="pl-smi">at</span> <span class="pl-s1">org</span>.<span class="pl-s1">elasticsearch</span>.<span class="pl-s1">node</span>.<span class="pl-s1">NodeBuilder</span>.<span class="pl-en">build</span>(<span class="pl-smi">NodeBuilder</span>.<span class="pl-smi">java</span>:<span class="pl-c1">145</span>)
<span class="pl-s1">at</span> <span class="pl-s1">org</span>.<span class="pl-s1">elasticsearch</span>.<span class="pl-s1">bootstrap</span>.<span class="pl-s1">Bootstrap</span>.<span class="pl-en">setup</span>(<span class="pl-smi">Bootstrap</span>.<span class="pl-smi">java</span>:<span class="pl-c1">170</span>)
<span class="pl-s1">at</span> <span class="pl-s1">org</span>.<span class="pl-s1">elasticsearch</span>.<span class="pl-s1">bootstrap</span>.<span class="pl-s1">Bootstrap</span>.<span class="pl-en">init</span>(<span class="pl-smi">Bootstrap</span>.<span class="pl-smi">java</span>:<span class="pl-c1">270</span>)
<span class="pl-s1">at</span> <span class="pl-s1">org</span>.<span class="pl-s1">elasticsearch</span>.<span class="pl-s1">bootstrap</span>.<span class="pl-s1">Elasticsearch</span>.<span class="pl-en">main</span>(<span class="pl-smi">Elasticsearch</span>.<span class="pl-smi">java</span>:<span class="pl-c1">35</span>)</pre></div>