<p dir="auto">After a 1-way network partition scenario, two nodes might be stuck in different opinions of which nodes are currently in the cluster.</p>
<p dir="auto">I.e. with 5 nodes:<br>
Node 1 sees node 1, 3 and 5. This is a RED cluster.<br>
Node 2 sees node 1, 2, 4 and 5. This is a GREEN cluster.<br>
All of this according to the <code class="notranslate">nodes</code> endpoint.</p>
<p dir="auto">The cluster is <em>stuck in this state</em>, even when every node could reach every other node (verified with curl on the es http port and binary port).</p>
<p dir="auto">Some of these nodes give an NPE for the <code class="notranslate">_status</code> endpoint:<br>
<code class="notranslate">{"error":"NullPointerException[null]","status":500}</code></p>
<p dir="auto">All nodes are configured to discover all the others through unicast.<br>
Discovered on ES 0.90.13 on RHEL 6.5.</p>
<p dir="auto">G'day,</p>
<p dir="auto">I'm using ElasticSearch 0.19.11 with the unicast Zen discovery protocol.</p>
<p dir="auto">With this setup, I can easily split a 3-node cluster into two 'hemispheres' (continuing with the brain metaphor) with one node acting as a participant in both hemispheres.  I believe this to be a significant problem, because now <code class="notranslate">minimum_master_nodes</code> is incapable of preventing certain split-brain scenarios.</p>
<p dir="auto">Here's what my 3-node test cluster looked like before I broke it:</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/417a448b9b397878cabc3289a1946f04aa9120ae6be3adefa99bf5e97e75136a/68747470733a2f2f73616a2e626574612e616e63686f7274726f76652e636f6d2f65732d73706c6974627261696e2d312e706e67"><img src="https://camo.githubusercontent.com/417a448b9b397878cabc3289a1946f04aa9120ae6be3adefa99bf5e97e75136a/68747470733a2f2f73616a2e626574612e616e63686f7274726f76652e636f6d2f65732d73706c6974627261696e2d312e706e67" alt="" data-canonical-src="https://saj.beta.anchortrove.com/es-splitbrain-1.png" style="max-width: 100%;"></a></p>
<p dir="auto">Here's what the cluster looked like after simulating a communications failure between nodes (2) and (3):</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/e565d01251307c275f8f677cc4455f0d820aed3d1b77dd1dca9cb82ddd611e6a/68747470733a2f2f73616a2e626574612e616e63686f7274726f76652e636f6d2f65732d73706c6974627261696e2d322e706e67"><img src="https://camo.githubusercontent.com/e565d01251307c275f8f677cc4455f0d820aed3d1b77dd1dca9cb82ddd611e6a/68747470733a2f2f73616a2e626574612e616e63686f7274726f76652e636f6d2f65732d73706c6974627261696e2d322e706e67" alt="" data-canonical-src="https://saj.beta.anchortrove.com/es-splitbrain-2.png" style="max-width: 100%;"></a></p>
<p dir="auto">Here's what seems to have happened immediately after the split:</p>
<ol dir="auto">
<li>Node (2) and (3) lose contact with one another.  (<code class="notranslate">zen-disco-node_failed</code> ... <code class="notranslate">reason failed to ping</code>)</li>
<li>Node (2), still master of the left hemisphere, notes the disappearance of node (3) and broadcasts an advisory message to all of its followers.  Node (1) takes note of the advisory.</li>
<li>Node (3) has now lost contact with its old master and decides to hold an election.  It declares itself winner of the election.  On declaring itself, it assumes master role of the right hemisphere, then broadcasts an advisory message to all of its followers.  Node (1) takes note of this advisory, too.</li>
</ol>
<p dir="auto">At this point, I can't say I know what to expect to find on node (1).  If I query both masters for a list of nodes, I see node (1) in both clusters.</p>
<p dir="auto">Let's look at <code class="notranslate">minimum_master_nodes</code> as it applies to this test cluster.  Assume I had set <code class="notranslate">minimum_master_nodes</code> to 2.  Had node (3) been completely isolated from nodes (1) and (2), I would not have run into this problem.  The left hemisphere would have enough nodes to satisfy the constraint; the right hemisphere would not.  This would continue to work for larger clusters (with an appropriately larger value for <code class="notranslate">minimum_master_nodes</code>).</p>
<p dir="auto">The problem with <code class="notranslate">minimum_master_nodes</code> is that it does not work when the split brains are intersecting, as in my example above.  Even on a larger cluster of, say, 7 nodes with <code class="notranslate">minimum_master_nodes</code> set to 4, all that needs to happen is for the 'right' two nodes to lose contact with one another (a master election has to take place) for the cluster to split.</p>
<p dir="auto">Is there anything that can be done to detect the intersecting split on node (1)?</p>
<p dir="auto">Would <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1101739" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/1057" data-hovercard-type="pull_request" data-hovercard-url="/elastic/elasticsearch/pull/1057/hovercard" href="https://github.com/elastic/elasticsearch/pull/1057">#1057</a> help?</p>
<p dir="auto">Am I missing something obvious? :)</p>