<p dir="auto">If a cluster is shutdown (all nodes at once), while there is a process running to increase the number of replicas, the index stays in a <code class="notranslate">UNASSIGNED</code> state with no messages in the logs.<br>
If the number of replica is changed again the index comes back.</p>
<p dir="auto">Steps to reproduce:</p>
<ol dir="auto">
<li>spin up a cluster with at least 3 nodes</li>
<li>Index some data (should be enough data for the replicas assignment to take some time)</li>
<li>Change replicas to 0 <code class="notranslate">curl -XPUT 'localhost:9200/_settings' -d '{ "index" : {"number_of_replicas" : 0 }}'</code></li>
<li>Raise number of replicas <code class="notranslate">curl -XPUT 'localhost:9200/_settings' -d '{ "index" : {"number_of_replicas" : 1 }}'</code> and shut down all nodes before the replicas have been fully assigned.</li>
<li>Start all nodes again</li>
<li>The Index will stay in an <code class="notranslate">UNASSIGNED</code> state, but there is nothing in the logs on why this is the case.</li>
<li>Changing the number of replicas to 0 again, will bring back the index into a <code class="notranslate">STARTED</code> state.</li>
</ol>
<p dir="auto">Reproduced on 2.0 and 2.1.</p>
<p dir="auto">Persist allocation IDs of active shards in cluster state and use them to recover correct shards upon cluster restart (i.e., we can recover with only one copy and we make sure we recover the right ones and not a stale copy left around).</p>
<p dir="auto">Relates to <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="116243337" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/14671" data-hovercard-type="issue" data-hovercard-url="/elastic/elasticsearch/issues/14671/hovercard" href="https://github.com/elastic/elasticsearch/issues/14671">#14671</a></p>
<h2 dir="auto">Steps</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Persist allocation ID with shard state metadata on nodes (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="117587084" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/14831" data-hovercard-type="pull_request" data-hovercard-url="/elastic/elasticsearch/pull/14831/hovercard" href="https://github.com/elastic/elasticsearch/pull/14831">#14831</a>).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Persist currently started allocation IDs to index metadata (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="118600739" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/14964" data-hovercard-type="pull_request" data-hovercard-url="/elastic/elasticsearch/pull/14964/hovercard" href="https://github.com/elastic/elasticsearch/pull/14964">#14964</a>).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Add allocation IDs to TransportNodesListGatewayStartedShards action. Use the above to assign a primary shard on recovery. In particular, don't assign a stale copy (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="120785950" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/15281" data-hovercard-type="pull_request" data-hovercard-url="/elastic/elasticsearch/pull/15281/hovercard" href="https://github.com/elastic/elasticsearch/pull/15281">#15281</a>).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Extend reroute with an option to force assign a stale copy. This is only relevant for primaries (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="124329872" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/15708" data-hovercard-type="pull_request" data-hovercard-url="/elastic/elasticsearch/pull/15708/hovercard" href="https://github.com/elastic/elasticsearch/pull/15708">#15708</a>).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Prioritize old primaries when choosing out existing <em>and</em> previously active allocationIds (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="127488612" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/16096" data-hovercard-type="pull_request" data-hovercard-url="/elastic/elasticsearch/pull/16096/hovercard" href="https://github.com/elastic/elasticsearch/pull/16096">#16096</a>).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Persist shard state immediately and not only on shard activation. This will help solving a race condition on primary relocation where the master has activated the target shard (removing old copy from the active set list) but the cluster crashes before the target node persisted the shard state (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="133183768" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/16625" data-hovercard-type="pull_request" data-hovercard-url="/elastic/elasticsearch/pull/16625/hovercard" href="https://github.com/elastic/elasticsearch/pull/16625">#16625</a>)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Remove version from ShardRouting (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="128881686" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/16243" data-hovercard-type="pull_request" data-hovercard-url="/elastic/elasticsearch/pull/16243/hovercard" href="https://github.com/elastic/elasticsearch/pull/16243">#16243</a>).</li>
</ul>