anchor.diff:  {GIT}8688[#8688] (STATUS: DONE)
   
   [float]
  -=== Loss of documents during network partition (STATUS: ONGOING)
  -
  -If a network partition separates a node from the master, there is some window of time before the node detects it. The length of the window is dependent on the type of the partition. This window is extremely small if a socket is broken. More adversarial partitions, for example, silently dropping requests without breaking the socket can take longer (up to 3x30s using current defaults).
  -
  -If the node hosts a primary shard at the moment of partition, and ends up being isolated from the cluster (which could have resulted in {GIT}2488[split-brain] before), some documents that are being indexed into the primary may be lost if they fail to reach one of the allocated replicas (due to the partition) and that replica is later promoted to primary by the master. {GIT}7572[#7572]
  -
  -A test to replicate this condition was added in {GIT}7493[#7493].
  -
  -[float]
  -=== Lucene checksums phase 3 (STATUS:ONGOING)
  +=== Add per-segment and per-commit ID to help replication (STATUS: DONE, v2.0.0)
  +
  +{JIRA}5895[LUCENE-5895] adds a unique ID for each segment and each commit point. File-based replication (as performed by snapshot/restore) can use this ID to know whether the segment/commit on the source and destination machines are the same.  Fixed in Lucene 5.0.
  +
  +[float]
  +=== Write index metadata on data nodes where shards allocated (STATUS: DONE, v2.0.0)
  +
  +Today, index metadata is written only on nodes that are master-eligible, not on
  +data-only nodes.  This is not a problem when running with multiple master nodes,
  +as recommended, as the loss of all but one master node is still recoverable.
  +However, users running with a single master node are at risk of losing
  +their index metadata if the master fails.  Instead, this metadata should
  +also be written on any node where a shard is allocated. {GIT}8823[#8823], {GIT}9952[#9952]
  +
  +[float]
  +=== Better file distribution with multiple data paths (STATUS: DONE, v2.0.0)
  +
  +Today, a node configured with multiple data paths distributes writes across
  +all paths by writing one file to each path in turn.  This can mean that the
  +failure of a single disk corrupts many shards at once.  Instead, by allocating
  +an entire shard to a single data path, the extent of the damage can be limited
  +to just the shards on that disk. {GIT}9498[#9498]
  +
  +[float]
  +=== Lucene checksums phase 3 (STATUS: DONE, v2.0.0)
   
   Almost all files in Elasticsearch now have checksums which are validated before use.  A few changes remain:
   
   * {GIT}7586[#7586] adds checksums for cluster and index state files. (STATUS: DONE, Fixed in v1.5.0)
   * {GIT}9183[#9183] supports validating the checksums on all files when starting a node. (STATUS: DONE, Fixed in v2.0.0)
   * {JIRA}5894[LUCENE-5894] lays the groundwork for extending more efficient checksum validation to all files during optimized bulk merges. (STATUS: DONE, Fixed in v2.0.0)
  -* {GIT}8403[#8403] to add validation of checksums on Lucene `segments_N` files. (STATUS: NOT STARTED)
  -
  -[float]
  -=== Add per-segment and per-commit ID to help replication (STATUS: ONGOING)
  -
  -{JIRA}5895[LUCENE-5895] adds a unique ID for each segment and each commit point. File-based replication (as performed by snapshot/restore) can use this ID to know whether the segment/commit on the source and destination machines are the same.  Fixed in Lucene 5.0.
  -
  -[float]
  -=== Report shard-level statuses on write operations (STATUS: ONGOING)
  +* {GIT}8403[#8403] to add validation of checksums on Lucene `segments_N` files. (STATUS: DONE, v2.0.0)
  +
  +[float]
  +=== Report shard-level statuses on write operations (STATUS: DONE, v2.0.0)
   
   Make write calls return the number of total/successful/missing shards in the same way that we do in search, which ensures transparency in the consistency of write operations. {GIT}7994[#7994]. (STATUS: DONE, v2.0.0)
   
   [float]
  -=== Jepsen Test Failures (STATUS: ONGOING)
  -
  -We have increased our test coverage to include scenarios tested by Jepsen. We make heavy use of randomization to expand on the scenarios that can be tested and to introduce new error conditions. You can follow the work on the master branch of the https://github.com/elastic/elasticsearch/blob/master/core/src/test/java/org/elasticsearch/discovery/DiscoveryWithServiceDisruptionsIT.java[`DiscoveryWithServiceDisruptionsIT` class], where we will add more tests as time progresses.
  -
  -[float]
  -=== Document guarantees and handling of failure (STATUS: ONGOING)
  -
  -This status page is a start, but we can do a better job of explicitly documenting the processes at work in Elasticsearch, and what happens in the case of each type of failure. The plan is to have a test case that validates each behavior under simulated conditions. Every test will document the expected results, the associated test code and an explicit PASS or FAIL status for each simulated case.
  -
  -
  -[float]
  -=== Take filter cache key size into account (STATUS: ONGOING)
  -
  -Commonly used filters are cached in Elasticsearch. That cache is limited in size (10% of node's memory by default) and is being evicted based on a least recently used policy. The amount of memory used by the cache depends on two primary components - the values it stores and the keys associated with them. Calculating the memory footprint of the values is easy enough but the keys accounting is trickier to achieve as they are, by default, raw Lucene objects. This is largely not a problem as the keys are dominated by the values. However, recent optimizations in Lucene have changed the balance causing the filter cache to grow beyond it's size.
  -
  -While we are working on a longer term solution ({GIT}9176[#9176]), we introduced a minimum weight of 1k for each cache entry. This puts an effective limit on the number of entries in the cache. See {GIT}8304[#8304] (STATUS: DONE, fixed in v1.4.0)
  -
  -[float]
  -=== Do not allow stale shards to automatically be promoted to primary (STATUS: ONGOING)
  -
  -In some scenarios, after loss of all valid copies, a stale replica shard can be assigned as a primary. This can lead to
  -a loss of acknowledged writes if the valid copies are not lost but are rather temporarily isolated. Work is underway
  -({GIT}14671[#14671]) to prevent the automatic promotion of a stale primary and only allow such promotion to occur when
  -a system operator manually intervenes.
  -
  -== Completed
  +=== Take filter cache key size into account (STATUS: DONE, v2.0.0)
  +
  +Commonly used filters are cached in Elasticsearch. That cache is limited in size
  +(10% of node's memory by default) and is being evicted based on a least recently
  +used policy. The amount of memory used by the cache depends on two primary
  +components - the values it stores and the keys associated with them. Calculating
  +the memory footprint of the values is easy enough but the keys accounting is
  +trickier to achieve as they are, by default, raw Lucene objects. This is largely
  +not a problem as the keys are dominated by the values. However, recent
  +optimizations in Lucene have changed the balance causing the filter cache to
  +grow beyond it's size.
  +
  +While we are working on a longer term solution ({GIT}9176[#9176]), we introduced
  +a minimum weight of 1k for each cache entry. This puts an effective limit on the number of entries in the cache. See {GIT}8304[#8304] (STATUS: DONE, fixed in v1.4.0)
  +
  +Note: this has been solved by the move to Lucene's query cache. See {GIT}10897[#10897]
   
   [float]
   === Ensure shard state ID is incremental (STATUS: DONE, v1.5.1)
  
positive.code: nodes have sent their joins request (based on the `minimum_master_nodes` settings) the cluster state is updated.
  {GIT}12161[#12161]
  
  
  [float]
  === Write index metadata on data nodes where shards allocated (STATUS: ONGOING)
  
  Today, index metadata is written only on nodes that are master-eligible, not on
  data-only nodes.  This is not a problem when running with multiple master nodes,
  as recommended, as the loss of all but one master node is still recoverable.
  However, users running with a single master node are at risk of losing
  their index metadata if the master fails.  Instead, this metadata should
  also be written on any node where a shard is allocated. {GIT}8823[#8823]
  
  [float]
  === Better file distribution with multiple data paths (STATUS: ONGOING)
  
  Today, a node configured with multiple data paths distributes writes across
  all paths by writing one file to each path in turn.  This can mean that the
  failure of a single disk corrupts many shards at once.  Instead, by allocating
  an entire shard to a single data path, the extent of the damage can be limited
  to just the shards on that disk. {GIT}9498[#9498]
  
  [float]
  === OOM resiliency (STATUS: ONGOING)
  
  The family of circuit breakers has greatly reduced the occurrence of OOM
  exceptions, but it is still possible to cause a node to run out of heap
  space.  The following issues have been identified:
  
  * Set a hard limit on `from`/`size` parameters {GIT}9311[#9311]. (STATUS: ONGOING)
  * Prevent combinatorial explosion in aggregations from causing OOM {GIT}8081[#8081]. (STATUS: ONGOING)
  * Add the byte size of each hit to the request circuit breaker {GIT}9310[#9310]. (STATUS: ONGOING)
  
  [float]
  === Mapping changes should be applied synchronously (STATUS: ONGOING)
  
  When introducing new fields using dynamic mapping, it is possible that the same
  field can be added to different shards with different data types.  Each shard
  
negative.code: See {GIT}9967[#9967]. (STATUS: ONGOING)
  
  [float]
  === OOM resiliency (STATUS: ONGOING)
  
  The family of circuit breakers has greatly reduced the occurrence of OOM
  exceptions, but it is still possible to cause a node to run out of heap
  space.  The following issues have been identified:
  
  * Set a hard limit on `from`/`size` parameters {GIT}9311[#9311]. (STATUS: ONGOING)
  * Prevent combinatorial explosion in aggregations from causing OOM {GIT}8081[#8081]. (STATUS: ONGOING)
  * Add the byte size of each hit to the request circuit breaker {GIT}9310[#9310]. (STATUS: ONGOING)
  
  [float]
  === Loss of documents during network partition (STATUS: ONGOING)
  
  If a network partition separates a node from the master, there is some window of time before the node detects it. The length of the window is dependent on the type of the partition. This window is extremely small if a socket is broken. More adversarial partitions, for example, silently dropping requests without breaking the socket can take longer (up to 3x30s using current defaults).
  
  If the node hosts a primary shard at the moment of partition, and ends up being isolated from the cluster (which could have resulted in {GIT}2488[split-brain] before), some documents that are being indexed into the primary may be lost if they fail to reach one of the allocated replicas (due to the partition) and that replica is later promoted to primary by the master. {GIT}7572[#7572]
  
  A test to replicate this condition was added in {GIT}7493[#7493].
  
  [float]
  === Jepsen Test Failures (STATUS: ONGOING)
  
  We have increased our test coverage to include scenarios tested by Jepsen. We make heavy use of randomization to expand on the scenarios that can be tested and to introduce new error conditions. You can follow the work on the master branch of the https://github.com/elastic/elasticsearch/blob/master/core/src/test/java/org/elasticsearch/discovery/DiscoveryWithServiceDisruptionsIT.java[`DiscoveryWithServiceDisruptionsIT` class], where we will add more tests as time progresses.
  
  [float]
  === Document guarantees and handling of failure (STATUS: ONGOING)
  
  This status page is a start, but we can do a better job of explicitly documenting the processes at work in Elasticsearch, and what happens in the case of each type of failure. The plan is to have a test case that validates each behavior under simulated conditions. Every test will document the expected results, the associated test code and an explicit PASS or FAIL status for each simulated case.
  
  [float]
  === Do not allow stale shards to automatically be promoted to primary (STATUS: ONGOING)
  
  In some scenarios, after loss of all valid copies, a stale replica shard can be assigned as a primary. This can lead to
  a loss of acknowledged writes if the valid copies are not lost but are rather temporarily isolated. Work is underway
  ({GIT}14671[#14671]) to prevent the automatic promotion of a stale primary and only allow such promotion to occur when
  a system operator manually intervenes.
  
  == Completed
  
  [float]
  === Wait on incoming joins before electing local node as master (STATUS: DONE, v2.0.0)
  
  During master election each node pings in order to discover other nodes and validate the liveness of existing
  nodes. Based on this information the node either discovers an existing master or, if enough nodes are found
  