<p dir="auto">This is tough for me to reliably reproduce, but I've seen it a half-dozen times and think it needs an issue. Elasticsearch can still lose inserted documents when a single primary node is isolated by a simple network partition. Nothing fancy, no overlapping partitions required. Sometimes it's only a few documents, from immediately following the partition:</p>
<div class="highlight highlight-source-clojure notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{:valid? false,
 :lost &quot;#{640 643 645 649}&quot;,
 :recovered
 &quot;#{140..142 144 146 148..152 155 157 159..162 164..168 171 174..175 177 179..185 187 190..192 194 197 199..200 202 205..208 214 224 234 238 240..247 249..254 256..264 266..282 285 296 317 342 363 388 431}&quot;,
 :ok
 &quot;#{0..137 140..142 144 146 148..152 155 157 159..162 164..168 171 174..175 177 179..185 187 190..192 194 197 199..200 202 205..208 210..212 214 224 234 238 240..247 249..254 256..264 266..639 641..642 644 646..648 650..654}&quot;,
 :recovered-frac 96/655,
 :unexpected-frac 0,
 :unexpected &quot;#{}&quot;,
 :lost-frac 4/655,
 :ok-frac 598/655}"><pre class="notranslate">{<span class="pl-c1">:valid?</span> <span class="pl-c1">false</span>,
 <span class="pl-c1">:lost</span> <span class="pl-s"><span class="pl-pds">"</span>#{640 643 645 649}<span class="pl-pds">"</span></span>,
 <span class="pl-c1">:recovered</span>
 <span class="pl-s"><span class="pl-pds">"</span>#{140..142 144 146 148..152 155 157 159..162 164..168 171 174..175 177 179..185 187 190..192 194 197 199..200 202 205..208 214 224 234 238 240..247 249..254 256..264 266..282 285 296 317 342 363 388 431}<span class="pl-pds">"</span></span>,
 <span class="pl-c1">:ok</span>
 <span class="pl-s"><span class="pl-pds">"</span>#{0..137 140..142 144 146 148..152 155 157 159..162 164..168 171 174..175 177 179..185 187 190..192 194 197 199..200 202 205..208 210..212 214 224 234 238 240..247 249..254 256..264 266..639 641..642 644 646..648 650..654}<span class="pl-pds">"</span></span>,
 <span class="pl-c1">:recovered-frac</span> <span class="pl-c1">96/655</span>,
 <span class="pl-c1">:unexpected-frac</span> <span class="pl-c1">0</span>,
 <span class="pl-c1">:unexpected</span> <span class="pl-s"><span class="pl-pds">"</span>#{}<span class="pl-pds">"</span></span>,
 <span class="pl-c1">:lost-frac</span> <span class="pl-c1">4/655</span>,
 <span class="pl-c1">:ok-frac</span> <span class="pl-c1">598/655</span>}</pre></div>
<p dir="auto">Other times, the cluster seems to get really confused, and loses over half of the acknowledged records, distributed throughout the history:</p>
<div class="highlight highlight-source-clojure notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{:valid? false,
 :lost
 &quot;#{53..55 126..129 131..142 144..167 169..189 191..212 214..235 237..260 262..281 283..305 307..327 329..352 354..367 374 376}&quot;,
 :recovered
 &quot;#{46..47 49 52 60 428..430 432..435 437..440 442..445 447..450 452..455 457..473}&quot;,
 :ok
 &quot;#{0..47 49 52 60 428..430 432..435 437..440 442..445 447..450 452..455 457..473}&quot;,
 :recovered-frac 15/158,
 :unexpected-frac 0,
 :unexpected &quot;#{}&quot;,
 :lost-frac 118/237,
 :ok-frac 91/474}"><pre class="notranslate">{<span class="pl-c1">:valid?</span> <span class="pl-c1">false</span>,
 <span class="pl-c1">:lost</span>
 <span class="pl-s"><span class="pl-pds">"</span>#{53..55 126..129 131..142 144..167 169..189 191..212 214..235 237..260 262..281 283..305 307..327 329..352 354..367 374 376}<span class="pl-pds">"</span></span>,
 <span class="pl-c1">:recovered</span>
 <span class="pl-s"><span class="pl-pds">"</span>#{46..47 49 52 60 428..430 432..435 437..440 442..445 447..450 452..455 457..473}<span class="pl-pds">"</span></span>,
 <span class="pl-c1">:ok</span>
 <span class="pl-s"><span class="pl-pds">"</span>#{0..47 49 52 60 428..430 432..435 437..440 442..445 447..450 452..455 457..473}<span class="pl-pds">"</span></span>,
 <span class="pl-c1">:recovered-frac</span> <span class="pl-c1">15/158</span>,
 <span class="pl-c1">:unexpected-frac</span> <span class="pl-c1">0</span>,
 <span class="pl-c1">:unexpected</span> <span class="pl-s"><span class="pl-pds">"</span>#{}<span class="pl-pds">"</span></span>,
 <span class="pl-c1">:lost-frac</span> <span class="pl-c1">118/237</span>,
 <span class="pl-c1">:ok-frac</span> <span class="pl-c1">91/474</span>}</pre></div>
<p dir="auto">The logs aren't particularly informative here, but I have noticed some interesting messages which seem correlated with data loss: for instance:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[2015-04-02][WARN ][index.shard] [n4] [jepsen-index][0] suspect illegal state: trying to move shard from primary mode to replica mode"><pre class="notranslate"><code class="notranslate">[2015-04-02][WARN ][index.shard] [n4] [jepsen-index][0] suspect illegal state: trying to move shard from primary mode to replica mode
</code></pre></div>
<p dir="auto">and</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[2015-04-02 17:19:54,370][DEBUG][action.index             ] [n2] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]"><pre class="notranslate"><code class="notranslate">[2015-04-02 17:19:54,370][DEBUG][action.index             ] [n2] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
</code></pre></div>
<p dir="auto">which I think is a hardcoded message.</p>
<p dir="auto">Again, these are tough to reproduce--only 1 in 10 runs or so--so I'll keep working on finding a failure schedule that reliably triggers the fault.</p>
<p dir="auto">This ticket is meant to capture an issue which was discovered as part of the work done in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="41361093" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/7493" data-hovercard-type="pull_request" data-hovercard-url="/elastic/elasticsearch/pull/7493/hovercard" href="https://github.com/elastic/elasticsearch/pull/7493">#7493</a> , which contains a <a href="https://github.com/elasticsearch/elasticsearch/blob/596a4a073584c4262d574828c9caea35b5ed1de5/src/test/java/org/elasticsearch/discovery/DiscoveryWithServiceDisruptions.java#L375">failing reproduction test</a> with <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/awaitFix/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/awaitFix">@awaitFix</a>.</p>
<p dir="auto">If a network partition separates a node from the master, there is some window of time before the node detects it. The length of the window is dependent on the type of the partition. This window is extremely small if a socket is broken. More adversarial partitions, for example, silently dropping requests without breaking the socket can take longer (up to 3x30s using current defaults).</p>
<p dir="auto">If the node hosts a <em>primary</em> shard at the moment of partition, and ends up being isolated from the cluster (which could have resulted in Split Brain before), some documents that are being indexed into the primary <em>may</em> be lost if they fail to reach one of the allocated replicas  (due to the partition) and that replica is later promoted to primary by the master.</p>