<p dir="auto">When searching trough a few documents (1206 in that case) in an index (updated with deletes, inserts, updates from time to time), I got some duplicates or not depending on the sorting I supply.</p>
<p dir="auto"><strong>Elasticsearch version</strong>:<br>
2.1.0</p>
<p dir="auto"><strong>JVM version</strong>:<br>
openjdk version "1.8.0_66-internal"<br>
OpenJDK Runtime Environment (build 1.8.0_66-internal-b17)<br>
OpenJDK 64-Bit Server VM (build 25.66-b17, mixed mode)</p>
<p dir="auto"><strong>OS version</strong>:<br>
Linux CBISES02 3.16.0-4-amd64 <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="129156" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/1" data-hovercard-type="issue" data-hovercard-url="/elastic/elasticsearch/issues/1/hovercard" href="https://github.com/elastic/elasticsearch/issues/1">#1</a> SMP Debian 3.16.7-ckt11-1+deb8u6 (2015-11-09) x86_64 GNU/Linux</p>
<p dir="auto"><strong>Description of the problem including expected versus actual behavior</strong>:<br>
I have an index with the following mapping:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{  
   &quot;state&quot;:&quot;open&quot;,
   &quot;settings&quot;:{  
      &quot;index&quot;:{  
         &quot;creation_date&quot;:&quot;1466088694234&quot;,
         &quot;number_of_shards&quot;:&quot;10&quot;,
         &quot;number_of_replicas&quot;:&quot;1&quot;,
         &quot;uuid&quot;:&quot;4tiKdB_6RWe-x2LoxDGOFg&quot;,
         &quot;version&quot;:{  
            &quot;created&quot;:&quot;2010099&quot;
         }
      }
   },
   &quot;mappings&quot;:{  
      &quot;product&quot;:{  
         &quot;_routing&quot;:{  
            &quot;required&quot;:true
         },
         &quot;dynamic&quot;:&quot;false&quot;,
         &quot;_all&quot;:{  
            &quot;enabled&quot;:false
         },
         &quot;properties&quot;:{  
            &quot;id&quot;:{  
               &quot;index&quot;:&quot;not_analyzed&quot;,
               &quot;type&quot;:&quot;string&quot;
            },
            &quot;entityId&quot;:{  
               &quot;index&quot;:&quot;not_analyzed&quot;,
               &quot;type&quot;:&quot;string&quot;
            },
            &quot;categories&quot;:{  
               &quot;type&quot;:&quot;integer&quot;
            },
            &quot;name&quot;:{  
               &quot;type&quot;:&quot;string&quot;,
               &quot;fields&quot;:{  
                  &quot;raw&quot;:{  
                     &quot;index&quot;:&quot;not_analyzed&quot;,
                     &quot;type&quot;:&quot;string&quot;
                  }
               }
            },
            &quot;routingKey&quot;:{  
               &quot;index&quot;:&quot;not_analyzed&quot;,
               &quot;type&quot;:&quot;string&quot;
            }
            (...)
         }
      }
   },
   &quot;aliases&quot;:[  
      (.. some aliases)
   }"><pre class="notranslate"><code class="notranslate">{  
   "state":"open",
   "settings":{  
      "index":{  
         "creation_date":"1466088694234",
         "number_of_shards":"10",
         "number_of_replicas":"1",
         "uuid":"4tiKdB_6RWe-x2LoxDGOFg",
         "version":{  
            "created":"2010099"
         }
      }
   },
   "mappings":{  
      "product":{  
         "_routing":{  
            "required":true
         },
         "dynamic":"false",
         "_all":{  
            "enabled":false
         },
         "properties":{  
            "id":{  
               "index":"not_analyzed",
               "type":"string"
            },
            "entityId":{  
               "index":"not_analyzed",
               "type":"string"
            },
            "categories":{  
               "type":"integer"
            },
            "name":{  
               "type":"string",
               "fields":{  
                  "raw":{  
                     "index":"not_analyzed",
                     "type":"string"
                  }
               }
            },
            "routingKey":{  
               "index":"not_analyzed",
               "type":"string"
            }
            (...)
         }
      }
   },
   "aliases":[  
      (.. some aliases)
   }
</code></pre></div>
<p dir="auto">Retrieving the documents using the following query returns 1206 documents, <strong>with</strong> duplicates and therefor some documents are missing. Note that re-indexing changes the number of duplicates.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
  &quot;from&quot;: page * 3, // page from 0 to no more documents 
  &quot;size&quot;: 3,
  &quot;sort&quot;: [
    {
      &quot;name.raw&quot;: {
        &quot;order&quot;: &quot;asc&quot;
      }
    }
  ]
}"><pre class="notranslate"><code class="notranslate">{
  "from": page * 3, // page from 0 to no more documents 
  "size": 3,
  "sort": [
    {
      "name.raw": {
        "order": "asc"
      }
    }
  ]
}
</code></pre></div>
<p dir="auto">Retrieving the documents using the following query returns 1206 documents, <strong>without</strong> duplicates.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
  &quot;from&quot;: page * 3, //page from 0 to no more documents 
  &quot;size&quot;: 3,
  &quot;sort&quot;: [
    {
      &quot;entityId&quot;: {
        &quot;order&quot;: &quot;asc&quot;
      }
    }
  ]
}"><pre class="notranslate"><code class="notranslate">{
  "from": page * 3, //page from 0 to no more documents 
  "size": 3,
  "sort": [
    {
      "entityId": {
        "order": "asc"
      }
    }
  ]
}
</code></pre></div>
<p dir="auto">As there is a routing key, only one shard is hit at any time for both searches so the sorting should be consistent.</p>
<p dir="auto"><strong>Expected result</strong>:<br>
No duplicates, regardless of the sorting supplied.</p>
<p dir="auto"><strong>Elasticsearch version</strong>: 5.0.0 beta1</p>
<p dir="auto"><strong>Plugins installed</strong>: []</p>
<p dir="auto"><strong>JVM version</strong>:Version 8 Update 101</p>
<p dir="auto"><strong>Steps to reproduce</strong>:</p>
<ol dir="auto">
<li>Definition of a mapping where I define 2 "testString" blocks.</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
  &quot;settings&quot;: {
    &quot;analysis&quot;: {
      &quot;analyzer&quot;: {
        &quot;default&quot;: {
          &quot;type&quot;: &quot;custom&quot;,
          &quot;tokenizer&quot;: &quot;standard&quot;,
          &quot;filter&quot;: [
            &quot;standard&quot;,
            &quot;lowercase&quot;,
            &quot;asciifolding&quot;
          ]
        },
        &quot;lowercasedRaw&quot;: {
          &quot;type&quot;: &quot;custom&quot;,
          &quot;tokenizer&quot;: &quot;keyword&quot;,
          &quot;filter&quot;: [
            &quot;lowercase&quot;
          ]
        }
      }
    }
  },
  &quot;mappings&quot;: {
    &quot;_default_&quot;: {
      &quot;dynamic_templates&quot;: [
        /* all &quot;test&quot; fields have an analyzed subfield added */
        {
          &quot;testString&quot;: {
            &quot;match_mapping_type&quot;: &quot;string&quot;,
            &quot;match&quot; : &quot;test&quot;,
            &quot;mapping&quot;: {
              &quot;type&quot;: &quot;string&quot;,
              &quot;analyzer&quot;: &quot;lowercasedRaw&quot;,
              &quot;index&quot;: &quot;analyzed&quot;,
              &quot;fields&quot;: {
                &quot;analyzed&quot;: {
                  &quot;type&quot;: &quot;string&quot;,
                  &quot;index&quot;: &quot;analyzed&quot;,
                  &quot;analyzer&quot;: &quot;default&quot;
                }
              }
            }
          }
        },
        /* string fields are by default analyzed with the lowercasedRaw analyzer */
        /* NOTICE the duplicate &quot;testString&quot; name */
        {
          &quot;testString&quot;: {
            &quot;match_mapping_type&quot;: &quot;string&quot;,
            &quot;mapping&quot;: {
              &quot;analyzer&quot;: &quot;lowercasedRaw&quot;,
              &quot;index&quot;: &quot;analyzed&quot;,
              &quot;type&quot;: &quot;string&quot;
            }
          }
        }
      ]
    }
  }
}"><pre class="notranslate"><code class="notranslate">{
  "settings": {
    "analysis": {
      "analyzer": {
        "default": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": [
            "standard",
            "lowercase",
            "asciifolding"
          ]
        },
        "lowercasedRaw": {
          "type": "custom",
          "tokenizer": "keyword",
          "filter": [
            "lowercase"
          ]
        }
      }
    }
  },
  "mappings": {
    "_default_": {
      "dynamic_templates": [
        /* all "test" fields have an analyzed subfield added */
        {
          "testString": {
            "match_mapping_type": "string",
            "match" : "test",
            "mapping": {
              "type": "string",
              "analyzer": "lowercasedRaw",
              "index": "analyzed",
              "fields": {
                "analyzed": {
                  "type": "string",
                  "index": "analyzed",
                  "analyzer": "default"
                }
              }
            }
          }
        },
        /* string fields are by default analyzed with the lowercasedRaw analyzer */
        /* NOTICE the duplicate "testString" name */
        {
          "testString": {
            "match_mapping_type": "string",
            "mapping": {
              "analyzer": "lowercasedRaw",
              "index": "analyzed",
              "type": "string"
            }
          }
        }
      ]
    }
  }
}
</code></pre></div>
<ol dir="auto">
<li>First case<br>
Add the following records (in bulk):</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{&quot;index&quot;:{&quot;_type&quot; : &quot;log&quot;, &quot;_id&quot; : &quot;1&quot; } }
{&quot;test&quot;:&quot;testing&quot;}
{&quot;index&quot;:{&quot;_type&quot; : &quot;log&quot;, &quot;_id&quot; : &quot;2&quot; } }
{&quot;group&quot;:&quot;U2&quot;}"><pre class="notranslate"><code class="notranslate">{"index":{"_type" : "log", "_id" : "1" } }
{"test":"testing"}
{"index":{"_type" : "log", "_id" : "2" } }
{"group":"U2"}
</code></pre></div>
<p dir="auto">Get the mapping now (I removed the "<em>default</em>" part, because that stays the same):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
  &quot;testing&quot;: {
    &quot;mappings&quot;: {
      &quot;log&quot;: {
        &quot;dynamic_templates&quot;: [
          {
            &quot;testString&quot;: {
              &quot;match&quot;: &quot;test&quot;,
              &quot;match_mapping_type&quot;: &quot;string&quot;,
              &quot;mapping&quot;: {
                &quot;analyzer&quot;: &quot;lowercasedRaw&quot;,
                &quot;fields&quot;: {
                  &quot;analyzed&quot;: {
                    &quot;analyzer&quot;: &quot;default&quot;,
                    &quot;index&quot;: &quot;analyzed&quot;,
                    &quot;type&quot;: &quot;string&quot;
                  }
                },
                &quot;index&quot;: &quot;analyzed&quot;,
                &quot;type&quot;: &quot;string&quot;
              }
            }
          },
          {
            &quot;testString&quot;: {
              &quot;match_mapping_type&quot;: &quot;string&quot;,
              &quot;mapping&quot;: {
                &quot;analyzer&quot;: &quot;lowercasedRaw&quot;,
                &quot;index&quot;: &quot;analyzed&quot;,
                &quot;type&quot;: &quot;string&quot;
              }
            }
          }
        ],
        &quot;properties&quot;: {
          &quot;group&quot;: {
            &quot;type&quot;: &quot;text&quot;,
            &quot;analyzer&quot;: &quot;lowercasedRaw&quot;
          },
          &quot;test&quot;: {
            &quot;type&quot;: &quot;text&quot;,
            &quot;fields&quot;: {
              &quot;analyzed&quot;: {
                &quot;type&quot;: &quot;text&quot;
              }
            },
            &quot;analyzer&quot;: &quot;lowercasedRaw&quot;
          }
        }
      }
    }
  }
}"><pre class="notranslate"><code class="notranslate">{
  "testing": {
    "mappings": {
      "log": {
        "dynamic_templates": [
          {
            "testString": {
              "match": "test",
              "match_mapping_type": "string",
              "mapping": {
                "analyzer": "lowercasedRaw",
                "fields": {
                  "analyzed": {
                    "analyzer": "default",
                    "index": "analyzed",
                    "type": "string"
                  }
                },
                "index": "analyzed",
                "type": "string"
              }
            }
          },
          {
            "testString": {
              "match_mapping_type": "string",
              "mapping": {
                "analyzer": "lowercasedRaw",
                "index": "analyzed",
                "type": "string"
              }
            }
          }
        ],
        "properties": {
          "group": {
            "type": "text",
            "analyzer": "lowercasedRaw"
          },
          "test": {
            "type": "text",
            "fields": {
              "analyzed": {
                "type": "text"
              }
            },
            "analyzer": "lowercasedRaw"
          }
        }
      }
    }
  }
}
</code></pre></div>
<ol dir="auto">
<li>Second case<br>
Add the following records (in bulk): (these are the same records as above, but only the order is changed:</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{&quot;index&quot;:{&quot;_type&quot; : &quot;log&quot;, &quot;_id&quot; : &quot;2&quot; } }
{&quot;group&quot;:&quot;U2&quot;}
{&quot;index&quot;:{&quot;_type&quot; : &quot;log&quot;, &quot;_id&quot; : &quot;1&quot; } }
{&quot;test&quot;:&quot;testing&quot;}"><pre class="notranslate"><code class="notranslate">{"index":{"_type" : "log", "_id" : "2" } }
{"group":"U2"}
{"index":{"_type" : "log", "_id" : "1" } }
{"test":"testing"}
</code></pre></div>
<p dir="auto">Get the mapping now (I removed the "<em>default</em>" part, because that stays the same):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
  &quot;testing&quot;: {
    &quot;mappings&quot;: {
      &quot;log&quot;: {
        &quot;dynamic_templates&quot;: [
          {
            &quot;testString&quot;: {
              &quot;match&quot;: &quot;test&quot;,
              &quot;match_mapping_type&quot;: &quot;string&quot;,
              &quot;mapping&quot;: {
                &quot;analyzer&quot;: &quot;lowercasedRaw&quot;,
                &quot;fields&quot;: {
                  &quot;analyzed&quot;: {
                    &quot;analyzer&quot;: &quot;default&quot;,
                    &quot;index&quot;: &quot;analyzed&quot;,
                    &quot;type&quot;: &quot;string&quot;
                  }
                },
                &quot;index&quot;: &quot;analyzed&quot;,
                &quot;type&quot;: &quot;string&quot;
              }
            }
          }
        ],
        &quot;properties&quot;: {
          &quot;group&quot;: {
            &quot;type&quot;: &quot;text&quot;,
            &quot;fields&quot;: {
              &quot;keyword&quot;: {
                &quot;type&quot;: &quot;keyword&quot;,
                &quot;ignore_above&quot;: 256
              }
            }
          },
          &quot;test&quot;: {
            &quot;type&quot;: &quot;text&quot;,
            &quot;fields&quot;: {
              &quot;analyzed&quot;: {
                &quot;type&quot;: &quot;text&quot;
              }
            },
            &quot;analyzer&quot;: &quot;lowercasedRaw&quot;
          }
        }
      }
    }
  }
}"><pre class="notranslate"><code class="notranslate">{
  "testing": {
    "mappings": {
      "log": {
        "dynamic_templates": [
          {
            "testString": {
              "match": "test",
              "match_mapping_type": "string",
              "mapping": {
                "analyzer": "lowercasedRaw",
                "fields": {
                  "analyzed": {
                    "analyzer": "default",
                    "index": "analyzed",
                    "type": "string"
                  }
                },
                "index": "analyzed",
                "type": "string"
              }
            }
          }
        ],
        "properties": {
          "group": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "test": {
            "type": "text",
            "fields": {
              "analyzed": {
                "type": "text"
              }
            },
            "analyzer": "lowercasedRaw"
          }
        }
      }
    }
  }
}
</code></pre></div>
<p dir="auto">If you compare these 2 mappings you'll see the differences.</p>
<ol dir="auto">
<li>Notice that if you correct the original mapping and give the 2 items a different name, the mapping stays consistent, regardless the order of the records.</li>
</ol>