<p dir="auto">I've got an issue with facet counts that I've managed to simplify into a re-creatable example (attached).</p>
<p dir="auto">When searching an index I get a facet count for 'fabric' as 12 however when I then filter on that attribute it increases to 13, not sure how this is possible as by adding a must query I can surely only decrease the facet counts.</p>
<p dir="auto">Attached is the script which should recreate the issue in a test_bug index, it will insert 118 documents and run a query where the "fabric" facet comes out at 12.</p>
<p dir="auto">Query looks like:</p>
<div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
  &quot;query&quot;: {
    &quot;bool&quot;: {
      &quot;must&quot;: [
        {
          &quot;query_string&quot;: {
            &quot;default_operator&quot;: &quot;AND&quot;,
            &quot;fields&quot;: [
              &quot;_all&quot;
            ],
            &quot;query&quot;: &quot;double&quot;
          }
        }
      ]
    }
  },
  &quot;facets&quot;: {
    &quot;styles&quot;: {
      &quot;terms&quot;: {
        &quot;field&quot;: &quot;product_styles&quot;
      }
    }
  },
  &quot;size&quot;: 1
}"><pre class="notranslate">{
  <span class="pl-ent">"query"</span>: {
    <span class="pl-ent">"bool"</span>: {
      <span class="pl-ent">"must"</span>: [
        {
          <span class="pl-ent">"query_string"</span>: {
            <span class="pl-ent">"default_operator"</span>: <span class="pl-s"><span class="pl-pds">"</span>AND<span class="pl-pds">"</span></span>,
            <span class="pl-ent">"fields"</span>: [
              <span class="pl-s"><span class="pl-pds">"</span>_all<span class="pl-pds">"</span></span>
            ],
            <span class="pl-ent">"query"</span>: <span class="pl-s"><span class="pl-pds">"</span>double<span class="pl-pds">"</span></span>
          }
        }
      ]
    }
  },
  <span class="pl-ent">"facets"</span>: {
    <span class="pl-ent">"styles"</span>: {
      <span class="pl-ent">"terms"</span>: {
        <span class="pl-ent">"field"</span>: <span class="pl-s"><span class="pl-pds">"</span>product_styles<span class="pl-pds">"</span></span>
      }
    }
  },
  <span class="pl-ent">"size"</span>: <span class="pl-c1">1</span>
}</pre></div>
<p dir="auto">Facet comes out as:</p>
<div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&quot;facets&quot; : {
    &quot;styles&quot; : {
      &quot;_type&quot; : &quot;terms&quot;,
      &quot;missing&quot; : 11,
      &quot;total&quot; : 243,
      &quot;other&quot; : 141,
      &quot;terms&quot; : [ {
        &quot;term&quot; : &quot;beds&quot;,
        &quot;count&quot; : 19
      }, {
        &quot;term&quot; : &quot;fabric&quot;,
        &quot;count&quot; : 12
      }, {
        &quot;term&quot; : &quot;products&quot;,
        &quot;count&quot; : 11
      }, {
        &quot;term&quot; : &quot;new&quot;,
        &quot;count&quot; : 11
      }, {
        &quot;term&quot; : &quot;luxury&quot;,
        &quot;count&quot; : 10
      }, {
        &quot;term&quot; : &quot;furniture&quot;,
        &quot;count&quot; : 10
      }, {
        &quot;term&quot; : &quot;upholstered&quot;,
        &quot;count&quot; : 8
      }, {
        &quot;term&quot; : &quot;kingsize&quot;,
        &quot;count&quot; : 7
      }, {
        &quot;term&quot; : &quot;double&quot;,
        &quot;count&quot; : 7
      }, {
        &quot;term&quot; : &quot;bedroom&quot;,
        &quot;count&quot; : 7
      } ]
    }"><pre class="notranslate"><span class="pl-ent">"facets"</span> : {
    <span class="pl-ent">"styles"</span> : {
      <span class="pl-ent">"_type"</span> : <span class="pl-s"><span class="pl-pds">"</span>terms<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"missing"</span> : <span class="pl-c1">11</span>,
      <span class="pl-ent">"total"</span> : <span class="pl-c1">243</span>,
      <span class="pl-ent">"other"</span> : <span class="pl-c1">141</span>,
      <span class="pl-ent">"terms"</span> : [ {
        <span class="pl-ent">"term"</span> : <span class="pl-s"><span class="pl-pds">"</span>beds<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"count"</span> : <span class="pl-c1">19</span>
      }, {
        <span class="pl-ent">"term"</span> : <span class="pl-s"><span class="pl-pds">"</span>fabric<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"count"</span> : <span class="pl-c1">12</span>
      }, {
        <span class="pl-ent">"term"</span> : <span class="pl-s"><span class="pl-pds">"</span>products<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"count"</span> : <span class="pl-c1">11</span>
      }, {
        <span class="pl-ent">"term"</span> : <span class="pl-s"><span class="pl-pds">"</span>new<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"count"</span> : <span class="pl-c1">11</span>
      }, {
        <span class="pl-ent">"term"</span> : <span class="pl-s"><span class="pl-pds">"</span>luxury<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"count"</span> : <span class="pl-c1">10</span>
      }, {
        <span class="pl-ent">"term"</span> : <span class="pl-s"><span class="pl-pds">"</span>furniture<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"count"</span> : <span class="pl-c1">10</span>
      }, {
        <span class="pl-ent">"term"</span> : <span class="pl-s"><span class="pl-pds">"</span>upholstered<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"count"</span> : <span class="pl-c1">8</span>
      }, {
        <span class="pl-ent">"term"</span> : <span class="pl-s"><span class="pl-pds">"</span>kingsize<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"count"</span> : <span class="pl-c1">7</span>
      }, {
        <span class="pl-ent">"term"</span> : <span class="pl-s"><span class="pl-pds">"</span>double<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"count"</span> : <span class="pl-c1">7</span>
      }, {
        <span class="pl-ent">"term"</span> : <span class="pl-s"><span class="pl-pds">"</span>bedroom<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"count"</span> : <span class="pl-c1">7</span>
      } ]
    }</pre></div>
<p dir="auto">When adding a filter on the fabric like this:</p>
<div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
  &quot;query&quot;: {
    &quot;bool&quot;: {
      &quot;must&quot;: [
        {
          &quot;query_string&quot;: {
            &quot;default_operator&quot;: &quot;AND&quot;,
            &quot;fields&quot;: [
              &quot;_all&quot;
            ],
            &quot;query&quot;: &quot;double&quot;
          }
        },
        {
          &quot;term&quot;: {
            &quot;product_styles&quot;: &quot;fabric&quot;
          }
        }
      ]
    }
  },
  &quot;facets&quot;: {
    &quot;styles&quot;: {
      &quot;terms&quot;: {
        &quot;field&quot;: &quot;product_styles&quot;
      }
    }
  },
  &quot;size&quot;: 1
}"><pre class="notranslate">{
  <span class="pl-ent">"query"</span>: {
    <span class="pl-ent">"bool"</span>: {
      <span class="pl-ent">"must"</span>: [
        {
          <span class="pl-ent">"query_string"</span>: {
            <span class="pl-ent">"default_operator"</span>: <span class="pl-s"><span class="pl-pds">"</span>AND<span class="pl-pds">"</span></span>,
            <span class="pl-ent">"fields"</span>: [
              <span class="pl-s"><span class="pl-pds">"</span>_all<span class="pl-pds">"</span></span>
            ],
            <span class="pl-ent">"query"</span>: <span class="pl-s"><span class="pl-pds">"</span>double<span class="pl-pds">"</span></span>
          }
        },
        {
          <span class="pl-ent">"term"</span>: {
            <span class="pl-ent">"product_styles"</span>: <span class="pl-s"><span class="pl-pds">"</span>fabric<span class="pl-pds">"</span></span>
          }
        }
      ]
    }
  },
  <span class="pl-ent">"facets"</span>: {
    <span class="pl-ent">"styles"</span>: {
      <span class="pl-ent">"terms"</span>: {
        <span class="pl-ent">"field"</span>: <span class="pl-s"><span class="pl-pds">"</span>product_styles<span class="pl-pds">"</span></span>
      }
    }
  },
  <span class="pl-ent">"size"</span>: <span class="pl-c1">1</span>
}</pre></div>
<p dir="auto">the fabric facet then increases to 13:</p>
<div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&quot;facets&quot; : {
    &quot;styles&quot; : {
      &quot;_type&quot; : &quot;terms&quot;,
      &quot;missing&quot; : 0,
      &quot;total&quot; : 186,
      &quot;other&quot; : 97,
      &quot;terms&quot; : [ {
        &quot;term&quot; : &quot;fabric&quot;,
        &quot;count&quot; : 13
      }, {
        &quot;term&quot; : &quot;beds&quot;,
        &quot;count&quot; : 12
      }, {
        &quot;term&quot; : &quot;upholstered&quot;,
        &quot;count&quot; : 9
      }, {
        &quot;term&quot; : &quot;luxury&quot;,
        &quot;count&quot; : 9
      }, {
        &quot;term&quot; : &quot;products&quot;,
        &quot;count&quot; : 8
      }, {
        &quot;term&quot; : &quot;new&quot;,
        &quot;count&quot; : 8
      }, {
        &quot;term&quot; : &quot;kingsize&quot;,
        &quot;count&quot; : 8
      }, {
        &quot;term&quot; : &quot;double&quot;,
        &quot;count&quot; : 8
      }, {
        &quot;term&quot; : &quot;furniture&quot;,
        &quot;count&quot; : 7
      }, {
        &quot;term&quot; : &quot;bedroom&quot;,
        &quot;count&quot; : 7
      } ]
    }"><pre class="notranslate"><span class="pl-ent">"facets"</span> : {
    <span class="pl-ent">"styles"</span> : {
      <span class="pl-ent">"_type"</span> : <span class="pl-s"><span class="pl-pds">"</span>terms<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"missing"</span> : <span class="pl-c1">0</span>,
      <span class="pl-ent">"total"</span> : <span class="pl-c1">186</span>,
      <span class="pl-ent">"other"</span> : <span class="pl-c1">97</span>,
      <span class="pl-ent">"terms"</span> : [ {
        <span class="pl-ent">"term"</span> : <span class="pl-s"><span class="pl-pds">"</span>fabric<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"count"</span> : <span class="pl-c1">13</span>
      }, {
        <span class="pl-ent">"term"</span> : <span class="pl-s"><span class="pl-pds">"</span>beds<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"count"</span> : <span class="pl-c1">12</span>
      }, {
        <span class="pl-ent">"term"</span> : <span class="pl-s"><span class="pl-pds">"</span>upholstered<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"count"</span> : <span class="pl-c1">9</span>
      }, {
        <span class="pl-ent">"term"</span> : <span class="pl-s"><span class="pl-pds">"</span>luxury<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"count"</span> : <span class="pl-c1">9</span>
      }, {
        <span class="pl-ent">"term"</span> : <span class="pl-s"><span class="pl-pds">"</span>products<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"count"</span> : <span class="pl-c1">8</span>
      }, {
        <span class="pl-ent">"term"</span> : <span class="pl-s"><span class="pl-pds">"</span>new<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"count"</span> : <span class="pl-c1">8</span>
      }, {
        <span class="pl-ent">"term"</span> : <span class="pl-s"><span class="pl-pds">"</span>kingsize<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"count"</span> : <span class="pl-c1">8</span>
      }, {
        <span class="pl-ent">"term"</span> : <span class="pl-s"><span class="pl-pds">"</span>double<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"count"</span> : <span class="pl-c1">8</span>
      }, {
        <span class="pl-ent">"term"</span> : <span class="pl-s"><span class="pl-pds">"</span>furniture<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"count"</span> : <span class="pl-c1">7</span>
      }, {
        <span class="pl-ent">"term"</span> : <span class="pl-s"><span class="pl-pds">"</span>bedroom<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"count"</span> : <span class="pl-c1">7</span>
      } ]
    }</pre></div>
<p dir="auto">Hopefully I've explained everything and you can also recreate this. I'm using elasticsearch-0.19.1 on Mac OS X Lion.</p>
<p dir="auto">Script for recreating data is at <a href="https://gist.github.com/2283964">https://gist.github.com/2283964</a></p>
<p dir="auto">I'm working with nested documents and have noticed that my faceted search interface is giving the wrong counts when I have more than one shard. To be more specific, I'm working with RDF triples (entity &gt; attribute &gt; value) and I'm nesting the attributes (called predicates in my example):</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
  &quot;_id&quot; : &quot;512a2c022f0b4e3daa341e6c8bcf6c2f&quot;,
  &quot;url&quot;: &quot;http://dbpedia.org/resource/Alan_Shepard&quot;,
  &quot;predicates&quot;: [
    {
      &quot;type&quot;: &quot;type&quot;,
      &quot;string_value&quot;: [&quot;thing&quot;, &quot;person&quot;, &quot;astronaut&quot;]
    }, {
      &quot;type&quot;: &quot;label&quot;,
      &quot;string_value&quot;: [&quot;Alan Shepard&quot;]
    }, {
      &quot;type&quot;: &quot;time in space&quot;,
      &quot;float_value&quot;: [216.950]
    },
    ... lots more
  ]
}"><pre class="notranslate"><code class="notranslate">{
  "_id" : "512a2c022f0b4e3daa341e6c8bcf6c2f",
  "url": "http://dbpedia.org/resource/Alan_Shepard",
  "predicates": [
    {
      "type": "type",
      "string_value": ["thing", "person", "astronaut"]
    }, {
      "type": "label",
      "string_value": ["Alan Shepard"]
    }, {
      "type": "time in space",
      "float_value": [216.950]
    },
    ... lots more
  ]
}
</code></pre></div>
<p dir="auto">I've created a shell script (<a href="https://gist.github.com/1196986">https://gist.github.com/1196986</a>) that recreates the problem with a fresh index. The created data set has these totals:</p>
<ul dir="auto">
<li>thing (30)</li>
<li>creative work (20)</li>
<li>video game (10)</li>
<li>tv show (10)</li>
<li>people (10)</li>
</ul>
<p dir="auto">With only <strong>one shard</strong> the following query gives the correct counts no matter what the size parameter is set to:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="{
  &quot;size&quot;: 0,
  &quot;query&quot;: {
    &quot;match_all&quot;: {}
  },
  &quot;facets&quot;: {
    &quot;type_counts&quot;: {
      &quot;terms&quot;: {
        &quot;field&quot;: &quot;string_value&quot;,
        &quot;size&quot;: 5
      },
      &quot;nested&quot;: &quot;predicates&quot;,
      &quot;facet_filter&quot;: {
        &quot;term&quot;: {
          &quot;type&quot;: &quot;type&quot;
        }
      }
    }
  }
}"><pre class="notranslate"><code class="notranslate">{
  "size": 0,
  "query": {
    "match_all": {}
  },
  "facets": {
    "type_counts": {
      "terms": {
        "field": "string_value",
        "size": 5
      },
      "nested": "predicates",
      "facet_filter": {
        "term": {
          "type": "type"
        }
      }
    }
  }
}
</code></pre></div>
<p dir="auto">However, with <strong>more than one shard</strong> the size parameter affects the accuracy of the counts. If it is equal to or greater than the number of terms returned by the facet query (5 in this case) then it works fine. However, the terms at the bottom of the list start to display low counts as you reduce the size parameter:</p>
<p dir="auto">With "size" : 4</p>
<ul dir="auto">
<li>thing (30)</li>
<li>creative work (20)</li>
<li>video game (10)</li>
<li><strong>tv show (9)</strong></li>
</ul>
<p dir="auto">With "size" : 3</p>
<ul dir="auto">
<li>thing (30)</li>
<li><strong>creative work (15)</strong></li>
<li><strong>video game (9)</strong></li>
</ul>
<p dir="auto">With "size" : 2</p>
<ul dir="auto">
<li>thing (30)</li>
<li><strong>creative work (15)</strong></li>
</ul>
<p dir="auto">So it looks like the sub-totals from some of the shards aren't being included for some reason. BTW I'm on ubuntu and the problem seems to affect all versions of ES I've tried (17.0, 17.1 and 17.6). Any ideas...?</p>
<p dir="auto">P.S. absolutely loving ES - it's made my life a lot easier :)</p>