<p dir="auto">NGram Tokenizers are a very useful filter to match parts of compound words. Compound words can be commonly found p.e. in the German language, but occur in English as well (p.e. "thunderstorm").<br>
<a href="https://www.elastic.co/guide/en/elasticsearch/guide/current/ngrams-compound-words.html" rel="nofollow">https://www.elastic.co/guide/en/elasticsearch/guide/current/ngrams-compound-words.html</a></p>
<p dir="auto">NGram (also called n-gram) Tokenizers split up words into token sequences. Each token has a specified minimum gram length (min_gram) and maximum gram length (max_gram). Tokens consisting of three letters p.e. are called trigrams.<br>
Using trigrams as our starting point, the word "peanut" for example would be divided into the three-letter-tokens "pea", "ean", "anu" and "nut".<br>
Searching for "peanut" would therefore also (partially) match words like "<strong>pea</strong>ce", "S<strong>ean</strong>", "<strong>Anu</strong>bis" and "<strong>nut</strong>rition". They all contain one of these tokens, but might not be words (or the document if they are all in one) we want to see as our results.<br>
Statistically speaking we would then be suffering type I errors (false positives).<br>
<a href="https://en.wikipedia.org/wiki/Type_I_and_type_II_errors" rel="nofollow">https://en.wikipedia.org/wiki/Type_I_and_type_II_errors</a></p>
<p dir="auto">Obviously we might then try longer min_gram lengths, p.e. four or five letters long.<br>
With five-grams, "peanut" would be split up in "peanu" and "eanut", giving us far more exact results.<br>
One might come to the conclusion, the longer the n-gram, the better.<br>
Though this is be true to some extent and for some purposes, we will run into another problem.</p>
<p dir="auto">Elasticsearch's NGram Tokenizers ignore search terms with a length smaller than the specified min_gram!<br>
That means the actual search term will often be significantly shorter and more vague than intended. If there will be a search term left at all.</p>
<p dir="auto">We might p.e. search for "step by step", trying to find backpacks of this company. Here we have multiple query strings, combined by the AND operator.<br>
If we use our five-gram Tokenizer every word will be ignored. We will have no results resulting in type II errors (false negatives).<br>
Bigrams would respect every word, but lead to a lot of type I errors (see above the "peanut" example).<br>
Trigrams would only exclude the word "by" and respect the rest while being more accurate. But trigrams match the unwanted "Steckdosenleiste" (= multi-outlet power strip). <strong>"Ste"</strong>, "tep", <strong>"Ste"</strong>, "tep" on the one hand and <strong>"ste"</strong>, "tec", "eck", "ckd", "kdo", "dos", "ose", "sen", "enl", "nle", "lei", "eis", "ist", <strong>"ste"</strong> on the other means 50% of the search terms are matched!<br>
So we would might opt for a four-gram Tokenizer, bringing us good results this time.<br>
But then searching for "Jay Smith" would also give us somebody called "John Smith" with the same exact match percentage (type I error).</p>
<p dir="auto">Even if we wouldn't care for words smaller than four letters, sure enough one can come up with an example where five-grams would be needed.<br>
Searching for "Recht" (= law), a four-gram Tokenizer will match "Echt + rechnung" (= calculation/bill + real):<br>
"Recht" splits up into <strong>"rech"</strong>, <strong>"echt"</strong> while "Echt + rechnung" splits up into <strong>"echt"</strong> and <strong>"rech"</strong>, "echn", "chnu", "hnun", "nung", presenting a perfect match. This might come as a surprise, as those words are in two differents fields (the fields "name" and "description"), making a good example of how unintended and undesired a match can be.<br>
Respecting the token order ("Rech" before "echt") of our search term would help here, but is not supported. Same goes for the "step by step" example.</p>
<p dir="auto">If it were possible for the Elasticsearch NGram Tokenizer to fall back to smaller sized n-grams, using five-grams for "Smith" and trigrams for "Jay" (instead of ignoring "Jay completely), our problems described above would be gone. We would have relative n-gram precision according to the length of our search terms. But as it stands, we are forced to choose between lots of type I or lots of type II errors.</p>
<p dir="auto">To sum the examples we have the following situation:</p>
<ol dir="auto">
<li>A gram size smaller than five leads to problems with our search term "Recht" (type I errors).</li>
<li>A gram size larger than three ignores "jay" in "jay smith" (type I error).</li>
<li>A gram size larger than four ignores "step by step" (type II error).</li>
</ol>
<p dir="auto">This is why a combination of three to five grams (p.e. min_gram=3 and max_gram=5) also fails.</p>
<p dir="auto">Feel free to play around with the examples in my gist: <a href="https://www.found.no/play/gist/6d27486e51bdda88dd84" rel="nofollow">https://www.found.no/play/gist/6d27486e51bdda88dd84</a><br>
Edit the min_gram and max_gram sizes to your liking in the "Analysis" section (center left), rerun the queries ("Run" in the top right) and see the altered "Results" section (bottom right) with our Searches # 1, # 2 and # 3.<br>
Searching for "jay smith" corresponds to Search # 1, searching for "step by step" to Search # 2 and finally searching for "recht" to Search # 3. To try out other search terms, edit the queries in the "Searches" section (top right).<br>
Or edit the "Documents" section (top left), to experiment with different words to be searched for.</p>
<p dir="auto">To summarize:<br>
If we want to match search terms to compound words ("thunderstorm"), NGram Tokenizers are a fine tool. They split up words into tokens of a predefined size.<br>
But Elasticsearch's fixed size NGram Tokenizers force us to choose between either lots of type I or type II errors. What works well for short words like "Jay" won't work for longer words like "Recht" (German for law) and vice versa.<br>
Desired would be a flexible NGram Tokenizer. This means being able to switch from longer NGram Tokenizers (p.e. five-grams) to shorter (p.e. trigrams), as the search terms become shorter. Another, maybe more complicated way could be to make the NGram Tokenizer respect the order of the tokens.</p>
<p dir="auto">It would be great if the nGram tokenizer came with an option to force creation of a whole-word token for cases when the word length falls outside the min-gram/max-gram range. Currently, we have to resort to a dual-mapping approach to make sure these tokens get created, which limits how we can construct per-field queries.</p>