<p dir="auto">If I <code class="notranslate">PUT</code> this template definition to <code class="notranslate">http://localhost:9200/_template/my_new_template</code> it works, but notice the field name <code class="notranslate">"some_field "</code> actually has a trailing whitespace in the String (I think that should be trimmed).</p>
<div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
  &quot;template&quot;: &quot;some_index_*&quot;,
  &quot;mappings&quot;: {
    &quot;some_mapping&quot;: {
      &quot;properties&quot;: {
        &quot;some_field &quot;: {
          &quot;type&quot;: &quot;double&quot;
        }
      }
    }
  }
}"><pre class="notranslate">{
  <span class="pl-ent">"template"</span>: <span class="pl-s"><span class="pl-pds">"</span>some_index_*<span class="pl-pds">"</span></span>,
  <span class="pl-ent">"mappings"</span>: {
    <span class="pl-ent">"some_mapping"</span>: {
      <span class="pl-ent">"properties"</span>: {
        <span class="pl-ent">"some_field "</span>: {
          <span class="pl-ent">"type"</span>: <span class="pl-s"><span class="pl-pds">"</span>double<span class="pl-pds">"</span></span>
        }
      }
    }
  }
}</pre></div>
<p dir="auto">In <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="37167454" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/6736" data-hovercard-type="issue" data-hovercard-url="/elastic/elasticsearch/issues/6736/hovercard" href="https://github.com/elastic/elasticsearch/issues/6736">#6736</a> I started trying to define specs for valid IDs, index names, field names etc, to avoid problems such as conflicts created by having an ID called <code class="notranslate">_mapping</code>.</p>
<p dir="auto">I think this is the wrong approach - a significant number of users will find that they have used identifiers which are no longer illegal.  Instead, we should identify where conflicts are possible and implement escaping to allow the unambiguous avoidance of conflict.</p>
<p dir="auto">Possible conflicts can arise in:</p>
<ul dir="auto">
<li>directory or file names (eg index "C:/")</li>
<li>URL paths (eg an ID called <code class="notranslate">_mapping</code> or <code class="notranslate">_create</code>)</li>
<li>query string (eg a routing value containing a comma)</li>
<li>field paths (eg a field containing a <code class="notranslate">.</code>)</li>
<li>scripting (eg a fieldname called <code class="notranslate">_id</code>)</li>
</ul>
<p dir="auto">We should provide a simple consistent escaping mechanism with minimal restrictions on allowable characters.</p>
<h2 dir="auto">Directory or file names</h2>
<p dir="auto">Today we use the actual index name to create a directory on the filesystem.  Instead, the original index name should be stored in the index metadata, and the directory name should be escaped as printable ASCII only (eg percent encoding?).</p>
<p dir="auto">There are more restrictions that need to be applied to filesystem names, most of which are already enforced today:</p>
<ul dir="auto">
<li>must be lowercase</li>
<li>must NOT start with an underscore, a <code class="notranslate">+</code> or a <code class="notranslate">-</code></li>
<li>index names starting with '.' are reserved</li>
<li>. and .. are forbidden, though they can occur in the name</li>
<li><code class="notranslate">\</code>, <code class="notranslate">/</code>, <code class="notranslate">*</code>, <code class="notranslate">?</code>, <code class="notranslate">"</code>, <code class="notranslate">&lt;</code>, <code class="notranslate">&gt;</code>, <code class="notranslate">|</code>, <code class="notranslate"> </code>,<code class="notranslate">,</code> are forbidden</li>
<li>illegal windows filenames are forbidden: CON, PRN, AUX, NUL, COM1, COM2, COM3, COM4, COM5, COM6, COM7, COM8, COM9, LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, and LPT9. Also avoid these names followed immediately by an extension; for example, NUL.txt is not recommended.</li>
</ul>
<p dir="auto">The same should apply to snapshot names.</p>
<p dir="auto">Filename length should be checked after escaping, not before.</p>
<h2 dir="auto">URL paths and query strings</h2>
<p dir="auto">Unicode characters and <code class="notranslate">/</code> are already handled by standard URI encoding.  The only special characters are:</p>
<ul dir="auto">
<li>a leading underscore <code class="notranslate">_</code></li>
<li>a comma <code class="notranslate">,</code></li>
</ul>
<p dir="auto">These can be escaped automatically by prepending <code class="notranslate">\</code> (so we would need to escape literal <code class="notranslate">\</code>'s as well).</p>
<p dir="auto">The <code class="notranslate">*</code> is used as a wildcard for various URL parameters, eg index and type names.  We cannot automatically escape the wildcard (because usually it should be a plain wildcard).  Similarly, the user can't escape it as <code class="notranslate">my_literal_\*</code>, because the <code class="notranslate">\</code> would itself be escaped as: <code class="notranslate">my_literal_\\*</code>, meaning that the wildcard would still function as a wildcard.</p>
<p dir="auto">The <code class="notranslate">*</code> should not be allowed in any identifiers which support wildcards (eg index, type, node names etc).</p>
<p dir="auto">Index wildcards can use a leading <code class="notranslate">+</code> or <code class="notranslate">-</code> to include/exclude patterns, but these characters should already be excluded by the index naming rules above.</p>
<h2 dir="auto">Field paths and Scripting</h2>
<p dir="auto">A field containing a <code class="notranslate">.</code> can be escaped as <code class="notranslate">foo\.bar</code> (and a field containing <code class="notranslate">\</code> as <code class="notranslate">foo\\bar</code>).</p>
<p dir="auto">Fields beginning with <code class="notranslate">_</code> can clash with metadata fields like <code class="notranslate">_id</code> or <code class="notranslate">_timestamp</code>, both in queries, aggregations, sorting, and scripting.</p>
<p dir="auto">There are four options:</p>
<ul dir="auto">
<li>Forbid fieldnames beginning with an underscore - this is too stringent and will impact existing data</li>
<li>Reserve existing metadata field names - this makes it difficult to add more</li>
<li>Allow <code class="notranslate">\</code> escaping to distinguish the metadata <code class="notranslate">_id</code> field from the body <code class="notranslate">\_id</code> field</li>
<li>Add a single namespace for metadata fields, eg <code class="notranslate">_root._id</code> or <code class="notranslate">_doc._id</code> (<code class="notranslate">_meta</code> is already taken unfortunately)</li>
</ul>
<p dir="auto">Also see <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="30222870" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/5558" data-hovercard-type="issue" data-hovercard-url="/elastic/elasticsearch/issues/5558/hovercard" href="https://github.com/elastic/elasticsearch/issues/5558">#5558</a></p>