title: Supporting query on dynamic columns in elastic search
body: <p>I need to support query on dynamic (not predefined) tags in elastic search. Lets say I have a blog document and wanted to support query on different set of columns i.e. tagTypeA=valueX &amp; tagTypeB=ValueY and these tagTypeX columns are not known beforehand. There will be only one value for each of these tags. The user will pass this additional data as Map String::String to my API (no strict model / structure). I also need to support aggregation queries on these tag columns.</p>
  
  <p>I am thinking of three ways to support this feature.</p>
  
  <ol>
  <li>Declare that I can support a maximum of N type of dynamic tags only per document (say 10) and create internal columns like Tag1,Tag2 ... Tag10. Now have a config to maintain the mapping of TagTypeA=Tag1, TagTypeB=Tag2 etc. In the code, iterate the input key value pair and generate ES search query dynamically by using key to columnName mapping.</li>
  </ol>
  
  <p>Pros : Simple to implement</p>
  
  <p>Cons : Overhead of maintaining the mapping. This has to modified every-time a new type of document/client is onboarded / new field has to be added for existing client.</p>
  
  <ol start="2">
  <li>Create a non-analyzed field in ES with array of strings. When storing the data, store in a concatenated format of key+"Delimiter"+value. So if the input map has TagTypeA=Good &amp; TagTypeB=High, then this will be stored as ["TagTypeA-Good","TagTypeB-High"] in ES. When user queries, construct back the contacted strings and search them.</li>
  </ol>
  
  <p>Pros : No code changes required to onboard new clients / to add or update new fields </p>
  
  <p>Cons : First of all it doesn't sound clean. The key should not have Delimeter. Changing mapping at later point of time is very tedious as we have to change values of all existing string values.</p>
  
  <ol start="3">
  <li>Don't define any schema and let the json key - value pair of tags passthrough to elastic search PUT call. For any new keys which are not already present elastic search will automatically add it to the indices with default type inference (controlled using dynamic templates with restriction that all field names should start with a certain prefix).</li>
  </ol>
  
  <p>Pros : No configuration or manual concatenation of input. Any addition of columns in handled transparently without any manual effort.</p>
  
  <p>Cons : I am sure there would be some cons with this. But can't think of anything concrete. Please suggest? One con which I can think is, as the field names are left out wild, any one abusive client can bloat up the index which can potentially bring down whole cluster due to large number of fields, unless we have a workflow to manually register &amp; approve these fields sotred in seperate DB. any other cons?</p>
  
  <p>I am personally thinking on aligning to Option #3.</p>
  
  <p>Can any one please share your views on above three approaches and if there is a better way to solve this.</p>
  
  <p>Thanks,
  Harish</p>