title: Elasticearch and Spark: Updating existing entities
body: <p>What is the correct way, when using Elasticsearch with Spark, to update existing entities?</p>
  
  <p>I wanted to something like the following:</p>
  
  <ol>
  <li>Get existing data as a map. </li>
  <li>Create a new map, and populate it with the updated fields.</li>
  <li>Persist the new map.</li>
  </ol>
  
  <p>However, there are several issues:</p>
  
  <ol>
  <li>The list of returned fields cannot contain the _id, <a href="https://stackoverflow.com/questions/30655030/is-it-possible-to-include-id-in-source-in-elasticsearch">as it is not part of the source</a>.</li>
  <li><p>If, for testing, I hardcode an existing <code>_id</code> in the map of new values, the following exception is thrown:</p>
  
  <p>org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest</p></li>
  </ol>
  
  <p>How should the <code>_id</code> be retrieved, and how should it be passed back to Spark?</p>
  
  <p>I include the following code below to better illustrate what I was trying to do:</p>
  
  <pre><code>JavaRDD&lt;Map&lt;String, Object&gt;&gt; esRDD = JavaEsSpark.esRDD(jsc, INDEX_NAME+"/"+TYPE_NAME, 
  "?source=,field1,field2).values();
  
  Iterator&lt;Map&lt;String, Object&gt;&gt; iter = esRDD.toLocalIterator();
  List&lt;Map&lt;String, Object&gt;&gt; listToPersist = new ArrayList&lt;Map&lt;String, Object&gt;&gt;();
  while(iter.hasNext()){
     Map&lt;String, Object&gt; map = iter.next();
     // Get existing values, and do transformation logic
  
     Map&lt;String, Object&gt; newMap = new HashMap&lt;String, Object&gt;();
     newMap.put("_id", ??????);
     newMap.put("field1", new_value);
     listToPersist.add(newMap);
  }
  JavaRDD javaRDD = jsc.parallelize(ImmutableList.copyOf(listToPersist));
  JavaEsSpark.saveToEs(javaRDD, INDEX_NAME+"/"+TYPE_NAME); 
  </code></pre>
  
  <p>Ideally, I would want to update the existing map in place, rather than create a new one.</p>
  
  <p>Does anyone have any example code to show, when using Spark, the correct way to update existing entities in elasticsearch?</p>
  
  <p>Thanks </p>