title: How to perform merge operation on spark Dataframe?
body: <p>I have spark dataframe <code>mainDF</code> and <code>deltaDF</code> both with a matching schema. </p>
  
  <p>Content of the mainDF is as follows:</p>
  
  <pre><code>id | name | age
  1  | abc  | 23
  2  | xyz  | 34
  3  | pqr  | 45
  </code></pre>
  
  <p>Content of <code>deltaDF</code> is as follows:</p>
  
  <pre><code>id | name | age
  1  | lmn  | 56
  4  | efg  | 37
  </code></pre>
  
  <p>I want to merge <code>deltaDF</code> with <code>mainDF</code> based on value of <code>id</code>. So if my <code>id</code> already exists in <code>mainDF</code> then the record should be updated and if <code>id</code> doesn't exist then the new record should be added. So the resulting data frame should be like this:</p>
  
  <pre><code>id | name | age
  1  | lmn  | 56
  2  | xyz  | 34
  3  | pqr  | 45
  4  | efg  | 37
  </code></pre>
  
  <p>This is my current code and it is working:</p>
  
  <pre><code>  val updatedDF = mainDF.as("main").join(deltaDF.as("delta"),$"main.id" === $"delta.id","inner").select($"main.id",$"main.name",$"main.age")
   mainDF= mainDF.except(updateDF).unionAll(deltaDF)
  </code></pre>
  
  <p>However here I need to explicitly provide list columns again in the select function which feels like overhead to me. Is there any other better/cleaner approach to achieve the same?</p>