title: Hive Incremental on Partitioned table
body: <p>I am working on implementing the incremental process on hive table A;
  Table A - is already created in hive with partitioned on YearMonth ( YYYYMM column ) with full volume.</p>
  
  <p>On-going basis, we are planning to import the updates/inserts from source and capture in hive Delta Table;</p>
  
  <p>as shown in below picture, Delta table indicates that new updates are pertaining to partitions ( 201804 / 201611 / 201705 ).</p>
  
  <p>For incremental process , I am planning to </p>
  
  <ol>
  <li>Select 3 Partitions from original table which are affected.</li>
  </ol>
  
  <blockquote>
    <p>INSERT INTO delta2 select YYYYMM from Table where YYYYMM in ( select
    distinct YYYYMM from Delta );</p>
  </blockquote>
  
  <ol start="2">
  <li><p>Merge these 3 partitions from Delta table with corresponding partitions from original table. ( I can follow Horton works 4 step strategy to apply the updates )</p>
  
  <pre><code>    Merge Delta2 + Delta : = new 3 partitions.
  </code></pre></li>
  <li><p>Drop 3 partitions from original table</p>
  
  <pre><code>Alter Table Drop partitions 201804 / 201611 / 201705
  </code></pre></li>
  <li><p>Add newly merged partitions back to Original table ( having new updates )</p></li>
  </ol>
  
  <p>I need to automate this scripts - Can you please suggest how to put above logic in hive QL or spark - Speacifically Identify partitions and drop them from original table.</p>
  
  <p><a href="https://i.stack.imgur.com/nac0r.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/nac0r.jpg" alt="enter image description here"></a></p>