title: spark hive insert transaction,batch data at most once
body: <p>I have data to insert to hive each week,for example,1st week I insert:</p>
  
  <pre><code>spark.range(0,2).write.mode("append").saveAsTable("batches")
  </code></pre>
  
  <p>and 2nd week I insert:</p>
  
  <pre><code>spark.range(2,4).write.mode("append").saveAsTable("batches")
  </code></pre>
  
  <p>I am worry when the record which id is 2 is inserted,some exception occur,3 is not inserted,then I insert the data of 2nd week again,there will be two 2.</p>
  
  <p>I googled,
  hive is not suit for delete particular set of records:<a href="https://stackoverflow.com/questions/17810537/how-to-delete-and-update-a-record-in-hive">link</a>,so I can not delete the data left before the exception of 2nd week.</p>
  
  <p>I think I can use hive transaction,but <a href="https://stackoverflow.com/questions/50254590/how-to-read-orc-transaction-hive-table-in-spark">spark is not right now (2.3 version) fully compliant with hive transactional tables.</a> ,seems even we can not read hive if we enable hive transaction.</p>
  
  <p>And I see another website:<a href="http://mail-archives.apache.org/mod_mbox/spark-user/201601.mbox/%3CCAC3gpCap4pMmp86PC6jjxHu4ag5T3-Cniheo6mEhNmjZoNCx4Q@mail.gmail.com%3E" rel="nofollow noreferrer">Hive's ACID feature (which introduces transactions) is not required for
  inserts, only updates and deletes. Inserts should be supported on a vanilla
  Hive shell.</a>,but I do not why transaction is not needed for insert,and what does it mean.</p>
  
  <p>So,if I do not want to get the duplicated data,or like run at most once,what shall I do?</p>
  
  <p>If you say there won`t be an exception between 2 and 3,but consider another case,if I have many tables to write:</p>
  
  <pre><code>   spark.range(2,4).write.mode("append").saveAsTable("batches")
     val x=1/0
     spark.range(2,4).write.mode("append").saveAsTable("batches2")
  </code></pre>
  
  <p>I tested it,new records 2 and 3 have been inserted into table "batches",but not inserted into "batches2",so if I want to insert again,must I only insert "batches2"?but how can I know where is the exception,and which tables should I insert again?I must to insert many try and catch,but it makes code hard to read and write.And what about the exception is disk is full or power off?</p>
  
  <p>How to prevent duplicated data?</p>