title: How to make concurrency work with dataframes writing into hive tables?
body: <p>I have multiple threads on Spark 1.6 writing into the same hive table (using parquet files), when they try to write at the same time it prompts an error during the renaming part of write files into HDFS. I'm searching a solution to bypass this known Spark issue.</p>
  
  <pre><code>class MyThread extends Runnable {
            def run {
            //some code
            myTable.write.format("parquet").mode("append")
                   .saveAsTable("hdfstable")
            //some code
            }
  }
  Executors.defaultThreadFactory().newThread(new MyThread).start()
  </code></pre>
  
  <p>I get this error :</p>
  
  <pre><code>org.apache.spark.SparkException: Job aborted.
      at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply$mcV$sp(InsertIntoHadoopFsRelation.scala:156)
      at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:108)
      at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:108)
      at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:53)
      at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation.run(InsertIntoHadoopFsRelation.scala:108)
      at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:58)
      at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:56)
      at org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:70)
      at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:132)
      at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:130)
      at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
      at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:130)
      at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:55)
      at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:55)
      at org.apache.spark.sql.DataFrameWriter.insertInto(DataFrameWriter.scala:189)
      at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:239)
      at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:221)
      at fr.neolink.spark.streaming.StreamingNeo$.algo(StreamingNeo.scala:837)
      at fr.neolink.spark.streaming.StreamingNeo$$anonfun$main$3$$anonfun$apply$18$MyThread$1.run(StreamingNeo.scala:374)
      at java.lang.Thread.run(Thread.java:748)
  </code></pre>
  
  <p>caused by :</p>
  
  <pre><code>java.io.IOException: Failed to rename 
  FileStatus{path=hdfs://my_hdfs_master/user/hive/warehouse/MYDB.db/hdfstable/_temporary/0/task_201812281010_1770_m_000000/part-r-00000-9a70cbea-d105-4f50-ba1b-372f555906ce.gz.parquet; 
  isDirectory=false; length=4608; replication=3; blocksize=134217728; modification_time=1545988247575; 
  access_time=1545988247494; owner=owner; group=hive; permission=rw-r--r--; isSymlink=false} 
  to hdfs://my_hdfs_master/user/hive/warehouse/MYDB.db/hdfstable/part-r-00000-9a70cbea-d105-4f50-ba1b-372f555906ce.gz.parquet
  </code></pre>
  
  <p>I found this issue on jira : <a href="https://issues.apache.org/jira/browse/SPARK-18626" rel="nofollow noreferrer">https://issues.apache.org/jira/browse/SPARK-18626</a></p>
  
  <p>Is there a way to make the writing part thread safe ? to make the execution one by one, one after another ? </p>
  
  <p>Thanks. </p>