title: Multiple spark jobs appending parquet data to same base path with partitioning
body: <p>I have multiple jobs that I want to execute in parallel that append daily data into the same path using partitioning.  </p>
  
  <p>e.g. </p>
  
  <pre><code>dataFrame.write().
           partitionBy("eventDate", "category")
              .mode(Append)
              .parquet("s3://bucket/save/path");
  </code></pre>
  
  <p>Job 1 - category = "billing_events"
  Job 2 - category = "click_events"</p>
  
  <p>Both of these jobs will truncate any existing partitions that exist in the s3 bucket prior to execution and then save the resulting parquet files to their respective partitions.</p>
  
  <p>i.e. </p>
  
  <p>job 1 - > s3://bucket/save/path/eventDate=20160101/channel=billing_events</p>
  
  <p>job 2 - > s3://bucket/save/path/eventDate=20160101/channel=click_events</p>
  
  <p>The problem im facing is the temporary files that get created during the job execution by spark.  It saves the working out files to the base path</p>
  
  <p>s3://bucket/save/path/_temporary/...</p>
  
  <p>so both jobs end up sharing the same temp folder and cause conflict, which ive noticed can cause one job to delete temp files, and the other job fail with a 404 from s3 saying an expected temp file doesnt exist.  </p>
  
  <p>Has anyone faced this issue and come up with a strategy to have parallel execution of jobs in the same base path?</p>
  
  <p>im using spark 1.6.0 for now</p>