title: Append new data to partitioned parquet files
body: <p>I am writing an ETL process where I will need to read hourly log files, partition the data, and save it. I am using Spark (in Databricks).
  The log files are CSV so I read them and apply a schema, then perform my transformations. </p>
  
  <p>My problem is, how can I save each hour's data as a parquet format but append to the existing data set? When saving, I need to partition by 4 columns present in the dataframe. </p>
  
  <p>Here is my save line:</p>
  
  <pre><code>data
      .filter(validPartnerIds($"partnerID"))
      .write
      .partitionBy("partnerID","year","month","day")
      .parquet(saveDestination)
  </code></pre>
  
  <p>The problem is that if the destination folder exists the save throws an error. 
  If the destination doesn't exist then I am not appending my files.</p>
  
  <p>I've tried using <code>.mode("append")</code> but I find that Spark sometimes fails midway through so I end up loosing how much of my data is written and how much I still need to write. </p>
  
  <p>I am using parquet because the partitioning substantially increases my querying in the future. As well, I must write the data as some file format on disk and cannot use a database such as Druid or Cassandra.</p>
  
  <p>Any suggestions for how to partition my dataframe and save the files (either sticking to parquet or another format) is greatly appreciated. </p>