title: Corrupt rows written to __HIVE_DEFAULT_PARTITION__ when attempting to overwrite Hive partition
body: <p>I am seeing some very odd behaviour when attempting to overwrite a partition in a Hive table using Spark 2.3</p>
  
  <p>Firstly I am setting the following setting when building my SparkSession:</p>
  
  <pre><code>.config("spark.sql.sources.partitionOverwriteMode", "dynamic")
  </code></pre>
  
  <p>I am then copying some data into  new table and partitioning by the date_id column.</p>
  
  <pre><code>ds
    .write
    .format("parquet")
    .option("compression", "snappy")
    .option("auto.purge", "true")
    .mode(saveMode)
    .partitionBy("date_id")
    .saveAsTable("tbl_copy")
  </code></pre>
  
  <p>I can see in HDFS that the relevant date_id directories have been created.</p>
  
  <p>I then create a DataSet containing data for the partition I wish to overwrite which contains data for a single date_id and insert into Hive as follows:</p>
  
  <pre><code>  ds
      .write
      .mode(SaveMode.Overwrite)
      .insertInto("tbl_copy")
  </code></pre>
  
  <p>As a sanity check I write the same Dataset to a new table.</p>
  
  <pre><code>      ds
          .write
          .format("parquet")
          .option("compression", "snappy")
          .option("auto.purge", "true")
          .mode(SaveMode.Overwrite)
          .saveAsTable("tmp_tbl")
  </code></pre>
  
  <p>The data in tmp_tbl is exactly as expected.</p>
  
  <p>However when I look at tbl_copy I see a new HDFS directory `date_id=<strong>HIVE_DEFAULT_PARTITION</strong></p>
  
  <p>Querying tbl_cpy</p>
  
  <pre><code>SELECT * from tbl_copy WHERE date_id IS NULL
  </code></pre>
  
  <p>I see the rows that should have been inserted into partition date_id=20180523 however the date_id column is null and an unrelated row_changed column has been populated with value 20180523.</p>
  
  <p>It appears the insert into Hive is somehow causing my data to get mangled. Writing the same Dataset into a new table causes no issues.</p>
  
  <p>Could anyone shed any light on this?</p>