title: Spark insert overwrite into dynamic partition table not working as expected
body: <p>I trying to merge a lot of small files generated into a bigger one. My files are in parquet format.
  I am creating a new temp table (external table) at a new location and then overwriting it with the old one.
  The files in the new table location gets perfectly merged if I do this with HiveSQL CLI (I get only one file per partition)... But when I try to do this via Spark SQL (running the same Hive commands) I am not getting the files merged but instead as previously they were there in the old location.
  I've set the confs </p>
  
  <pre><code>sqlContext.setConf("hive.exec.dynamic.partition", "true")
  sqlContext.setConf("hive.exec.dynamic.partition.mode", "nonstrict")
  
  sqlContext.sql("INSERT OVERWRITE TABLE &lt;newtable&gt; PARTITION (&lt;partition_cols&gt;) SELECT * FROM &lt;oldtable&gt;")
  </code></pre>
  
  <p>Please help!!</p>
  
  <p>P.S.:- I am using Spark 1.6.0 </p>