title: Spark missing column when write and read back
body: <p>I am creating a dataset and writing it to s3 in a directory structure <code>${BasePath}/y=2107/m=11/d=16/</code> in <strong>parquet</strong> format. After I creates current day data and writes it to s3, I need to read back the data for last 10 days. </p>
  
  <p>I can do this is 2 ways, </p>
  
  <p>1st Approach :  Write current day data to s3, read rest of the 9 days data and do a union, like</p>
  
  <pre><code>dataset.write.mode(SaveMode.Overwrite)
                .format(sourceConfig.format).save(getWriteBasePath(sourceConfig.sourcePath
                , replaceDate))
  newDf = spark.read.parquet("path1",...,"path9").union(dataset)
  </code></pre>
  
  <p>2nd Approach:  Write current day data to s3, and read all 10 days data back again. Something like : </p>
  
  <pre><code>     dataset.write.mode(SaveMode.Overwrite)
            .format(sourceConfig.format).save(getWriteBasePath(sourceConfig.sourcePath
            , replaceDate))
      newDf = spark.read.parquet("path1",...,"path10")
  </code></pre>
  
  <p>1st approach is working without any issues <strong>but very slow compared to 2nd approach</strong>. But with the second approach, some of the columns from current day's are getting null values after I read the complete data back to spark. I verified that the current day's data which is written is proper. </p>
  
  <p>I am not able to figure out why this is happening. I am creating spark context with following properties :</p>
  
  <pre><code>sparkSession.sparkContext.hadoopConfiguration.set("mapreduce.fileoutputcommitter.algorithm.version", "2")
        sparkSession.sparkContext.getConf.set("spark.hadoop.parquet.enable.summary-metadata", "false")
        sparkSession.sparkContext.getConf.set("spark.sql.parquet.mergeSchema", "false")
        sparkSession.sparkContext.getConf.set("spark.sql.parquet.filterPushdown", "true")
        sparkSession.sparkContext.getConf.set("spark.sql.hive.metastorePartitionPruning", "true")
  </code></pre>