title: Spark Dataframe loosing data to s3
body: <p>I have a spark dataframe thats has the cleansed data. I have a total of 512 records in the data frame.
  Before I write to the S3, I am repartitioning to a single file since my total number of records are low and I want to keep the number of partitions less.</p>
  
  <pre><code>val t =tripAttributesDF.repartition(1)
  t.write.mode("overwrite").format("csv")
  .save("s3://&lt;Bucket&gt;/Trips_Detail_Table/csv/valid_trips_csv")
  </code></pre>
  
  <p>The DataFrame count shows correctly as follows
         scala> t.count()
         res32: Long = 512</p>
  
  <p>But when I open the CSV file I have only 34 rows. Not 512 as I expected.</p>
  
  <p>I looked hard in all posts, and no one seems to have encountered this issue.
  I am using Spark 2.2</p>
  
  <p>Please help if anyone knows the answer</p>
  
  <p>Thanks</p>