title: Writing files to S3 using spark and scala is extremely slow. What is a better way to optimize this?
body: <pre><code>df.write.option("header", "false").
            option("quote", null).
            option("delimiter", Delimiter).
            csv(tempPath)
  </code></pre>
  
  <p>When I save 2 KB files, it takes less than 5 seconds to save to S3 but when I try to save large files of about 20GB, it takes more than 1 hour.</p>
  
  <p>Any suggestions to speed up the writing process?</p>
  
  <p>I am using "s3a//" for saving.</p>
  
  <p>UPDATE: When I manipulate the data of size 5 KB and generate 20KB file to store to S3, it takes 8 secs.
  When I try to manipulate  the data of size 250MB and generate 20KB file to store to S3 it takes 45 mins.
  I am doing a count before saving so it is evaluated by spark before saving.</p>
  
  <p>And it takes the less than a sec when I copy same 20KBB file to S3 using "aws S3 cp" command.</p>
  
  <p>So what is Spark doing that it is slowing down the save process? </p>