title: How to process data in parallel but write results in a single file in Spark
body: <p>I have a Spark job that:</p>
  
  <ul>
  <li>Reads data from hdfs</li>
  <li>Does some intensive transformation <strong>without shuffling and aggregation (only map operations)</strong></li>
  <li>Writes results back to hdfs</li>
  </ul>
  
  <p>Let's say I have 10GB of raw data (40 blocks = 40 input partitions), which results in 100MB of processed data. To avoid generating many small files in hdfs I use "coalesce(1)" statement in order to write single file with results.
  Doing so I get only 1 task running (because of "coalesce(1)" and absence of shuffling), which processes all 10GB in a single thread.</p>
  
  <p><strong>Is there a way to do actual intensive processing in 40 parallel tasks and reduce number of partitions right before writing to disk and avoid data shuffle?</strong></p>
  
  <p>I have an idea that might work - to cache dataframe in memory after all processing (do a count to force Spark to cache the data) and then put "coalesce(1)" and write dataframe to disk</p>