title: How to control the number of output part files created by Spark job upon writing?
body: <p>I am having couple of Spark jobs which processes thousands of files every day. File size may very from MBs to GBs. After finishing job I usually save using the following code</p>
  <pre><code>finalJavaRDD.saveAsParquetFile(&quot;/path/in/hdfs&quot;); OR
  dataFrame.write.format(&quot;orc&quot;).save(&quot;/path/in/hdfs&quot;) //storing as ORC file as of Spark 1.4
  </code></pre>
  <p>Spark job creates plenty of small part files in final output directory. As far as I understand Spark creates part file for each partition/task please correct me if I am wrong. How do we control amount of part files Spark creates?</p>
  <p>Finally, I would like to create Hive table using these parquet/orc directory and I heard Hive is slow when we have large no of small files.</p>