title: get size of parquet file in HDFS for repartition with Spark in Scala
body: <p>I have many parquet file directories on HDFS that contain a few thousands of small(most &lt; 100kb) parquet files each. They slow down my Spark job, so I want to combine them.</p>
  
  <p>With the following code I can repartition the local parquet file to smaller number of parts:</p>
  
  <pre><code>val pqFile = sqlContext.read.parquet("file:/home/hadoop/data/file.parquet")
  pqFile.coalesce(4).write.save("file:/home/hadoop/data/fileSmaller.parquet")
  </code></pre>
  
  <p>But I don't know how to get the size of a directory on HDFS through Scala code programmatically, hence I can't work out the number of partitions to pass to the <code>coalesce</code> function for the real data set.</p>
  
  <p>How can I do this? Or is there a convenient way within Spark so that I can configure the writer to write fixed size of parquet partition?</p>