title: How to control the size of output parquet files in spark
body: <p>I have a spark job that reads the data from the hive table.<br>
  Ex: </p>
  
  <pre><code>r = spark.sql("select * from table")
  </code></pre>
  
  <p>and I have to write the result to hdfs location with 256mb parquet files.</p>
  
  <p>I am trying </p>
  
  <pre><code>r.write.parquet("/data_dev/work/experian/test11")
  </code></pre>
  
  <p>This generates 30MB files
  But I need it to generate 256MB files</p>
  
  <p>I also tried these configurations</p>
  
  <pre><code>r.write.option("parquet.block.size", 256 * 1024 * 1024 ). \
                 parquet("/path")
  </code></pre>
  
  <p>Still, the generated files seem to be ~30MB files</p>