title: Writing RDD partitions to individual parquet files in its own directory
body: <p>I am struggling with step where I want to write each RDD partition to separate parquet file with its own directory. Example will be:</p>
  
  <pre><code>    &lt;root&gt;
          &lt;entity=entity1&gt;
              &lt;year=2015&gt;
                  &lt;week=45&gt;
                      data_file.parquet
  </code></pre>
  
  <p>Advantage of this format is I can use this directly in SparkSQL as columns and I will not have to repeat this data in actual file. This would be good way to get to get to specific partition without storing separate partitioning metadata someplace else.</p>
  
  <p>​As a preceding step I have all the data loaded from large number of gzip files and partitioned based on the above key.</p>
  
  <p>Possible way would be to get each partition as separate RDD and then write it though I couldn't find any good way of doing it. </p>
  
  <p>Any help will be appreciated. By the way I am new to this stack.</p>