title: Spark - Saving data to Parquet file in case of dynamic schema
body: <p>I have a JavaPairRDD of the following typing: <br/></p>
  
  <p><code>Tuple2&lt;String, Iterable&lt;Tuple2&lt;String, Iterable&lt;Tuple2&lt;String, String&gt;&gt;&gt;&gt;&gt;</code></p>
  
  <p>that denotes the following object: <br/>
  (Table_name, <em>Iterable</em>(Tuple_ID, <em>Iterable</em>(Column_name, Column_value)))</p>
  
  <p>This means each record in the RDD will create one Parquet file.</p>
  
  <p>The idea is, as you may have guessed, to save each object as a new Parquet table called <em>Table_name</em>. In this table, there is one column called <em>ID</em> that stores the value <em>Tuple_ID</em>, and each column <em>Column_name</em> stores the value <em>Column_value</em>.</p>
  
  <p>The challenge I'm facing is that the table's columns (the schema) are collected on the fly on runtime, AND, as it is not possible to create nested RDDs in Spark, I can't create an RDD within the previous RDD (for each record) and save it finally to a Parquet file --after converting it to a DataFrame of course.</p>
  
  <p>And I can't just convert the previous RDD to a DataFrame, for the obvious reason (need to iterate to get column/value).</p>
  
  <p>As a temporarily workaround, I flattened the RDD into a list of the same typing as the RDD using <code>collect()</code>, but this is not the proper way as the data could be larger than the available disk space on the driver machine, causing an out of memory.</p>
  
  <p>Any advice on how to achieve this? please let me know if the question is not clear enough.</p>