title: Spark - Performing union of Dataframes inside a for loop starting from empty DataFrame
body: <p>I have a Dataframe with a column called "generationId" and other fields. Field "generationId" takes a range of integer values from 1 to N (upper bound to N is known and is small, between 10 and 15) and I want to process the DataFrame in the following way (pseudo code):</p>
  
  <pre><code>results = emptyDataFrame &lt;=== how do I do this ?
  for (i &lt;- 0 until getN(df)) {
      val input = df.filter($"generationId" === i)
      results.union(getModel(i).transform(input))
  }
  </code></pre>
  
  <p>Here getN(df) gives the N for that data frame based on some criteria. In the loop, input is filtered based on matching against "i" and then fed to some model (some internal library) which transforms the input by adding 3 more columns to it.</p>
  
  <p>Ultimately I would like to get union of all those transformed data frames, so I have all columns of the original data frame plus the 3 additional columns added by the model for each row. I am not able to figure out how to initialize results and unionize the results in each iteration. I do know the exact schema of the result ahead of time. So I did</p>
  
  <pre><code>val newSchema = ...
  </code></pre>
  
  <p>but I am not sure how to pass that to emptyRDD function and build a empty Dataframe and use it inside the loop.</p>
  
  <p>Also, if there is a much efficient way to do this inside map operation, please suggest.</p>