title: Spark merge dataframe with mismatching schemas without extra disk IO
body: <p>I would like to merge 2 dataframes with (potentially) mismatching schemas</p>
  
  <pre><code>org.apache.spark.sql.DataFrame = [name: string, age: int, height: int]
  org.apache.spark.sql.DataFrame = [name: string, age: int]
  
  scala&gt; A.unionAll(B)
  </code></pre>
  
  <p>would result in :</p>
  
  <pre><code>org.apache.spark.sql.AnalysisException: Union can only be performed on tables with the same number of columns, but the left table has 2 columns and the right has 3;
  </code></pre>
  
  <p>I would like to do this from within Spark.
  However, the Spark docs only propose to write the whole 2 dataframes out to a directory and read them back in using <code>spark.read.option("mergeSchema", "true")</code>.</p>
  
  <p><a href="http://spark.apache.org/docs/latest/sql-programming-guide.html#schema-merging" rel="noreferrer">link to docs</a></p>
  
  <p>So a union doesn't help me out, and neither does the documentation. I would like to keep this extra I/O out of my job if at all possible. Am I missing some undocumented info, or is it not possible (yet)?</p>