title: Spark SQL: How to append new row to dataframe table (from another table)
body: <p>I am using Spark SQL with dataframes. I have an input dataframe, and I would like to append (or insert) its rows to a larger dataframe that has more columns. How would I do that? </p>
  
  <p>If this were SQL, I would use <code>INSERT INTO OUTPUT SELECT ... FROM INPUT</code>, but I don't know how to do that with Spark SQL.</p>
  
  <p>For concreteness:</p>
  
  <pre><code>var input = sqlContext.createDataFrame(Seq(
          (10L, "Joe Doe", 34),
          (11L, "Jane Doe", 31),
          (12L, "Alice Jones", 25)
          )).toDF("id", "name", "age")
  
  var output = sqlContext.createDataFrame(Seq(
          (0L, "Jack Smith", 41, "yes", 1459204800L),
          (1L, "Jane Jones", 22, "no", 1459294200L),
          (2L, "Alice Smith", 31, "", 1459595700L)
          )).toDF("id", "name", "age", "init", "ts")
  
  
  scala&gt; input.show()
  +---+-----------+---+
  | id|       name|age|
  +---+-----------+---+
  | 10|    Joe Doe| 34|
  | 11|   Jane Doe| 31|
  | 12|Alice Jones| 25|
  +---+-----------+---+
  
  scala&gt; input.printSchema()
  root
   |-- id: long (nullable = false)
   |-- name: string (nullable = true)
   |-- age: integer (nullable = false)
  
  
  scala&gt; output.show()
  +---+-----------+---+----+----------+
  | id|       name|age|init|        ts|
  +---+-----------+---+----+----------+
  |  0| Jack Smith| 41| yes|1459204800|
  |  1| Jane Jones| 22|  no|1459294200|
  |  2|Alice Smith| 31|    |1459595700|
  +---+-----------+---+----+----------+
  
  scala&gt; output.printSchema()
  root
   |-- id: long (nullable = false)
   |-- name: string (nullable = true)
   |-- age: integer (nullable = false)
   |-- init: string (nullable = true)
   |-- ts: long (nullable = false)
  </code></pre>
  
  <p>I would like to append all the rows of <code>input</code> to the end of <code>output</code>. At the same time, I would like to set the <code>output</code> column of <code>init</code> to be an empty string <code>''</code> and the <code>ts</code> column to be the current timestamp, e.g. 1461883875L.</p>
  
  <p>Any help would be appreciated.</p>