title: Extract Spark data frame from a nested structure
body: <p>I have a DataFrame with nested structure (originally an Avro output from a mapreduce job). I would like to flatten it. The schema of original DataFrame looks like this (simplified):</p>
  
  <pre><code>|-- key: struct
      |    |-- outcome: boolean
      |    |-- date: string
      |    |-- age: int
      |    |-- features: map
           |    |    |-- key: string
           |    |    |-- value: double
  |-- value: struct (nullable = true)
      |    |-- nullString: string (nullable = true)
  </code></pre>
  
  <p>In Json representation one row of the data looks like this:</p>
  
  <pre><code>{"key": 
      {"outcome": false,
       "date": "2015-01-01",
       "age" : 20,
       "features": {
          {"f1": 10.0,
           "f2": 11.0,
           ...
           "f100": 20.1
          }
       },
    "value": null
   }
  </code></pre>
  
  <p><strong>The <code>features</code> map has the same structure for all rows, i.e. the key set is the same (f1, f2, ..., f100).</strong> By "flatten" I mean the following. </p>
  
  <pre><code>+----------+----------+---+----+----+-...-+------+
  |   outcome|      date|age|  f1|  f2| ... |  f100|
  +----------+----------+---+----+----+-...-+------+
  |      true|2015-01-01| 20|10.0|11.0| ... |  20.1|
  ...
  (truncated)
  </code></pre>
  
  <p>I am using Spark 2.1.0 the spark-avro package from <a href="https://github.com/databricks/spark-avro" rel="nofollow noreferrer">https://github.com/databricks/spark-avro</a>. </p>
  
  <p>The original dataframe is read in by</p>
  
  <pre><code>import com.databricks.spark.avro._
  val df = spark.read.avro("path/to/my/file.avro")
  // it's nested
  df.show()
  +--------------------+------+
  |                 key| value|
  +--------------------+------+
  |[false,2015...      |[null]|
  |[false,2015...      |[null]|
  ...
  (truncated)
  </code></pre>
  
  <p>Any help is greatly appreciated!</p>