title: How to unwind array in DataFrame (from JSON)?
body: <p>Each record in an RDD contains a json. I'm using SQLContext to create a DataFrame from the Json like this:</p>
  
  <pre><code>val signalsJsonRdd = sqlContext.jsonRDD(signalsJson)
  </code></pre>
  
  <p>Below is the schema. datapayload is an array of items. I want to explode the array of items to get a dataframe where each row is an item from datapayload. I tried to do something based on <a href="https://stackoverflow.com/questions/30501300/is-spark-dataframe-nested-structure-limited-for-selection">this</a> answer, but it seems that I would need to model the entire structure of the item in the <strong>case Row(arr: Array[...])</strong> statement. I'm probably missing something. </p>
  
  <pre><code>val payloadDfs = signalsJsonRdd.explode($"data.datapayload"){ 
      case org.apache.spark.sql.Row(arr: Array[String]) =&gt;  arr.map(Tuple1(_)) 
  }
  </code></pre>
  
  <p>The above code throws a scala.MatchError, because the type of the actual Row is very different from Row(arr: Array[String]). There is probably a simple way to do what I want, but I can't find it. Please help.</p>
  
  <p>Schema give below</p>
  
  <pre><code>signalsJsonRdd.printSchema()
  
  root
   |-- _corrupt_record: string (nullable = true)
   |-- data: struct (nullable = true)
   |    |-- dataid: string (nullable = true)
   |    |-- datapayload: array (nullable = true)
   |    |    |-- element: struct (containsNull = true)
   |    |    |    |-- Reading: struct (nullable = true)
   |    |    |    |    |-- A2DPActive: boolean (nullable = true)
   |    |    |    |    |-- Accuracy: double (nullable = true)
   |    |    |    |    |-- Active: boolean (nullable = true)
   |    |    |    |    |-- Address: string (nullable = true)
   |    |    |    |    |-- Charging: boolean (nullable = true)
   |    |    |    |    |-- Connected: boolean (nullable = true)
   |    |    |    |    |-- DeviceName: string (nullable = true)
   |    |    |    |    |-- Guid: string (nullable = true)
   |    |    |    |    |-- HandsFree: boolean (nullable = true)
   |    |    |    |    |-- Header: double (nullable = true)
   |    |    |    |    |-- Heading: double (nullable = true)
   |    |    |    |    |-- Latitude: double (nullable = true)
   |    |    |    |    |-- Longitude: double (nullable = true)
   |    |    |    |    |-- PositionSource: long (nullable = true)
   |    |    |    |    |-- Present: boolean (nullable = true)
   |    |    |    |    |-- Radius: double (nullable = true)
   |    |    |    |    |-- SSID: string (nullable = true)
   |    |    |    |    |-- SSIDLength: long (nullable = true)
   |    |    |    |    |-- SpeedInKmh: double (nullable = true)
   |    |    |    |    |-- State: string (nullable = true)
   |    |    |    |    |-- Time: string (nullable = true)
   |    |    |    |    |-- Type: string (nullable = true)
   |    |    |    |-- Time: string (nullable = true)
   |    |    |    |-- Type: string (nullable = true)
  </code></pre>