title: PySpark explode stringified array of dictionaries into rows
body: <p>I have a pyspark dataframe with StringType column (<code>edges</code>), which contains a list of dictionaries (see example below). The dictionaries contain a mix of value types, including another dictionary (<code>nodeIDs</code>). I need to explode the top-level dictionaries in the <code>edges</code> field into rows; ideally, I should then be able to convert their component values into separate fields. </p>
  
  <p><strong>Input:</strong></p>
  
  <pre><code>import findspark
  findspark.init()
  
  SPARK = SparkSession.builder.enableHiveSupport() \
                      .getOrCreate()
  
  data = [
      Row(trace_uuid='aaaa', timestamp='2019-05-20T10:36:33+02:00', edges='[{"distance":4.382441320292239,"duration":1.5,"speed":2.9,"nodeIDs":{"nodeA":954752475,"nodeB":1665827480}},{"distance":14.48582171131768,"duration":2.6,"speed":5.6,"nodeIDs":{"nodeA":1665827480,"nodeB":3559056131}}]', count=156, level=36),
      Row(trace_uuid='bbbb', timestamp='2019-05-20T11:36:10+03:00', edges='[{"distance":0,"duration":0,"speed":0,"nodeIDs":{"nodeA":520686131,"nodeB":520686216}},{"distance":8.654358326561642,"duration":3.1,"speed":2.8,"nodeIDs":{"nodeA":520686216,"nodeB":506361795}}]', count=179, level=258)
      ]
  
  df = SPARK.createDataFrame(data)
  
  </code></pre>
  
  <p><strong>Desired output:</strong></p>
  
  <pre><code>    data_reshaped = [
          Row(trace_uuid='aaaa', timestamp='2019-05-20T10=36=33+02=00', distance=4.382441320292239, duration=1.5, speed=2.9, nodeA=954752475, nodeB=1665827480, count=156, level=36),
          Row(trace_uuid='aaaa', timestamp='2019-05-20T10=36=33+02=00', distance=16.134844841712574, duration=2.9,speed=5.6, nodeA=1665827480, nodeB=3559056131, count=156, level=36),
          Row(trace_uuid='bbbb', timestamp='2019-05-20T11=36=10+03=00', distance=0, duration=0, speed=0, nodeA=520686131, nodeB=520686216, count=179, level=258),
          Row(trace_uuid='bbbb', timestamp='2019-05-20T11=36=10+03=00', distance=8.654358326561642, duration=3.1, speed=2.8, nodeA=520686216, nodeB=506361795, count=179, level=258)
         ]
  </code></pre>
  
  <p>Is there a way to do that? I've tried using <code>cast</code> to cast the <code>edges</code> field into an array first, but I can't figure out how to get it to work with the mixed data types.</p>
  
  <p>I'm using Spark 2.4.0.</p>