title: PySpark Explode JSON String into Multiple Columns
body: <p>I have a dataframe with a column of string datatype. The string represents an api request that returns a json.</p>
  <pre><code>df = spark.createDataFrame([
             (&quot;[{original={ranking=1.0, input=top3}, response=[{to=Sam, position=guard}, {to=John, position=center}, {to=Andrew, position=forward}]}]&quot;,1)], 
             &quot;col1:string, col2:int&quot;)
  df.show()
  </code></pre>
  <p>Which generates a dataframe like:</p>
  <pre><code>+--------------------+----+
  |                col1|col2|
  +--------------------+----+
  |[{original={ranki...|   1|
  +--------------------+----+
  </code></pre>
  <p>The output I would like to have col2 and have two additional columns from the response. Col3 would capture the player name, indicated by to= and col 4 would have their position indicated by position=. As well as the dataframe would now have three rows, since there's three players. Example:</p>
  <pre><code>+----+------+-------+
  |col2|  col3|   col4|
  +----+------+-------+
  |   1|   Sam|  guard|
  |   1|  John| center|
  |   1|Andrew|forward|
  +----+------+-------+
  </code></pre>
  <p>I've read that I can leverage something like:</p>
  <pre><code>df.withColumn(&quot;col3&quot;,explode(from_json(&quot;col1&quot;)))
  </code></pre>
  <p>However, I'm not sure how to explode given I want two columns instead of one and need the schema.</p>
  <p>Note, I can modify the response using json_dumps to return only the response piece of the string or...</p>
  <pre><code>[{to=Sam, position=guard}, {to=John, position=center}, {to=Andrew, position=forward}]}]
  </code></pre>