title: pyspark dataframe pivot a json column to new columns
body: <p>I would like to extract data from a json column in pyspark dataframe by python3.</p>
  <p>My dataframe:</p>
  <pre><code>  year month p_name json_col 
    2010 05    rchsc  [{&quot;attri_name&quot;: &quot;in_market&quot;, &quot;value&quot;: &quot;yes&quot;}, {&quot;attri_name&quot;: &quot;weight&quot;, &quot;value&quot;: &quot;12.56&quot;}, {&quot;attri_name&quot; : &quot;color&quot;, &quot;value&quot; : &quot;red&quot;} ]
  </code></pre>
  <p>I need a dataframe like:</p>
  <pre><code> year month p_name in_market weight color 
   2010 05    rchsc  yes       12.56  red
  </code></pre>
  <p>I have tried</p>
  <pre><code> from pyspark.sql.functions import from_json, col
   from pyspark.sql.types import StructType, StructField, StringType
  
   schema = StructType(
     [
       StructField('attri_name', StringType(), True),
      StructField('value', StringType(), True)
     ]
   )
   df.withColumn(&quot;new_col&quot;, from_json(&quot;json_col&quot;, schema))
  </code></pre>
  <p>But, no new columns are created.
  I am not sure how to decompose the json column and pivot them to new columns.</p>