title: Parsing json column of Pyspark dataframe that has one of the key value as None
body: <p>I have dataframe with two columns - id, cast. column 'cast'  has values in json array format as shown below.</p>
  <p><a href="https://i.stack.imgur.com/5Zvsd.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/5Zvsd.png" alt="enter image description here" /></a></p>
  <p>Json structure for column cast for id=862</p>
  <blockquote>
  <p>&quot;[{'cast_id': 14, 'character': 'Woody (voice)', 'credit_id': '52fe4284c3a36847f8024f95', 'gender': 2, 'id': 31, 'name': 'Tom Hanks', 'order': 0, <strong>'profile_path'</strong>: '/pQFoyx7rp09CJTAb932F2g8Nlho.jpg'}]&quot;</p>
  </blockquote>
  <p>Json structure for column cast for id=8844</p>
  <blockquote>
  <p>&quot;[{'cast_id': 1, 'character': 'Alan Parrish', 'credit_id': '52fe44bfc3a36847f80a7c73', 'gender': 2, 'id': 2157, 'name': 'Robin Williams', 'order': 0, <strong>'profile_path'</strong>: None}]&quot;</p>
  </blockquote>
  <p>To parse column 'cast', I have below code</p>
  <pre><code>cast_schema=ArrayType(StructType([
    StructField('cast_id',IntegerType(),nullable=True),
    StructField('character',StringType(),nullable=True),
    StructField('credit_id',StringType(),nullable=True),
    StructField('gender',IntegerType(),nullable=True),
    StructField('id',IntegerType(),nullable=True),
    StructField('name',StringType(),nullable=True),
    StructField('order',IntegerType(),nullable=True),
    StructField('profile_path',StringType(),nullable=True)
  ]))
  credits_upd.withColumn('movies_cast',from_json(col('cast'),cast_schema).getField(&quot;name&quot;).alias('movies_cast')).show()
  </code></pre>
  <p>It returns below output:</p>
  <p><a href="https://i.stack.imgur.com/BfrtA.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/BfrtA.png" alt="enter image description here" /></a></p>
  <p>As shown above, for id - 8844, 'cast' column did not get parsed correctly. My guess is 'profile_path' key of the 'cast' column has value as <strong>None</strong> for id 8844 due to which json is not getting parsed.</p>
  <p>I am not sure how to define schema such that it is able to parse when one of the Json key has value as <strong>None</strong>.</p>
  <p>My actual file has 45k records.</p>