title: Can I change the nullability of a column in my Spark dataframe?
body: <p>I have a StructField in a dataframe that is not nullable. Simple example:</p>
  
  <pre><code>import pyspark.sql.functions as F
  from pyspark.sql.types import *
  l = [('Alice', 1)]
  df = sqlContext.createDataFrame(l, ['name', 'age'])
  df = df.withColumn('foo', F.when(df['name'].isNull(),False).otherwise(True))
  df.schema.fields
  </code></pre>
  
  <p>which returns:</p>
  
  <blockquote>
    <p>[StructField(name,StringType,true),
     StructField(age,LongType,true),
     StructField(foo,BooleanType,false)]</p>
  </blockquote>
  
  <p>Notice that the field <code>foo</code> is not nullable. Problem is that (for reasons I won't go into) I want it to be nullable. I found this post <a href="https://stackoverflow.com/questions/33193958/change-nullable-property-of-column-in-spark-dataframe">Change nullable property of column in spark dataframe</a> which suggested a way of doing it so I adapted the code therein to this:</p>
  
  <pre><code>import pyspark.sql.functions as F
  from pyspark.sql.types import *
  l = [('Alice', 1)]
  df = sqlContext.createDataFrame(l, ['name', 'age'])
  df = df.withColumn('foo', F.when(df['name'].isNull(),False).otherwise(True))
  df.schema.fields
  newSchema = [StructField('name',StringType(),True), StructField('age',LongType(),True),StructField('foo',BooleanType(),False)]
  df2 = sqlContext.createDataFrame(df.rdd, newSchema)
  </code></pre>
  
  <p>which failed with:</p>
  
  <blockquote>
    <p>TypeError: StructField(name,StringType,true) is not JSON serializable</p>
  </blockquote>
  
  <p>I also see this in the stack trace:</p>
  
  <blockquote>
    <p>raise ValueError("Circular reference detected")</p>
  </blockquote>
  
  <p>So I'm a bit stuck. Can anyone modify this example in a way that enables me to define a dataframe where column <code>foo</code> is nullable?</p>