title: Why do columns change to nullable in Apache Spark SQL?
body: <p>Why is <code>nullable = true</code> used after some functions are executed even though there are no NaN values in the <code>DataFrame</code>.</p>
  
  <pre class="lang-scala prettyprint-override"><code>val myDf = Seq((2,"A"),(2,"B"),(1,"C"))
           .toDF("foo","bar")
           .withColumn("foo", 'foo.cast("Int"))
  
  myDf.withColumn("foo_2", when($"foo" === 2 , 1).otherwise(0)).select("foo", "foo_2").show
  </code></pre>
  
  <p>When <code>df.printSchema</code> is called now <code>nullable</code> will be <code>false</code> for both columns.</p>
  
  <pre class="lang-scala prettyprint-override"><code>val foo: (Int =&gt; String) = (t: Int) =&gt; {
      fooMap.get(t) match {
        case Some(tt) =&gt; tt
        case None =&gt; "notFound"
      }
    }
  
  val fooMap = Map(
      1 -&gt; "small",
      2 -&gt; "big"
   )
  val fooUDF = udf(foo)
  
  myDf
      .withColumn("foo", fooUDF(col("foo")))
      .withColumn("foo_2", when($"foo" === 2 , 1).otherwise(0)).select("foo", "foo_2")
      .select("foo", "foo_2")
      .printSchema
  </code></pre>
  
  <p>However now, <code>nullable</code> is <code>true</code> for at least one column which was <code>false</code> before. How can this be explained?</p>