title: Dataframe null values transformed to 0 after UDF. Why?
body: <p>How can Nulls be handled when accessing dataframe <code>Row</code> values? Does the Null pointer Exception really require to be handled manually? There must be a better solution.</p>
  
  <pre><code>case class FirstThing(id:Int, thing:String, other:Option[Double])
  
  val df = Seq(FirstThing(1, "first", None), FirstThing(1, "second", Some(2)), FirstThing(1, "third", Some(3))).toDS
  df.show
  
  val list = df.groupBy("id").agg(collect_list(struct("thing", "other")).alias("mylist"))
  list.show(false)
  </code></pre>
  
  <p>This fails with NPE:</p>
  
  <pre><code>val xxxx = udf((t:Seq[Row])=&gt; t.map(elem =&gt; elem.getDouble(1)))
  list.withColumn("aa", xxxx(col("mylist"))).show(false)
  </code></pre>
  
  <p>This strangely gives 0:</p>
  
  <pre><code>val xxxx = udf((t:Seq[Row])=&gt; t.map(elem =&gt; elem.getAs[Double]("other")))
  list.withColumn("aa", xxxx(col("mylist"))).show(false)
  
  +---+-----------------------------------------+---------------+
  |id |mylist                                   |aa             |
  +---+-----------------------------------------+---------------+
  |1  |[[first,null], [second,2.0], [third,3.0]]|[0.0, 2.0, 3.0]|
  +---+-----------------------------------------+---------------+
  </code></pre>
  
  <p>Sadly this approach which works fine with data frames/datasets fails as well:</p>
  
  <pre><code>val xxxx = udf((t:Seq[Row])=&gt; t.map(elem =&gt; elem.getAs[Option[Double]]("other")))
  list.withColumn("aa", xxxx(col("mylist"))).show(false)
  </code></pre>
  
  <blockquote>
    <p>ClassCastException: java.lang.Double cannot be cast to scala.Option</p>
  </blockquote>