title: SparkSQL: How to deal with null values in user defined function?
body: <p>Given Table 1 with one column "x" of type String.
  I want to create Table 2 with a column "y" that is an integer representation of the date strings given in "x".</p>
  
  <p><strong>Essential</strong> is to keep <code>null</code> values in column "y".</p>
  
  <p>Table 1 (Dataframe df1):</p>
  
  <pre><code>+----------+
  |         x|
  +----------+
  |2015-09-12|
  |2015-09-13|
  |      null|
  |      null|
  +----------+
  root
   |-- x: string (nullable = true)
  </code></pre>
  
  <p>Table 2 (Dataframe df2):</p>
  
  <pre><code>+----------+--------+                                                                  
  |         x|       y|
  +----------+--------+
  |      null|    null|
  |      null|    null|
  |2015-09-12|20150912|
  |2015-09-13|20150913|
  +----------+--------+
  root
   |-- x: string (nullable = true)
   |-- y: integer (nullable = true)
  </code></pre>
  
  <p>While the user-defined function (udf) to convert values from column "x" into those of column "y" is:</p>
  
  <pre><code>val extractDateAsInt = udf[Int, String] (
    (d:String) =&gt; d.substring(0, 10)
        .filterNot( "-".toSet)
        .toInt )
  </code></pre>
  
  <p>and works, dealing with null values is not possible.</p>
  
  <p>Even though, I can do something like </p>
  
  <pre><code>val extractDateAsIntWithNull = udf[Int, String] (
    (d:String) =&gt; 
      if (d != null) d.substring(0, 10).filterNot( "-".toSet).toInt 
      else 1 )
  </code></pre>
  
  <p>I have found no way, to "produce" <code>null</code> values via udfs (of course, as <code>Int</code>s can not be <code>null</code>). </p>
  
  <p>My current solution for creation of df2 (Table 2) is as follows:</p>
  
  <pre><code>// holds data of table 1  
  val df1 = ... 
  
  // filter entries from df1, that are not null
  val dfNotNulls = df1.filter(df1("x")
    .isNotNull)
    .withColumn("y", extractDateAsInt(df1("x")))
    .withColumnRenamed("x", "right_x")
  
  // create df2 via a left join on df1 and dfNotNull having 
  val df2 = df1.join( dfNotNulls, df1("x") === dfNotNulls("right_x"), "leftouter" ).drop("right_x")
  </code></pre>
  
  <p><strong>Questions</strong>:</p>
  
  <ul>
  <li>The current solution seems cumbersome (and probably not efficient wrt. performance). Is there a better way?</li>
  <li>@Spark-developers: Is there a type <code>NullableInt</code> planned / avaiable, such that the following udf is possible (see Code excerpt ) ?</li>
  </ul>
  
  <p>Code excerpt</p>
  
  <pre><code>val extractDateAsNullableInt = udf[NullableInt, String] (
    (d:String) =&gt; 
      if (d != null) d.substring(0, 10).filterNot( "-".toSet).toInt 
      else null )
  </code></pre>