title: What are Untyped Scala UDF and Typed Scala UDF? What are their differences?
body: <p>I've been using Spark 2.4 for a while and just started switching to Spark 3.0 in these last few days. I got this error after switching to Spark 3.0 for running <code>udf((x: Int) =&gt; x, IntegerType)</code>:</p>
  <pre><code>Caused by: org.apache.spark.sql.AnalysisException: You're using untyped Scala UDF, which does not have the input type information. Spark may blindly pass null to the Scala closure with primitive-type argument, and the closure will see the default value of the Java type for the null argument, e.g. `udf((x: Int) =&gt; x, IntegerType)`, the result is 0 for null input. To get rid of this error, you could:
  1. use typed Scala UDF APIs(without return type parameter), e.g. `udf((x: Int) =&gt; x)`
  2. use Java UDF APIs, e.g. `udf(new UDF1[String, Integer] { override def call(s: String): Integer = s.length() }, IntegerType)`, if input types are all non primitive
  3. set spark.sql.legacy.allowUntypedScalaUDF to true and use this API with caution;
  </code></pre>
  <p>The solutions are proposed by Spark itself and after googling for a while I got to Spark Migration guide page:</p>
  <blockquote>
  <p>In Spark 3.0, using org.apache.spark.sql.functions.udf(AnyRef, DataType) is not allowed by default. Remove the return type parameter to automatically switch to typed Scala udf is recommended, or set spark.sql.legacy.allowUntypedScalaUDF to true to keep using it. In Spark version 2.4 and below, if org.apache.spark.sql.functions.udf(AnyRef, DataType) gets a Scala closure with primitive-type argument, the returned UDF returns null if the input values is null. However, in Spark 3.0, the UDF returns the default value of the Java type if the input value is null. For example, val f = udf((x: Int) =&gt; x, IntegerType), f($&quot;x&quot;) returns null in Spark 2.4 and below if column x is null, and return 0 in Spark 3.0. This behavior change is introduced because Spark 3.0 is built with Scala 2.12 by default.</p>
  <p>source: <a href="https://spark.apache.org/docs/latest/sql-migration-guide.html#:%7E:text=In%20Spark%203.0%2C%20using%20org.apache.spark.sql.functions.udf(AnyRef%2C%20DataType)%20is%20not%20allowed%20by%20default.%20Remove%20the%20return%20type%20parameter%20to%20automatically%20switch%20to%20typed%20Scala%20udf%20is%20recommended%2C%20or%20set%20spark.sql.legacy.allowUntypedScalaUDF%20to%20true%20to%20keep%20using%20it.%20In%20Spark%20version%202.4%20and%20below%2C%20if%20org.apache.spark.sql.functions.udf(AnyRef%2C%20DataType)%20gets%20a%20Scala%20closure%20with%20primitive-type%20argument%2C%20the%20returned%20UDF%20returns%20null%20if%20the%20input%20values%20is%20null.%20However%2C%20in%20Spark%203.0%2C%20the%20UDF%20returns%20the%20default%20value%20of%20the%20Java%20type%20if%20the%20input%20value%20is%20null.%20For%20example%2C%20val%20f%20%3D%20udf((x%3A%20Int)%20%3D%3E%20x%2C%20IntegerType)%2C%20f(%24%22x%22)%20returns%20null%20in%20Spark%202.4%20and%20below%20if%20column%20x%20is%20null%2C%20and%20return%200%20in%20Spark%203.0.%20This%20behavior%20change%20is%20introduced%20because%20Spark%203.0%20is%20built%20with%20Scala%202.12%20by%20default." rel="noreferrer">Spark Migration Guide</a></p>
  </blockquote>
  <p>I notice that my usual way of using <code>function.udf</code> API, which is <code>udf(AnyRef, DataType)</code>, is called <code>UnTyped Scala UDF</code> and the proposed solution, which is <code>udf(AnyRef)</code>, is called <code>Typed Scala UDF</code>.</p>
  <ul>
  <li>To my understanding, the first one looks more strictly typed than the second one where the first one has its output type explicitly defined and the second one does not, hence my confusion on why it's called UnTyped.</li>
  <li>Also the function got passed to <code>udf</code>, which is <code>(x:Int) =&gt; x</code>, clearly has its input type defined but Spark claiming <code>You're using untyped Scala UDF, which does not have the input type information</code>?</li>
  </ul>
  <p>Is my understanding correct? Even after more intensive searching I still can't find any material explaining what is UnTyped Scala UDF and what is Typed Scala UDF.</p>
  <p><strong>So my questions are: What are they? What are their differences?</strong></p>