title: Spark UDF as function parameter, UDF is not in function scope
body: <p>I have a few UDFs that I'd like to pass along as a function argument along with data frames. </p>
  
  <p>One way to do this might be to create the UDF within the function, but that would create and destroy several instances of the UDF without reusing it which might not be the best way to approach this problem.</p>
  
  <p>Here's a sample piece of code - </p>
  
  <pre><code>val lkpUDF = udf{(i: Int) =&gt; if (i &gt; 0) 1 else 0}
  
  val df =   inputDF1
      .withColumn("new_col", lkpUDF(col("c1")))
  val df2 =   inputDF2.
    .withColumn("new_col", lkpUDF(col("c1")))
  </code></pre>
  
  <p>Instead of doing the above, I'd ideally want to do something like this - </p>
  
  <pre><code>val lkpUDF = udf{(i: Int) =&gt; if (i &gt; 0) 1 else 0}
  
  def appendCols(df: DataFrame, lkpUDF: ?): DataFrame = {
  
      df
        .withColumn("new_col", lkpUDF(col("c1")))
  
    }
  val df = appendCols(inputDF, lkpUDF)
  </code></pre>
  
  <p>The above UDF is pretty simple, but in my case it can return a primitive type or a user defined case class type. Any thoughts/ pointers would be much appreciated. Thanks.</p>