title: Failed to execute user defined function
body: <p>I have the following UDF:</p>
  
  <pre><code>val jac_index:(Array[String],Array[String])=&gt;Float=(Sq1:Array[String],Sq2:Array[String])=&gt;
  {
      val Sq3=Sq1.intersect(Sq2)
      val Sq4=Sq1.union(Sq2).distinct
      if (!Sq4.isEmpty) Sq3.length.toFloat/Sq4.length.toFloat else 0F
  }
  val jacUDF=udf(jac_index)
  </code></pre>
  
  <p>and when I execute the following sentence</p>
  
  <pre><code>val movie_jac_df=movie_pairs_df.withColumn("jac",jacUDF(movie_pairs_df("name"),movie_pairs_df("name2")))
  </code></pre>
  
  <p>I get the error "Failed to execute user defined function"</p>
  
  <p>the schema of <strong>movie_pairs_df</strong> is the following</p>
  
  <pre><code>root
   |-- movie: string (nullable = true)
   |-- name: array (nullable = true)
   |    |-- element: string (containsNull = true)
   |-- movie2: string (nullable = true)
   |-- name2: array (nullable = true)
   |    |-- element: string (containsNull = true)
  </code></pre>
  
  <p>So what's the cause?</p>