title: Can not register a Scala UDF for pyspark
body: <p>In pyspark, I want to use a Scala UDF to filter an array with arbitrary member type.</p>
  
  <pre><code>package com.example.spark.udf
  
  import scala.collection.mutable.WrappedArray
  import org.apache.spark.sql.api.java.UDF2
  
  
  class ArrayFilterGt[T] extends UDF2[WrappedArray[T], T, WrappedArray[T]] {
  
    override def call(x: WrappedArray[T], y: T): WrappedArray[T] = (x, y) match {
      case (null, _) =&gt; null
      case (_, null) =&gt; x
      case (x, y) =&gt; x.filter(_ &gt; y )
    }
  }
  </code></pre>
  
  <p>But build error.</p>
  
  <pre><code>[info] Loading settings from plugin.sbt ...
  [info] Loading project definition from /some_path/spark-scala-util/project
  [info] Loading settings from build.sbt ...
  [info] Set current project to spark-scala-util (in build file:/some_path/spark-scala-util/)
  [success] Total time: 0 s, completed 2018/04/19 8:32:32
  [info] Updating ...
  [info] Done updating.
  [info] Compiling 3 Scala sources to /some_path/spark-scala-util/target/scala-2.12/classes ...
  [error] /some_path/spark-scala-util/src/main/scala/ArrayFilterGt.scala:13:31: value &gt; is not a member of type parameter T
  [error]     case (x, y) =&gt; x.filter(_ &gt; y )
  [error]                               ^
  [error] one error found
  [error] (Compile / compileIncremental) Compilation failed
  [error] Total time: 7 s, completed 2018/04/19 8:32:39
  </code></pre>
  
  <p>It would be quite basic mistake.
  But, since I am not familiar with Scala/Java, I cannot resolve it. Can someone help me?</p>
  
  <p><strong>EDIT:</strong></p>
  
  <p>I tried <code>[T &lt;% Ordered[T]]</code> and built it.</p>
  
  <pre><code>class ArrayFilterGt[T &lt;% Ordered[T]] extends UDF2[WrappedArray[T], T, WrappedArray[T]] {
  
    override def call(x: WrappedArray[T], y: T): WrappedArray[T] = (x, y) match {
      case (null, _) =&gt; null
      case (_, null) =&gt; x
      case (x, y) =&gt; x.filter(_ &gt; y )
    }
  }
  </code></pre>
  
  <p>But I could not register it on Pyspark.</p>
  
  <pre><code>spark_session.udf.registerJavaFunction(
      name='date_array_gt',
      javaClassName='com.example.spark.udf.ArrayFilterGt',
      returnType=ArrayType(elementType=DateType(), containsNull=True),
  )
  </code></pre>
  
  <p>error:</p>
  
  <pre><code>AnalysisException: 'Can not instantiate class com.example.spark.udf.ArrayFilterGt, please make sure it has public non argument constructor;'
  </code></pre>
  
  <p>Also, <code>com.example.spark.udf.ArrayFilterGt[Date]</code> raises same error.</p>
  
  <p>Since the other scala UDFs can be successfully registered and used, <code>--jars</code> option is not problem.</p>