title: Spark Custom Aggregator -- register and invoke through PySpark
body: <p>According to various docs, to write a custom Aggregator in Spark it must be written in Java/Scala.</p>
  <p><a href="https://medium.com/swlh/apache-spark-3-0-remarkable-improvements-in-custom-aggregation-41dbaf725903" rel="nofollow noreferrer">https://medium.com/swlh/apache-spark-3-0-remarkable-improvements-in-custom-aggregation-41dbaf725903</a></p>
  <p>I have built and compiled a test implementation of a custom aggregator, but would now like to register and invoke it through PySpark and SparkSQL.</p>
  <p>I tried spark.udf.registerJavaUDAF ... but that seems only to work with the older style UDAF functions not the new Aggregators.</p>
  <p>How can I register a new Aggregator function written in Java through PySpark if at all possible? (I know how to pass the JAR to spark-submit etc the problem is the registration call).</p>