title: Deserialization of task fails only in test runner
body: <p>I am encountering a deserialization issue that is only showing when I run my code via a test runner. Running an assembled uberjar (with AOT compilation) does not show this behavior and neither does running the same code from a REPL. The test runners that have tried are: <a href="https://github.com/cognitect-labs/test-runner" rel="nofollow noreferrer">cognitect/test-runner</a> and <a href="https://github.com/lambdaisland/kaocha" rel="nofollow noreferrer">lambdaisland/kaocha</a>.</p>
  <p>My general question is: Why would serialization behave differently when in a test runner versus the REPL or uberjar?</p>
  <p>I initially suspected that AOT compilation was causing the difference, but I would expect to see the exception in the REPL as well.</p>
  <p>Below is the relevant parts of the stack exception stack trace:</p>
  <pre><code>java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in instance of org.apache.spark.rdd.MapPartitionsRDD
      at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
      at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1405)
      ... trimmed for easier reading ...
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
      at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
      at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
      at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:83)
      at org.apache.spark.scheduler.Task.run(Task.scala:131)
      at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:497)
      at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:500)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
  </code></pre>
  <hr />
  <p>More details in case it is helpful: My Clojure application uses Spark via Scala/Java interop. The deserialization exception happens when an executor/worker processes receives a new task in the form of a <code>MapPartitionsRDD</code> object.</p>
  <p>Below is a simple <code>deftype</code> I created that extends the <code>MapFunction</code> interface provided by Spark. This interface allow for the serialization and broadcasting of a function to worker processes and is leveraged by <code>MapPartitionsRDD</code> although I am not sure how.</p>
  <pre class="lang-clj prettyprint-override"><code>(deftype CljMapFunction [func]
    org.apache.spark.api.java.function.MapFunction
    (call [_ var1]
      (-&gt; var1 row/row-&gt;map func row/to-row))) ; The row/* functions are my own.
  </code></pre>
  <p>When I pass an instance of <code>CljMapFunction</code> to <a href="https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L2799-L2809" rel="nofollow noreferrer">the java-specific <code>map</code> method of a spark Dataset</a> method, I see the above exception if the code is executed by a test runner. Again, both assembled uberjars and running form the REPL behave as expected.</p>