title: User class threw exception: com.esotericsoftware.kryo.KryoException: Buffer overflow. Available: 0, required: 1
body: <p>I'm doing following operation on pyspark,</p>
  <pre><code>df = spark.read.table(&quot;pseudonymized_cdp_offline_store_snapshot_ssp_request_admonet_v3&quot;)
     req=spark.read.table(&quot;rtf_request_table_ssp_segment_2022_06_15_0&quot;).select(&quot;pseudonym_id&quot;).distinct().rdd.map(lambda x : x).collect()
  reqid = []
  for i in req:
     reqid.append(i['pseudonym_id'])
  df = df.filter(~(col(&quot;userId&quot;).isin(reqid)))
  df.show(truncate=False)
  </code></pre>
  <p>On df.show the error thrown is,</p>
  <pre><code>Py4JJavaError: An error occurred while calling o1441558.showString.
  : com.esotericsoftware.kryo.KryoException: Buffer overflow. Available: 0, required: 4
  Serialization trace:
  literalList (org.apache.orc.storage.ql.io.sarg.SearchArgumentImpl$PredicateLeafImpl)
  leaves (org.apache.orc.storage.ql.io.sarg.SearchArgumentImpl)
      at com.esotericsoftware.kryo.io.Output.require(Output.java:167)
      at com.esotericsoftware.kryo.io.Output.writeString_slow(Output.java:467)
      at com.esotericsoftware.kryo.io.Output.writeString(Output.java:368)
      at com.esotericsoftware.kryo.serializers.DefaultSerializers$StringSerializer.write(DefaultSerializers.java:195)
      at com.esotericsoftware.kryo.serializers.DefaultSerializers$StringSerializer.write(DefaultSerializers.java:188)
      at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651)
      at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100)
      at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
      at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
      at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:79)
      at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508)
      at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651)
      at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100)
      at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40)
      at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:575)
      at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:79)
      at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508)
      at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:557)
      at org.apache.orc.mapred.OrcInputFormat.setSearchArgument(OrcInputFormat.java:97)
      at org.apache.orc.mapreduce.OrcInputFormat.setSearchArgument(OrcInputFormat.java:57)
      at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat$$anonfun$buildReaderWithPartitionValues$1.apply(OrcFileFormat.scala:159)
      at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat$$anonfun$buildReaderWithPartitionValues$1.apply(OrcFileFormat.scala:158)
      at scala.Option.foreach(Option.scala:257)
      at org.apache.spark.sql.execution.datasources.orc.OrcFileFormat.buildReaderWithPartitionValues(OrcFileFormat.scala:158)
  </code></pre>
  <p>Data in df from read.table is not corrupted and I have verified that.</p>
  <p>I have checked few solutions on internet where they have suggested adding this spark config,
  spark.kryoserializer.buffer.max=2047
  But that didn't work for me.
  When I try the same thing on smaller dataframe it seems to work fine.</p>