title: Exception on PySpark when write to Parquet
body: <p>I am working with huge data at JSON format that is coverted to parquet using pySpark. In gral, all it is ok, but currently I am having issue on some transformation. But I could not debug or I don;t how overpass. I could transform a lot of data, but It seem issue but I could not detect or skip</p>
  <pre><code>    pc_attrs = json.map(js_loads).filter(shrink_).map(
          shrink_attrs).map(convert_time_pc_atts)
  
      s3aRdd = spark_session.createDataFrame(pc_attrs, procAttrsSchema)
      s3aRdd.write.mode('overwrite').partitionBy(&quot;day&quot;,&quot;tenantId&quot;).parquet(
          s3_table_parquet)```
  
  
  The exception:
  
  ```Caused by: org.apache.spark.api.python.PythonException: Traceback (most recent call last):
    File &quot;/mnt/yarn/usercache/root/appcache/application_1595414885588_0004/container_1595414885588_0004_01_000228/pyspark.zip/pyspark/worker.py&quot;, line 377, in main
      process()
    File &quot;/mnt/yarn/usercache/root/appcache/application_1595414885588_0004/container_1595414885588_0004_01_000228/pyspark.zip/pyspark/worker.py&quot;, line 372, in process
      serializer.dump_stream(func(split_index, iterator), outfile)
    File &quot;/mnt/yarn/usercache/root/appcache/application_1595414885588_0004/container_1595414885588_0004_01_000228/pyspark.zip/pyspark/serializers.py&quot;, line 393, in dump_stream
      vs = list(itertools.islice(iterator, batch))
    File &quot;/mnt/yarn/usercache/root/appcache/application_1595414885588_0004/container_1595414885588_0004_01_000228/pyspark.zip/pyspark/util.py&quot;, line 99, in wrapper
      return f(*args, **kwargs)
    File &quot;/usr/lib64/python3.6/json/__init__.py&quot;, line 354, in loads
      return _default_decoder.decode(s)
    File &quot;/usr/lib64/python3.6/json/decoder.py&quot;, line 339, in decode
      obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    File &quot;/usr/lib64/python3.6/json/decoder.py&quot;, line 355, in raw_decode
      obj, end = self.scan_once(s, idx)
  json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 2649 (char 2648)
  
      at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:452)
      at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:588)
      at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:571)
      at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:406)
      at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
      at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
      at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
      at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
      at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
      at org.apache.spark.sql.execution.UnsafeExternalRowSorter.sort(UnsafeExternalRowSorter.java:216)
      at org.apache.spark.sql.execution.SortExec$$anonfun$1.apply(SortExec.scala:108)
      at org.apache.spark.sql.execution.SortExec$$anonfun$1.apply(SortExec.scala:101)
      at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)
      at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)
      at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
      at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
      at org.apache.spark.scheduler.Task.run(Task.scala:121)
      at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
      at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      ... 1 more```
  </code></pre>