title: IndexError: list index out of range Error while writing to parquet or csv file
body: <p>All,</p>
  
  <p>I'm trying to write a pyspark df to either csv or parquet using these commands.</p>
  
  <pre><code>df.write.save("df-0418.tsv",format = "csv",header = 'true', delimiter = '\t')
  
  df.write.parquet("df-0418.parquet")
  </code></pre>
  
  <p>and I'm getting the following error everytime. The df has about 14 columns and 165K rows. I'm using Spark 2.0 standalone on a Windows professional 7 machine.</p>
  
  <p>Getting the following error. Any ideas on how to resolve this:</p>
  
  <pre><code>[Stage 12:==================================&gt;                   (128 + 8) / 200]16/08/23 12:57:58 ERROR PythonRunner: Python worker exited unexpectedly (cr
  hed)
  org.apache.spark.api.python.PythonException: Traceback (most recent call last):
    File "C:\Users\yrxt028\Downloads\spark-2.0.0\spark-2.0.0\python\lib\pyspark.zip\pyspark\worker.py", line 172, in main
    File "C:\Users\yrxt028\Downloads\spark-2.0.0\spark-2.0.0\python\lib\pyspark.zip\pyspark\worker.py", line 167, in process
    File "C:\Users\yrxt028\Downloads\spark-2.0.0\spark-2.0.0\python\lib\pyspark.zip\pyspark\worker.py", line 106, in &lt;lambda&gt;
    File "&lt;string&gt;", line 1, in &lt;lambda&gt;
    File "C:\Users\yrxt028\Downloads\spark-2.0.0\spark-2.0.0\python\lib\pyspark.zip\pyspark\worker.py", line 70, in &lt;lambda&gt;
    File "&lt;stdin&gt;", line 8, in eventSplit
  IndexError: list index out of range
  
          at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:193)
          at org.apache.spark.api.python.PythonRunner$$anon$1.&lt;init&gt;(PythonRDD.scala:234)
          at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:152)
          at org.apache.spark.sql.execution.python.BatchEvalPythonExec$$anonfun$doExecute$1.apply(BatchEvalPythonExec.scala:124)
          at org.apache.spark.sql.execution.python.BatchEvalPythonExec$$anonfun$doExecute$1.apply(BatchEvalPythonExec.scala:68)
          at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:766)
          at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:766)
          at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
          at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
          at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
          at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
          at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
          at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
          at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
          at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
          at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
          at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
          at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
          at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
          at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
          at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
          at org.apache.spark.scheduler.Task.run(Task.scala:85)
          at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
  16/08/23 12:57:58 ERROR PythonRunner: This may have been caused by a prior exception:
  java.net.SocketException: Connection reset by peer: socket write error
          at java.net.SocketOutputStream.socketWrite0(Native Method)
          at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
          at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
          at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
          at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
          at java.io.DataOutputStream.write(DataOutputStream.java:107)
          at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
          at org.apache.spark.api.python.PythonRDD$.org$apache$spark$api$python$PythonRDD$$write$1(PythonRDD.scala:492)
          at org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:504)
          at org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:504)
          at scala.collection.Iterator$class.foreach(Iterator.scala:893)
          at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
          at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:504)
          at org.apache.spark.api.python.PythonRunner$WriterThread$$anonfun$run$3.apply(PythonRDD.scala:328)
          at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1857)
          at org.apache.spark.api.python.PythonRunner$WriterThread.run(PythonRDD.scala:269)
  16/08/23 12:57:58 ERROR Executor: Exception in task 131.0 in stage 12.0 (TID 766)
  java.net.SocketException: Connection reset by peer: socket write error
          at java.net.SocketOutputStream.socketWrite0(Native Method)
          at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
          at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
          at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
          at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
          at java.io.DataOutputStream.write(DataOutputStream.java:107)
          at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
          at org.apache.spark.api.python.PythonRDD$.org$apache$spark$api$python$PythonRDD$$write$1(PythonRDD.scala:492)
          at org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:504)
          at org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:504)
          at scala.collection.Iterator$class.foreach(Iterator.scala:893)
          at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
          at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:504)
          at org.apache.spark.api.python.PythonRunner$WriterThread$$anonfun$run$3.apply(PythonRDD.scala:328)
          at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1857)
          at org.apache.spark.api.python.PythonRunner$WriterThread.run(PythonRDD.scala:269)
  16/08/23 12:57:58 WARN TaskSetManager: Lost task 131.0 in stage 12.0 (TID 766, localhost): java.net.SocketException: Connection reset by peer: socket write
  rror
          at java.net.SocketOutputStream.socketWrite0(Native Method)
          at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
          at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
          at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
          at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
          at java.io.DataOutputStream.write(DataOutputStream.java:107)
          at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
          at org.apache.spark.api.python.PythonRDD$.org$apache$spark$api$python$PythonRDD$$write$1(PythonRDD.scala:492)
          at org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:504)
          at org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:504)
          at scala.collection.Iterator$class.foreach(Iterator.scala:893)
          at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
          at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:504)
          at org.apache.spark.api.python.PythonRunner$WriterThread$$anonfun$run$3.apply(PythonRDD.scala:328)
          at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1857)
          at org.apache.spark.api.python.PythonRunner$WriterThread.run(PythonRDD.scala:269)
  
  16/08/23 12:57:58 ERROR TaskSetManager: Task 131 in stage 12.0 failed 1 times; aborting job
  16/08/23 12:57:58 ERROR InsertIntoHadoopFsRelationCommand: Aborting job.
  org.apache.spark.SparkException: Job aborted due to stage failure: Task 131 in stage 12.0 failed 1 times, most recent failure: Lost task 131.0 in stage 12.
  (TID 766, localhost): java.net.SocketException: Connection reset by peer: socket write error
          at java.net.SocketOutputStream.socketWrite0(Native Method)
          at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
          at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
          at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
          at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
          at java.io.DataOutputStream.write(DataOutputStream.java:107)
          at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
          at org.apache.spark.api.python.PythonRDD$.org$apache$spark$api$python$PythonRDD$$write$1(PythonRDD.scala:492)
          at org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:504)
          at org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:504)
          at scala.collection.Iterator$class.foreach(Iterator.scala:893)
          at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
          at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:504)
          at org.apache.spark.api.python.PythonRunner$WriterThread$$anonfun$run$3.apply(PythonRDD.scala:328)
          at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1857)
          at org.apache.spark.api.python.PythonRunner$WriterThread.run(PythonRDD.scala:269)
  
  Driver stacktrace:
          at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1450)
          at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1438)
          at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1437)
          at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
          at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
          at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1437)
          at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)
          at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)
          at scala.Option.foreach(Option.scala:257)
          at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:811)
          at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1659)
          at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1618)
          at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1607)
          at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
          at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:632)
          at org.apache.spark.SparkContext.runJob(SparkContext.scala:1871)
          at org.apache.spark.SparkContext.runJob(SparkContext.scala:1884)
          at org.apache.spark.SparkContext.runJob(SparkContext.scala:1904)
          at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand$$anonfun$run$1.apply$mcV$sp(InsertIntoHadoopFsRelationCommand.scala
  43)
          at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand$$anonfun$run$1.apply(InsertIntoHadoopFsRelationCommand.scala:115)
          at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand$$anonfun$run$1.apply(InsertIntoHadoopFsRelationCommand.scala:115)
          at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:57)
          at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:115)
          at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:60)
          at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:58)
          at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
          at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
          at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
          at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:136)
          at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
          at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133)
          at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114)
          at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:86)
          at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:86)
          at org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:487)
          at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:211)
          at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:194)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:497)
          at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)
          at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
          at py4j.Gateway.invoke(Gateway.java:280)
          at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:128)
          at py4j.commands.CallCommand.execute(CallCommand.java:79)
          at py4j.GatewayConnection.run(GatewayConnection.java:211)
          at java.lang.Thread.run(Thread.java:745)
  Caused by: java.net.SocketException: Connection reset by peer: socket write error
          at java.net.SocketOutputStream.socketWrite0(Native Method)
          at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
          at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
          at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
          at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
          at java.io.DataOutputStream.write(DataOutputStream.java:107)
          at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
          at org.apache.spark.api.python.PythonRDD$.org$apache$spark$api$python$PythonRDD$$write$1(PythonRDD.scala:492)
          at org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:504)
          at org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:504)
          at scala.collection.Iterator$class.foreach(Iterator.scala:893)
          at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
          at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:504)
          at org.apache.spark.api.python.PythonRunner$WriterThread$$anonfun$run$3.apply(PythonRDD.scala:328)
          at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1857)
          at org.apache.spark.api.python.PythonRunner$WriterThread.run(PythonRDD.scala:269)
  16/08/23 12:57:58 WARN TaskSetManager: Lost task 136.0 in stage 12.0 (TID 771, localhost): TaskKilled (killed intentionally)
  16/08/23 12:57:58 ERROR DefaultWriterContainer: Job job_201608231246_0000 aborted.
  Traceback (most recent call last):
    File "&lt;stdin&gt;", line 1, in &lt;module&gt;
    File "C:\Users\yrxt028\Downloads\spark-2.0.0\spark-2.0.0\python\pyspark\sql\readwriter.py", line 532, in save
  16/08/23 12:57:58 WARN PythonRunner: Incomplete task interrupted: Attempting to kill Python Worker
      self._jwrite.save(path)
    File "C:\Users\yrxt028\Downloads\spark-2.0.0\spark-2.0.0\python\lib\py4j-0.10.1-src.zip\py4j\java_gateway.py", line 933, in __call__
    File "C:\Users\yrxt028\Downloads\spark-2.0.0\spark-2.0.0\python\pyspark\sql\utils.py", line 63, in deco
      return f(*a, **kw)
    File "C:\Users\yrxt028\Downloads\spark-2.0.0\spark-2.0.0\python\lib\py4j-0.10.1-src.zip\py4j\protocol.py", line 312, in get_return_value
  py4j.protocol.Py4JJavaError: An error occurred while calling o368.save.
  : org.apache.spark.SparkException: Job aborted.
          at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand$$anonfun$run$1.apply$mcV$sp(InsertIntoHadoopFsRelationCommand.scala
  49)
          at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand$$anonfun$run$1.apply(InsertIntoHadoopFsRelationCommand.scala:115)
          at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand$$anonfun$run$1.apply(InsertIntoHadoopFsRelationCommand.scala:115)
          at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:57)
          at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:115)
          at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:60)
          at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:58)
          at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
          at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
          at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
          at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:136)
          at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
          at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133)
          at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114)
          at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:86)
          at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:86)
          at org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:487)
          at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:211)
          at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:194)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:497)
          at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)
          at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
          at py4j.Gateway.invoke(Gateway.java:280)
          at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:128)
          at py4j.commands.CallCommand.execute(CallCommand.java:79)
          at py4j.GatewayConnection.run(GatewayConnection.java:211)
          at java.lang.Thread.run(Thread.java:745)
  Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 131 in stage 12.0 failed 1 times, most recent failure: Lost task 131.0 i
  stage 12.0 (TID 766, localhost): java.net.SocketException: Connection reset by peer: socket write error
          at java.net.SocketOutputStream.socketWrite0(Native Method)
          at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
          at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
          at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
          at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
          at java.io.DataOutputStream.write(DataOutputStream.java:107)
          at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
          at org.apache.spark.api.python.PythonRDD$.org$apache$spark$api$python$PythonRDD$$write$1(PythonRDD.scala:492)
          at org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:504)
          at org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:504)
          at scala.collection.Iterator$class.foreach(Iterator.scala:893)
          at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
          at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:504)
          at org.apache.spark.api.python.PythonRunner$WriterThread$$anonfun$run$3.apply(PythonRDD.scala:328)
          at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1857)
          at org.apache.spark.api.python.PythonRunner$WriterThread.run(PythonRDD.scala:269)
  
  Driver stacktrace:
          at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1450)
          at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1438)
          at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1437)
          at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
          at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
          at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1437)
          at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)
          at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)
          at scala.Option.foreach(Option.scala:257)
          at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:811)
          at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1659)
          at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1618)
          at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1607)
          at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
          at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:632)
          at org.apache.spark.SparkContext.runJob(SparkContext.scala:1871)
          at org.apache.spark.SparkContext.runJob(SparkContext.scala:1884)
          at org.apache.spark.SparkContext.runJob(SparkContext.scala:1904)
          at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand$$anonfun$run$1.apply$mcV$sp(InsertIntoHadoopFsRelationCommand.scala
  43)
          ... 29 more
  Caused by: java.net.SocketException: Connection reset by peer: socket write error
          at java.net.SocketOutputStream.socketWrite0(Native Method)
          at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
          at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
          at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
          at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
          at java.io.DataOutputStream.write(DataOutputStream.java:107)
          at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
          at org.apache.spark.api.python.PythonRDD$.org$apache$spark$api$python$PythonRDD$$write$1(PythonRDD.scala:492)
          at org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:504)
          at org.apache.spark.api.python.PythonRDD$$anonfun$writeIteratorToStream$1.apply(PythonRDD.scala:504)
          at scala.collection.Iterator$class.foreach(Iterator.scala:893)
          at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
          at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:504)
          at org.apache.spark.api.python.PythonRunner$WriterThread$$anonfun$run$3.apply(PythonRDD.scala:328)
          at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1857)
          at org.apache.spark.api.python.PythonRunner$WriterThread.run(PythonRDD.scala:269)
  
  &gt;&gt;&gt; 16/08/23 12:57:59 WARN PythonRunner: Incomplete task interrupted: Attempting to kill Python Worker
  16/08/23 12:57:59 WARN TaskSetManager: Lost task 130.0 in stage 12.0 (TID 765, localhost): TaskKilled (killed intentionally)
  16/08/23 12:57:59 WARN PythonRunner: Incomplete task interrupted: Attempting to kill Python Worker
  16/08/23 12:57:59 WARN PythonRunner: Incomplete task interrupted: Attempting to kill Python Worker
  16/08/23 12:57:59 WARN TaskSetManager: Lost task 135.0 in stage 12.0 (TID 770, localhost): TaskKilled (killed intentionally)
  16/08/23 12:57:59 WARN TaskSetManager: Lost task 134.0 in stage 12.0 (TID 769, localhost): TaskKilled (killed intentionally)
  16/08/23 12:57:59 WARN TaskSetManager: Lost task 128.0 in stage 12.0 (TID 763, localhost): TaskKilled (killed intentionally)
  16/08/23 12:57:59 WARN TaskSetManager: Lost task 133.0 in stage 12.0 (TID 768, localhost): TaskKilled (killed intentionally)
  16/08/23 12:57:59 WARN TaskSetManager: Lost task 132.0 in stage 12.0 (TID 767, localhost): TaskKilled (killed intentionally)
  </code></pre>
  
  <p>`</p>