title: Py4JJavaError when trying to write pyspark DataFrame to parquet
body: <p>I wanted to convert a large .csv vile into .parquet format using pyspark.</p>
  
  <p>I am using python 3. I tried changing the codec used for compression, as suggested in a similar thread, but still the same error.</p>
  
  <p>This is the code I am using to read the file and save it into parquet format.</p>
  
  <pre class="lang-py prettyprint-override"><code>spark.conf.set("spark.sql.parquet.compression.codec", "gzip")
  
  df.write.parquet("adobe20180615.parquet")
  </code></pre>
  
  <p>I get the following error:</p>
  
  <pre><code>Py4JJavaError: An error occurred while calling o1071.parquet.
  : org.apache.spark.SparkException: Job aborted.
      at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:196)
      at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:159)
      at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)
      at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)
      at org.apache.spark.sql.execution.command.DataWritingCommandExec.doExecute(commands.scala:122)
      at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
      at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
      at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
      at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
      at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
      at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
      at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
      at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
      at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668)
      at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:668)
      at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
      at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
      at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
      at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:668)
      at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:276)
      at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:270)
      at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:228)
      at org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:557)
      at sun.reflect.GeneratedMethodAccessor86.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      at java.lang.reflect.Method.invoke(Unknown Source)
      at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
      at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
      at py4j.Gateway.invoke(Gateway.java:282)
      at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
      at py4j.commands.CallCommand.execute(CallCommand.java:79)
      at py4j.GatewayConnection.run(GatewayConnection.java:238)
      at java.lang.Thread.run(Unknown Source)
  Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 41.0 failed 1 times, most recent failure: Lost task 0.0 in stage 41.0 (TID 146, localhost, executor driver): java.io.IOException: (null) entry in command string: null chmod 0644 C:\Users\b35884\Documents\Python Scripts\Spark\adobe.parquet\_temporary\0\_temporary\attempt_20190411174312_0041_m_000000_0\part-00000-e2381e5d-0a9d-407e-8bcb-52d589f7569a-c000.gz.parquet
      at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:770)
      at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)
      at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)
      at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)
      at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.&lt;init&gt;(RawLocalFileSystem.java:225)
      at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.&lt;init&gt;(RawLocalFileSystem.java:209)
      at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
      at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)
      at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
      at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.&lt;init&gt;(ChecksumFileSystem.java:398)
      at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)
      at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)
      at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
      at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
      at org.apache.parquet.hadoop.util.HadoopOutputFile.create(HadoopOutputFile.java:74)
      at org.apache.parquet.hadoop.ParquetFileWriter.&lt;init&gt;(ParquetFileWriter.java:248)
      at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:390)
      at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:349)
      at org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.&lt;init&gt;(ParquetOutputWriter.scala:37)
      at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:151)
      at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:120)
      at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.&lt;init&gt;(FileFormatDataWriter.scala:108)
      at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:233)
      at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)
      at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:168)
      at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
      at org.apache.spark.scheduler.Task.run(Task.scala:121)
      at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
      at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at java.lang.Thread.run(Unknown Source)
  
  Driver stacktrace:
      at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1887)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1875)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1874)
      at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
      at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
      at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1874)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
      at scala.Option.foreach(Option.scala:257)
      at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:926)
      at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2108)
      at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2057)
      at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2046)
      at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
      at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:737)
      at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)
      at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:166)
      ... 32 more
  Caused by: java.io.IOException: (null) entry in command string: null chmod 0644 C:\Users\b35884\Documents\Python Scripts\Spark\adobe.parquet\_temporary\0\_temporary\attempt_20190411174312_0041_m_000000_0\part-00000-e2381e5d-0a9d-407e-8bcb-52d589f7569a-c000.gz.parquet
      at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:770)
      at org.apache.hadoop.util.Shell.execCommand(Shell.java:866)
      at org.apache.hadoop.util.Shell.execCommand(Shell.java:849)
      at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:733)
      at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.&lt;init&gt;(RawLocalFileSystem.java:225)
      at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.&lt;init&gt;(RawLocalFileSystem.java:209)
      at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
      at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)
      at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
      at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.&lt;init&gt;(ChecksumFileSystem.java:398)
      at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)
      at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)
      at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
      at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
      at org.apache.parquet.hadoop.util.HadoopOutputFile.create(HadoopOutputFile.java:74)
      at org.apache.parquet.hadoop.ParquetFileWriter.&lt;init&gt;(ParquetFileWriter.java:248)
      at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:390)
      at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:349)
      at org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.&lt;init&gt;(ParquetOutputWriter.scala:37)
      at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:151)
      at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:120)
      at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.&lt;init&gt;(FileFormatDataWriter.scala:108)
      at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:233)
      at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:169)
      at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:168)
      at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
      at org.apache.spark.scheduler.Task.run(Task.scala:121)
      at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
      at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      ... 1 more
  </code></pre>