title: Pyspark save to S3
body: <p>I am trying to save a large file to an Amazon S3 bucket.
  Following code works perfectly: </p>
  
  <pre><code>sqlContext.createDataFrame([('1', '4'), ('2', '5'), ('3', '6')], ["A", "B"]).select('A').repartition(1).write \
      .format("text") \
      .mode("overwrite") \
      .option("header", "false") \
      .option("codec", "gzip") \
      .save("s3n://BUCKETNAME/temp.txt")
  </code></pre>
  
  <p>saving my full dataframe however fails. with following error in my notebook:</p>
  
  <pre><code>Py4JJavaError: An error occurred while calling o1274.save.
  : org.apache.spark.SparkException: Job aborted.
      at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply$mcV$sp(InsertIntoHadoopFsRelation.scala:156)
      at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:108)
      at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation$$anonfun$run$1.apply(InsertIntoHadoopFsRelation.scala:108)
      at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:56)
      at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelation.run(InsertIntoHadoopFsRelation.scala:108)
      at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:58)
      at org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:56)
      at org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:70)
      at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:132)
      at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5.apply(SparkPlan.scala:130)
      at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
      at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:130)
      at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:55)
      at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:55)
      at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:256)
      at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:148)
      at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:139)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:497)
      at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
      at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381)
      at py4j.Gateway.invoke(Gateway.java:259)
      at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
      at py4j.commands.CallCommand.execute(CallCommand.java:79)
      at py4j.GatewayConnection.run(GatewayConnection.java:209)
      at java.lang.Thread.run(Thread.java:745)
  Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to java.util.Date
      at org.jets3t.service.model.StorageObject.getLastModifiedDate(StorageObject.java:376)
      at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:176)
  </code></pre>
  
  <p>In the spark application UI the job is described as succeeded.</p>
  
  <p>I have following configuration:</p>
  
  <pre><code>sc._jsc.hadoopConfiguration().set("fs.s3n.multipart.uploads.enabled", "true")
  </code></pre>
  
  <p>Trying to debug I tried the following, which works as it should...</p>
  
  <pre><code>sqlContext.createDataFrame(full_df.select('columnA').take(5),['columnA']).select('columnA').repartition(1).write \
      .format("text") \
      .mode("overwrite") \
      .option("header", "false") \
      .option("codec", "gzip") \
      .save("s3n://BUCKETNAME/temp.txt")
  </code></pre>
  
  <p>I found following link which seems to be about this problem, but I can't find a working package
  <a href="https://bitbucket.org/jmurty/jets3t/pull-requests/38/archiveorg-support/diff" rel="nofollow noreferrer">Jets3t</a></p>
  
  <p>Who can help with this mysterious error?</p>