title: Unable to write csv file to S3
body: <pre><code>sc = SparkContext()
  glueContext = GlueContext(sc)
  spark = glueContext.spark_session
  
  df = spark.read.csv(&quot;s3://bucket1/file1.csv&quot;, header=True)
  df.show(5)
  
  df.write.mode(&quot;overwrite&quot;).csv(&quot;s3://bucket1/file2.csv&quot;, header=True)
  </code></pre>
  <p>The write to S3 does not work. I see the following statement in the logs,</p>
  <blockquote>
  <p>INFO DirectFileOutputCommitter: Direct Write: DISABLED</p>
  </blockquote>
  <p>As suggested in <a href="https://stackoverflow.com/questions/56432696/use-spark-fileoutputcommitter-algorithm-version-2-with-aws-glue">Use Spark fileoutputcommitter.algorithm.version=2 with AWS Glue</a>, I tried after adding the following,</p>
  <pre><code>sc._jsc.hadoopConfiguration().set(&quot;mapred.output.committer.class&quot;, &quot;org.apache.hadoop.mapred.FileOutputCommitter&quot;)
      sc._jsc.hadoopConfiguration().set(&quot;mapreduce.fileoutputcommitter.algorithm.version&quot;, &quot;2&quot;)
  </code></pre>
  <p>It did not help. What am i missing here?</p>