title: AWS Glue RDD.saveAsTextFile() raises Class org.apache.hadoop.mapred.DirectOutputCommitter not found
body: <p>I'm creating the simple ETL that reads a billion of files and re-partition them (in other words, compact to smaller amount for further processing).</p>
  <p>Simple AWS Glue application:</p>
  <pre><code>import org.apache.spark.SparkContext
  
  object Hello {
    def main(sysArgs: Array[String]) {
      val spark: SparkContext = new SparkContext()
      val input_path =  &quot;s3a://my-bucket-name/input/*&quot;
      val output_path = &quot;s3a://my-bucket-name/output/*&quot;
      val num_partitions = 5
      val ingestRDD = spark.textFile(input_path)
      ingestRDD.repartition(num_partitions).saveAsTextFile(output_path)    
    }
  }
  </code></pre>
  <p>raises the following traceback:</p>
  <pre><code>ERROR [main] glue.ProcessLauncher (Logging.scala:logError(70)): Exception in User Class: java.lang.RuntimeException : java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.mapred.DirectOutputCommitter not found
  org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2401)
  org.apache.hadoop.mapred.JobConf.getOutputCommitter(JobConf.java:725)
  org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply$mcV$sp(PairRDDFunctions.scala:1048)
  org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply(PairRDDFunctions.scala:1032)
  org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply(PairRDDFunctions.scala:1032)
  org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
  org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
  org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:1032)
  org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply$mcV$sp(PairRDDFunctions.scala:958)
  org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply(PairRDDFunctions.scala:958)
  org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply(PairRDDFunctions.scala:958)
  org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
  org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
  org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:957)
  org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply$mcV$sp(RDD.scala:1499)
  org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply(RDD.scala:1478)
  org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply(RDD.scala:1478)
  org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
  org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
  org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:1478)
  Hello$.main(hello_world_parallel_rdd_scala:18)
  Hello.main(hello_world_parallel_rdd_scala)
  sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  java.lang.reflect.Method.invoke(Method.java:498)
  com.amazonaws.services.glue.SparkProcessLauncherPlugin$class.invoke(ProcessLauncher.scala:38)
  com.amazonaws.services.glue.ProcessLauncher$$anon$1.invoke(ProcessLauncher.scala:67)
  com.amazonaws.services.glue.ProcessLauncher.launch(ProcessLauncher.scala:108)
  com.amazonaws.services.glue.ProcessLauncher$.main(ProcessLauncher.scala:21)
  com.amazonaws.services.glue.ProcessLauncher.main(ProcessLauncher.scala)
  </code></pre>
  <p>At the same time this code code works in Local Environment, in Cluster and in EMR Cluster.</p>