title: Write to a file in S3 using Spark on EMR
body: <p>I use the following Scala code to create a text file in S3, with Apache Spark on AWS EMR.</p>
  
  <pre><code>def createS3OutputFile() {
      val conf = new SparkConf().setAppName("Spark Pi")
      val spark = new SparkContext(conf)
      // use s3n !
      val outputFileUri = s"s3n://$s3Bucket/emr-output/test-3.txt"
      val arr = Array("hello", "World", "!")
      val rdd = spark.parallelize(arr)
      rdd.saveAsTextFile(outputFileUri)
      spark.stop()
    }
  
  def main(args: Array[String]): Unit = {
      createS3OutputFile()
    }
  </code></pre>
  
  <p>I create a fat JAR and upload it to S3. I then SSH into the cluster master and run the code with:</p>
  
  <pre><code>spark-submit \
      --deploy-mode cluster \
      --class "$class_name" \
      "s3://$s3_bucket/$app_s3_key"
  </code></pre>
  
  <p>I am seeing this in the S3 console: instead of files there are folders.</p>
  
  <p><a href="https://i.stack.imgur.com/86bBh.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/86bBh.png" alt="enter image description here"></a></p>
  
  <p>Each folder (for example test-3.txt) contains a long list of block files. Picture below:</p>
  
  <p><a href="https://i.stack.imgur.com/NpQjQ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/NpQjQ.png" alt="enter image description here"></a></p>
  
  <p>How do I output a simple text file to S3 as the output of my Spark job?</p>