title: PySpark EMR step fails with exit code 1
body: <p>I'm learning to use AWS EMR for the first time to submit my Spark jobs. The script I'm using is very short (<strong>restaurant.py</strong>):</p>
  
  <pre><code>from pyspark import SparkContext, SQLContext
  from pyspark.sql import SparkSession
  
  class SparkRawConsumer:
  
  def __init__(self):
      self.sparkContext = SparkContext.getOrCreate()
  
      self.sparkContext.setLogLevel("ERROR")
      self.sqlContext = SQLContext(self.sparkContext)
      self.df = self.sqlContext.read.format('com.databricks.spark.csv').options(header='true', inferschema='true').load('zomato.csv')
  
  
  if __name__ == "__main__":
      sparkConsumer = SparkRawConsumer()
      print(sparkConsumer.df.count())
      sparkConsumer.df.groupBy("City").agg({"Average Cost for two": "avg", "Aggregate rating": "avg"})
  </code></pre>
  
  <p>I used the AWS GUI to submit my step, but the CLI export is </p>
  
  <pre><code>spark-submit --deploy-mode cluster s3://data-pipeline-testing-yu-chen/dependencies/restaurant.py -files s3://data-pipeline-testing-yu-chen/dependencies/zomato.csv
  </code></pre>
  
  <p>However, the step will run for a few minutes, and then return an exit code of 1. I'm pretty confused what exactly is going on, and finding it difficult to interpret the output of my syserr:</p>
  
  <pre><code>18/07/28 06:40:10 INFO Client: Application report for application_1532756827478_0012 (state: RUNNING)
  18/07/28 06:40:11 INFO Client: Application report for application_1532756827478_0012 (state: RUNNING)
  18/07/28 06:40:12 INFO Client: Application report for application_1532756827478_0012 (state: RUNNING)
  18/07/28 06:40:13 INFO Client: Application report for application_1532756827478_0012 (state: FINISHED)
  18/07/28 06:40:13 INFO Client: 
       client token: N/A
       diagnostics: User application exited with status 1
       ApplicationMaster host: myip
       ApplicationMaster RPC port: 0
       queue: default
       start time: 1532759825922
       final status: FAILED
       tracking URL: http://myip.compute.internal:20888/proxy/application_1532756827478_0012/
       user: hadoop
  18/07/28 06:40:13 INFO Client: Deleted staging directory hdfs://myip.compute.internal:8020/user/hadoop/.sparkStaging/application_1532756827478_0012
  Exception in thread "main" org.apache.spark.SparkException: Application application_1532756827478_0012 finished with failed status
      at org.apache.spark.deploy.yarn.Client.run(Client.scala:1165)
      at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1520)
      at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
      at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
      at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  18/07/28 06:40:13 INFO ShutdownHookManager: Shutdown hook called
  18/07/28 06:40:13 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-dedwd323x
  18/07/28 06:40:13 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-dedwd323x
  Command exiting with ret '1'
  </code></pre>
  
  <p>I am able to run the script by SSH-ing into my master instance, and then issuing <code>spark-submit restaurant.py</code>. I've loaded a CSV file into my master instance by using:</p>
  
  <pre><code>[hadoop@my-ip ~]$ aws s3 sync s3://data-pipeline-testing-yu-chen/dependencies/ .
  </code></pre>
  
  <p>Then I load my <code>restaurant.csv</code> file into HDFS: </p>
  
  <pre><code>hadoop fs -put zomato.csv ./zomato.csv
  </code></pre>
  
  <p>My guess is that the <code>-files</code> option I am passing in isn't getting used the way I intend it to be used, but I'm really at a loss for how to interpret the console output and begin debugging.</p>