title: ERROR: java.lang.IllegalStateException: User did not initialize spark context
body: <p><strong>Scala version: 2.11.12</strong></p>
  
  <p><strong>Spark version: 2.4.0</strong></p>
  
  <p><strong>emr-5.23.0</strong></p>
  
  <p>Get the following when running the below command to create an Amazon EMR cluster</p>
  
  <pre><code>spark-submit --class etl.SparkDataProcessor --master yarn --deploy-mode cluster --conf spark.yarn.appMasterEnv.ETL_NAME=foo --conf spark.yarn.appMasterEnv.ETL_SPARK_MASTER=yarn --conf spark.yarn.appMasterEnv.ETL_AWS_ACCESS_KEY_ID=123 --conf spark.yarn.appMasterEnv.ETL_AWS_SECRET_ACCESS_KEY=abc MY-Tool.jar
  </code></pre>
  
  <p><strong>Exception</strong></p>
  
  <pre><code>ERROR ApplicationMaster: Uncaught exception: 
  java.lang.IllegalStateException: User did not initialize spark context!
      at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:485)
      at org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$$runImpl(ApplicationMaster.scala:305)
      at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply$mcV$sp(ApplicationMaster.scala:245)
      at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply(ApplicationMaster.scala:245)
      at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$1.apply(ApplicationMaster.scala:245)
      at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:773)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
      at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:772)
      at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:244)
      at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:797)
      at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
  </code></pre>
  
  <p><strong>How I create my spark session (where sparkMaster = yarn)</strong></p>
  
  <pre><code>lazy val spark: SparkSession = {
      val logger: Logger = Logger.getLogger("etl");
      val sparkAppName = EnvConfig.ETL_NAME
      val sparkMaster = EnvConfig.ETL_SPARK_MASTER
  
      val sparkInstance = SparkSession
        .builder()
        .appName(sparkAppName)
        .master(sparkMaster)
        .getOrCreate()
  
      val hadoopConf = sparkInstance.sparkContext.hadoopConfiguration
      hadoopConf.set("fs.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
      hadoopConf.set("fs.s3a.access.key", EnvConfig.ETL_AWS_ACCESS_KEY_ID)
      hadoopConf.set("fs.s3a.secret.key", EnvConfig.ETL_AWS_SECRET_ACCESS_KEY)
  
      logger.info("Created My SparkSession")
      logger.info(s"Spark Application Name: $sparkAppName")
      logger.info(s"Spark Master: $sparkMaster")
  
      sparkInstance
    }
  </code></pre>
  
  <p>UPDATE:</p>
  
  <p>I determined that due to the application logic, in certain cases, we did not initialize the spark session. Because of this, it seems that when the cluster terminates, it also tries to do something with the session (perhaps close it) and is thus failing. Now that I have figured out this issue, the application runs but never actually completes. Currently, it seems to be hanging in a particular part involving spark when running in cluster mode:</p>
  
  <pre><code>val data: DataFrame = spark.read
        .option("header", "true")
        .option("inferSchema", "true")
        .csv(s"s3://$csvPath/$fileKey")
        .toDF()
  
  20/03/16 18:38:35 INFO Client: Application report for application_1584324418613_0031 (state: RUNNING)
  </code></pre>