title: Spark Jobs crashing with ExitCodeException exitCode=15
body: <p>I am running a very long spark job which crashes with the following error</p>
  
  <pre><code>Application application_1456200816465_347125 failed 2 times due to AM Container for appattempt_1456200816465_347125_000002 exited with exitCode: 15
  For more detailed output, check application tracking page:http://foo.com:8088/proxy/application_1456200816465_347125/Then, click on links to logs of each attempt.
  Diagnostics: Exception from container-launch.
  Container id: container_e24_1456200816465_347125_02_000001
  Exit code: 15
  Stack trace: ExitCodeException exitCode=15:
  at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
  at org.apache.hadoop.util.Shell.run(Shell.java:455)
  at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
  at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
  at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
  at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)
  Container exited with a non-zero exit code 15
  Failing this attempt. Failing the application.
  </code></pre>
  
  <p>I click on the link provided in the error message above and that shows me</p>
  
  <pre><code>java.io.IOException: Target log file already exists (hdfs://nameservice1/user/spark/applicationHistory/application_1456200816465_347125)
      at org.apache.spark.scheduler.EventLoggingListener.stop(EventLoggingListener.scala:201)
      at org.apache.spark.SparkContext$$anonfun$stop$5.apply(SparkContext.scala:1394)
      at org.apache.spark.SparkContext$$anonfun$stop$5.apply(SparkContext.scala:1394)
      at scala.Option.foreach(Option.scala:236)
      at org.apache.spark.SparkContext.stop(SparkContext.scala:1394)
      at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:107)
      at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
  </code></pre>
  
  <p>If I restart the job it works fine for 1 hour or so and then again fails with this error. Note that <code>hdfs://nameservice1/user/spark/applicationHistory/application_1456200816465_347125</code> is some system generated thing. this folder has nothing to do with my application.</p>
  
  <p>I searched the internet and many people got this error because they were setting the master to local in their code. This is how I initialize my spark context</p>
  
  <pre><code>val conf = new SparkConf().setAppName("Foo")
  val context = new SparkContext(conf)
  context.hadoopConfiguration.set("mapreduce.input.fileinputformat.input.dir.recursive","true")
  val sc = new SQLContext(context)
  </code></pre>
  
  <p>and I run my spark job like</p>
  
  <pre><code>sudo -u web nohup spark-submit --class com.abhi.Foo--master yarn-cluster 
  Foo-assembly-1.0.jar "2015-03-18" "2015-03-30" &gt; fn_output.txt 2&gt; fn_error.txt &amp;
  </code></pre>