title: Spark on yarn jar upload problems
body: <p>I am trying to run a simple Map/Reduce java program using spark over yarn (Cloudera Hadoop 5.2 on CentOS). I have tried this 2 different ways. The first way is the following:</p>
  
  <pre><code>YARN_CONF_DIR=/usr/lib/hadoop-yarn/etc/hadoop/; 
  /var/tmp/spark/spark-1.4.0-bin-hadoop2.4/bin/spark-submit --class MRContainer --master yarn-cluster --jars /var/tmp/spark/spark-1.4.0-bin-hadoop2.4/lib/spark-assembly-1.4.0-hadoop2.4.0.jar  simplemr.jar
  </code></pre>
  
  <p>This method gives the following error:</p>
  
  <blockquote>
    <p>diagnostics: Application application_1434177111261_0007 failed 2 times
    due to AM Container for appattempt_1434177111261_0007_000002 exited
    with  exitCode: -1000 due to: Resource
    hdfs://kc1ltcld29:9000/user/myuser/.sparkStaging/application_1434177111261_0007/spark-assembly-1.4.0-hadoop2.4.0.jar
    changed on src filesystem (expected 1434549639128, was 1434549642191</p>
  </blockquote>
  
  <p>Then I tried without the --jars:</p>
  
  <pre><code>YARN_CONF_DIR=/usr/lib/hadoop-yarn/etc/hadoop/; 
  /var/tmp/spark/spark-1.4.0-bin-hadoop2.4/bin/spark-submit --class MRContainer --master yarn-cluster simplemr.jar
  </code></pre>
  
  <blockquote>
    <p>diagnostics: Application application_1434177111261_0008 failed 2 times
    due to AM Container for appattempt_1434177111261_0008_000002 exited
    with  exitCode: -1000 due to: File does not exist:
    hdfs://kc1ltcld29:9000/user/myuser/.sparkStaging/application_1434177111261_0008/spark-assembly-1.4.0-hadoop2.4.0.jar
    .Failing this attempt.. Failing the application.
             ApplicationMaster host: N/A
             ApplicationMaster RPC port: -1
             queue: root.myuser
             start time: 1434549879649
             final status: FAILED
             tracking URL: <a href="http://kc1ltcld29:8088/cluster/app/application_1434177111261_0008" rel="nofollow">http://kc1ltcld29:8088/cluster/app/application_1434177111261_0008</a>
             user: myuser Exception in thread "main" org.apache.spark.SparkException: Application
    application_1434177111261_0008 finished with failed status
            at org.apache.spark.deploy.yarn.Client.run(Client.scala:841)
            at org.apache.spark.deploy.yarn.Client$.main(Client.scala:867)
            at org.apache.spark.deploy.yarn.Client.main(Client.scala)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:601)
            at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)
            at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)
            at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
            at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)
            at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 15/06/17 10:04:57 INFO util.Utils: Shutdown hook called 15/06/17
    10:04:57 INFO util.Utils: Deleting directory
    /tmp/spark-2aca3f35-abf1-4e21-a10e-4778a039d0f4</p>
  </blockquote>
  
  <p>I tried deleting all the .jars from hdfs://users//.sparkStaging and resubmitting but that didn't help. </p>