title: Why does spark-shell --master yarn-client fail (yet pyspark --master yarn seems to work)?
body: <p>I'm trying to run the spark shell on my Hadoop cluster via Yarn.
  I use </p>
  
  <ul>
  <li>Hadoop 2.4.1</li>
  <li>Spark 1.0.0</li>
  </ul>
  
  <p>My Hadoop cluster already works. In order to use Spark, I built Spark as described <a href="http://spark.apache.org/docs/latest/building-with-maven.html">here</a> : </p>
  
  <pre><code>mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.1 -DskipTests clean package
  </code></pre>
  
  <p>The compilation works fine, and I can run <code>spark-shell</code> without troubles. However, running it on yarn :</p>
  
  <pre><code>spark-shell --master yarn-client
  </code></pre>
  
  <p>gets me the following error :</p>
  
  <pre><code>14/07/07 11:30:32 INFO cluster.YarnClientSchedulerBackend: Application report from ASM:
           appMasterRpcPort: -1
           appStartTime: 1404725422955
           yarnAppState: ACCEPTED
  
  14/07/07 11:30:33 INFO cluster.YarnClientSchedulerBackend: Application report from ASM:
           appMasterRpcPort: -1
           appStartTime: 1404725422955
           yarnAppState: FAILED
  
  org.apache.spark.SparkException: Yarn application already ended,might be killed or not able to launch application master
  .
          at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApp(YarnClientSchedulerBackend.scala:105
  )
          at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:82)
          at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:136)
          at org.apache.spark.SparkContext.&lt;init&gt;(SparkContext.scala:318)
          at org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:957)
          at $iwC$$iwC.&lt;init&gt;(&lt;console&gt;:8)
          at $iwC.&lt;init&gt;(&lt;console&gt;:14)
          at &lt;init&gt;(&lt;console&gt;:16)
          at .&lt;init&gt;(&lt;console&gt;:20)
          at .&lt;clinit&gt;(&lt;console&gt;)
          at .&lt;init&gt;(&lt;console&gt;:7)
          at .&lt;clinit&gt;(&lt;console&gt;)
          at $print(&lt;console&gt;)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:606)
          at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:788)
          at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1056)
          at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:614)
          at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:645)
          at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:609)
          at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:796)
          at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:841)
          at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:753)
          at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:121)
          at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:120)
          at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:263)
          at org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:120)
          at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:56)
          at org.apache.spark.repl.SparkILoop$$anonfun$process$1$$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:913)
          at org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:142)
          at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:56)
          at org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:104)
          at org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:56)
          at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:930)
          at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
          at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
          at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
          at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:884)
          at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:982)
          at org.apache.spark.repl.Main$.main(Main.scala:31)
          at org.apache.spark.repl.Main.main(Main.scala)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:606)
          at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:292)
          at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
          at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  </code></pre>
  
  <p>Spark manages to communicate with my cluster, but it doesn't work out.
  Another interesting thing is that I can access my cluster using <code>pyspark --master yarn</code>. However, I get the following warning </p>
  
  <pre><code>14/07/07 14:10:11 WARN cluster.YarnClientClusterScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
  </code></pre>
  
  <p>and an infinite computation time when doing something as simple as </p>
  
  <pre><code>sc.wholeTextFiles('hdfs://vm7x64.fr/').collect()
  </code></pre>
  
  <p>What may be causing this problem ?</p>