title: setup/run spark (spark-shell) on yarn client mode
body: <p>I am trying to make spark-shell working with YARN, however when I try to run the code like below:</p>
  
  <pre><code>spark-shell \
    --master yarn \
    --deploy-mode client \
    --driver-memory 1g \
    --executor-memory 1g \
    --executor-cores 1
  </code></pre>
  
  <p>The stacktrace I got is:</p>
  
  <pre><code>   17/02/07 01:52:41 ERROR spark.SparkContext: Error initializing SparkContext.
  org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
      at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:85)
      at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62)
      at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:156)
      at org.apache.spark.SparkContext.&lt;init&gt;(SparkContext.scala:509)
      at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313)
      at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
      at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
      at scala.Option.getOrElse(Option.scala:121)
      at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
      at org.apache.spark.repl.Main$.createSparkSession(Main.scala:95)
      at $line3.$read$$iw$$iw.&lt;init&gt;(&lt;console&gt;:15)
      at $line3.$read$$iw.&lt;init&gt;(&lt;console&gt;:42)
      at $line3.$read.&lt;init&gt;(&lt;console&gt;:44)
      at $line3.$read$.&lt;init&gt;(&lt;console&gt;:48)
      at $line3.$read$.&lt;clinit&gt;(&lt;console&gt;)
      at $line3.$eval$.$print$lzycompute(&lt;console&gt;:7)
      at $line3.$eval$.$print(&lt;console&gt;:6)
      at $line3.$eval.$print(&lt;console&gt;)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786)
      at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1047)
      at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:638)
      at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:637)
      at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
      at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
      at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:637)
      at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:569)
      at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565)
      at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:807)
      at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:681)
      at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:395)
      at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply$mcV$sp(SparkILoop.scala:38)
      at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:37)
      at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:37)
      at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:214)
      at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:37)
      at org.apache.spark.repl.SparkILoop.loadFiles(SparkILoop.scala:105)
      at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:920)
      at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
      at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
      at scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97)
      at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:909)
      at org.apache.spark.repl.Main$.doMain(Main.scala:68)
      at org.apache.spark.repl.Main$.main(Main.scala:51)
      at org.apache.spark.repl.Main.main(Main.scala)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
      at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
      at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  17/02/07 01:52:42 ERROR client.TransportClient: Failed to send RPC 7605279734171920371 to /172.17.0.2:35136: java.nio.channels.ClosedChannelException
  java.nio.channels.ClosedChannelException
      at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
  17/02/07 01:52:42 ERROR cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Sending RequestExecutors(0,0,Map()) to AM was unsuccessful
  java.io.IOException: Failed to send RPC 7605279734171920371 to /172.17.0.2:35136: java.nio.channels.ClosedChannelException
      at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:249)
      at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:233)
      at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:514)
      at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:488)
      at io.netty.util.concurrent.DefaultPromise.access$000(DefaultPromise.java:34)
      at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:438)
      at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:408)
      at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:455)
      at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
      at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
      at java.lang.Thread.run(Thread.java:745)
  Caused by: java.nio.channels.ClosedChannelException
      at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
  17/02/07 01:52:42 ERROR util.Utils: Uncaught exception in thread main
  org.apache.spark.SparkException: Exception thrown in awaitResult
      at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77)
      at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:75)
      at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
      at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
      at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
      at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)
      at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83)
      at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.requestTotalExecutors(CoarseGrainedSchedulerBackend.scala:512)
      at org.apache.spark.scheduler.cluster.YarnSchedulerBackend.stop(YarnSchedulerBackend.scala:93)
      at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.stop(YarnClientSchedulerBackend.scala:151)
      at org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:467)
      at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1588)
      at org.apache.spark.SparkContext$$anonfun$stop$8.apply$mcV$sp(SparkContext.scala:1826)
      at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1283)
      at org.apache.spark.SparkContext.stop(SparkContext.scala:1825)
      at org.apache.spark.SparkContext.&lt;init&gt;(SparkContext.scala:587)
      at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313)
      at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
      at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
      at scala.Option.getOrElse(Option.scala:121)
      at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
      at org.apache.spark.repl.Main$.createSparkSession(Main.scala:95)
      at $line3.$read$$iw$$iw.&lt;init&gt;(&lt;console&gt;:15)
      at $line3.$read$$iw.&lt;init&gt;(&lt;console&gt;:42)
      at $line3.$read.&lt;init&gt;(&lt;console&gt;:44)
      at $line3.$read$.&lt;init&gt;(&lt;console&gt;:48)
      at $line3.$read$.&lt;clinit&gt;(&lt;console&gt;)
      at $line3.$eval$.$print$lzycompute(&lt;console&gt;:7)
      at $line3.$eval$.$print(&lt;console&gt;:6)
      at $line3.$eval.$print(&lt;console&gt;)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786)
      at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1047)
      at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:638)
      at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:637)
      at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
      at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
      at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:637)
      at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:569)
      at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565)
      at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:807)
      at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:681)
      at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:395)
      at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply$mcV$sp(SparkILoop.scala:38)
      at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:37)
      at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:37)
      at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:214)
      at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:37)
      at org.apache.spark.repl.SparkILoop.loadFiles(SparkILoop.scala:105)
      at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:920)
      at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
      at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
      at scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97)
      at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:909)
      at org.apache.spark.repl.Main$.doMain(Main.scala:68)
      at org.apache.spark.repl.Main$.main(Main.scala:51)
      at org.apache.spark.repl.Main.main(Main.scala)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
      at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
      at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  Caused by: java.io.IOException: Failed to send RPC 7605279734171920371 to /172.17.0.2:35136: java.nio.channels.ClosedChannelException
      at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:249)
      at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:233)
      at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:514)
      at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:488)
      at io.netty.util.concurrent.DefaultPromise.access$000(DefaultPromise.java:34)
      at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:438)
      at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:408)
      at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:455)
      at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
      at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
      at java.lang.Thread.run(Thread.java:745)
  Caused by: java.nio.channels.ClosedChannelException
      at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
  17/02/07 01:52:42 WARN metrics.MetricsSystem: Stopping a MetricsSystem that is not running
  org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
    at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:85)
    at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62)
    at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:156)
    at org.apache.spark.SparkContext.&lt;init&gt;(SparkContext.scala:509)
    at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313)
    at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868)
    at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860)
    at org.apache.spark.repl.Main$.createSparkSession(Main.scala:95)
    ... 47 elided
  &lt;console&gt;:14: error: not found: value spark
         import spark.implicits._
                ^
  &lt;console&gt;:14: error: not found: value spark
         import spark.sql
                ^
  </code></pre>
  
  <p>To me this looks more like YARN does not have spark libraries in it's context when running in client mode. I tried to add the <code>$SPARK_HOME/jars</code> into the yarn-site.xml file within the <code>yarn.application.classpath</code> configuration, but this seems not working either.</p>
  
  <p>How this could work with spark-shell and yarn? I was trying to find an official instructions in terms of handling such issue, but no guidance about this is found.</p>