title: How to know what is the reason for ClosedChannelExceptions with spark-shell in YARN client mode?
body: <p>I have been trying to run <code>spark-shell</code> in YARN <code>client</code> mode, but I am getting a lot of <code>ClosedChannelException</code> errors. I am using spark 2.0.0 build for Hadoop 2.6.</p>
  
  <p>Here are the exceptions :</p>
  
  <pre><code>$ spark-2.0.0-bin-hadoop2.6/bin/spark-shell --master yarn --deploy-mode client
  Setting default log level to "WARN".
  To adjust logging level use sc.setLogLevel(newLevel).
  16/09/13 14:12:36 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  16/09/13 14:12:38 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
  16/09/13 14:12:55 ERROR cluster.YarnClientSchedulerBackend: Yarn application has already exited with state FINISHED!
  16/09/13 14:12:55 ERROR client.TransportClient: Failed to send RPC 7920194824462016141 to /172.27.1.63:41034: java.nio.channels.ClosedChannelException
  java.nio.channels.ClosedChannelException
  16/09/13 14:12:55 ERROR spark.SparkContext: Error initializing SparkContext.
  java.lang.IllegalStateException: Spark context stopped while waiting for backend
      at org.apache.spark.scheduler.TaskSchedulerImpl.waitBackendReady(TaskSchedulerImpl.scala:581)
      at org.apache.spark.scheduler.TaskSchedulerImpl.postStartHook(TaskSchedulerImpl.scala:162)
      at org.apache.spark.SparkContext.&lt;init&gt;(SparkContext.scala:549)
      at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2256)
      at org.apache.spark.sql.SparkSession$Builder$$anonfun$8.apply(SparkSession.scala:831)
      at org.apache.spark.sql.SparkSession$Builder$$anonfun$8.apply(SparkSession.scala:823)
      at scala.Option.getOrElse(Option.scala:121)
      at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:823)
      at org.apache.spark.repl.Main$.createSparkSession(Main.scala:95)
      at $line3.$read$$iw$$iw.&lt;init&gt;(&lt;console&gt;:15)
      at $line3.$read$$iw.&lt;init&gt;(&lt;console&gt;:31)
      at $line3.$read.&lt;init&gt;(&lt;console&gt;:33)
      at $line3.$read$.&lt;init&gt;(&lt;console&gt;:37)
      at $line3.$read$.&lt;clinit&gt;(&lt;console&gt;)
      at $line3.$eval$.$print$lzycompute(&lt;console&gt;:7)
      at $line3.$eval$.$print(&lt;console&gt;:6)
      at $line3.$eval.$print(&lt;console&gt;)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786)
      at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1047)
      at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:638)
      at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:637)
      at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
      at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
      at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:637)
      at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:569)
      at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565)
      at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:807)
      at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:681)
      at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:395)
      at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply$mcV$sp(SparkILoop.scala:38)
      at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:37)
      at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:37)
      at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:214)
      at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:37)
      at org.apache.spark.repl.SparkILoop.loadFiles(SparkILoop.scala:94)
      at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:920)
      at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
      at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
      at scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97)
      at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:909)
      at org.apache.spark.repl.Main$.doMain(Main.scala:68)
      at org.apache.spark.repl.Main$.main(Main.scala:51)
      at org.apache.spark.repl.Main.main(Main.scala)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:729)
      at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
      at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  16/09/13 14:12:55 WARN netty.NettyRpcEndpointRef: Error sending message [message = RequestExecutors(0,0,Map())] in 1 attempts
  org.apache.spark.SparkException: Exception thrown in awaitResult
      at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77)
      at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:75)
      at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
      at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
      at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
      at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)
      at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83)
      at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:102)
      at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:78)
      at org.apache.spark.scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$1.apply$mcV$sp(YarnSchedulerBackend.scala:271)
      at org.apache.spark.scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$1.apply(YarnSchedulerBackend.scala:271)
      at org.apache.spark.scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$1.apply(YarnSchedulerBackend.scala:271)
      at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
      at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
  Caused by: java.io.IOException: Failed to send RPC 7920194824462016141 to /172.27.1.63:41034: java.nio.channels.ClosedChannelException
      at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:239)
      at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:226)
      at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
      at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:567)
      at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424)
      at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:801)
      at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:699)
      at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1122)
      at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:633)
      at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32)
      at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:908)
      at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:960)
      at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:893)
      at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
      at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
      at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
      ... 1 more
  Caused by: java.nio.channels.ClosedChannelException
  java.lang.IllegalStateException: Spark context stopped while waiting for backend
    at org.apache.spark.scheduler.TaskSchedulerImpl.waitBackendReady(TaskSchedulerImpl.scala:581)
    at org.apache.spark.scheduler.TaskSchedulerImpl.postStartHook(TaskSchedulerImpl.scala:162)
    at org.apache.spark.SparkContext.&lt;init&gt;(SparkContext.scala:549)
    at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2256)
    at org.apache.spark.sql.SparkSession$Builder$$anonfun$8.apply(SparkSession.scala:831)
    at org.apache.spark.sql.SparkSession$Builder$$anonfun$8.apply(SparkSession.scala:823)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:823)
    at org.apache.spark.repl.Main$.createSparkSession(Main.scala:95)
    ... 47 elided
  &lt;console&gt;:14: error: not found: value spark
         import spark.implicits._
                ^
  &lt;console&gt;:14: error: not found: value spark
         import spark.sql
                ^
  Welcome to
        ____              __
       / __/__  ___ _____/ /__
      _\ \/ _ \/ _ `/ __/  '_/
     /___/ .__/\_,_/_/ /_/\_\   version 2.0.0
        /_/
  
  Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_101)
  Type in expressions to have them evaluated.
  Type :help for more information.
  
  scala&gt; 16/09/13 14:12:59 ERROR client.TransportClient: Failed to send RPC 5797372389565173518 to /172.27.1.63:41034: java.nio.channels.ClosedChannelException
  16/09/13 14:12:59 WARN netty.NettyRpcEndpointRef: Error sending message [message = RequestExecutors(0,0,Map())] in 2 attempts
  org.apache.spark.SparkException: Exception thrown in awaitResult
      at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77)
      at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:75)
      at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
      at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
      at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
      at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)
      at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83)
      at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:102)
      at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:78)
      at org.apache.spark.scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$1.apply$mcV$sp(YarnSchedulerBackend.scala:271)
      at org.apache.spark.scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$1.apply(YarnSchedulerBackend.scala:271)
      at org.apache.spark.scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$1.apply(YarnSchedulerBackend.scala:271)
      at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
      at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
  Caused by: java.io.IOException: Failed to send RPC 5797372389565173518 to /172.27.1.63:41034: java.nio.channels.ClosedChannelException
      at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:239)
      at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:226)
      at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
      at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:567)
      at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424)
      at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:801)
      at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:699)
      at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1122)
      at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:633)
      at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32)
      at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:908)
      at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:960)
      at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:893)
      at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
      at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
      at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
      ... 1 more
  Caused by: java.nio.channels.ClosedChannelException
  </code></pre>