title: Failed to submit local jar to spark cluster: java.nio.file.NoSuchFileException
body: <pre><code>~/spark/spark-2.1.1-bin-hadoop2.7/bin$ ./spark-submit --master spark://192.168.42.80:32141 --deploy-mode cluster file:///home/me/workspace/myproj/target/scala-2.11/myproj-assembly-0.1.0.jar
  
  Running Spark using the REST application submission protocol.
  Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
  17/06/20 16:41:30 INFO RestSubmissionClient: Submitting a request to launch an application in spark://192.168.42.80:32141.
  17/06/20 16:41:31 INFO RestSubmissionClient: Submission successfully created as driver-20170620204130-0005. Polling submission state...
  17/06/20 16:41:31 INFO RestSubmissionClient: Submitting a request for the status of submission driver-20170620204130-0005 in spark://192.168.42.80:32141.
  17/06/20 16:41:31 INFO RestSubmissionClient: State of driver driver-20170620204130-0005 is now ERROR.
  17/06/20 16:41:31 INFO RestSubmissionClient: Driver is running on worker worker-20170620203037-172.17.0.5-45429 at 172.17.0.5:45429.
  17/06/20 16:41:31 ERROR RestSubmissionClient: Exception from the cluster:
  java.nio.file.NoSuchFileException: /home/me/workspace/myproj/target/scala-2.11/myproj-assembly-0.1.0.jar
      sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
      sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
      sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
      sun.nio.fs.UnixCopyFile.copy(UnixCopyFile.java:526)
      sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:253)
      java.nio.file.Files.copy(Files.java:1274)
      org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$copyRecursive(Utils.scala:608)
      org.apache.spark.util.Utils$.copyFile(Utils.scala:579)
      org.apache.spark.util.Utils$.doFetchFile(Utils.scala:664)
      org.apache.spark.util.Utils$.fetchFile(Utils.scala:463)
      org.apache.spark.deploy.worker.DriverRunner.downloadUserJar(DriverRunner.scala:154)
      org.apache.spark.deploy.worker.DriverRunner.prepareAndRunDriver(DriverRunner.scala:172)
      org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:91)
  17/06/20 16:41:31 INFO RestSubmissionClient: Server responded with CreateSubmissionResponse:
  {
    "action" : "CreateSubmissionResponse",
    "message" : "Driver successfully submitted as driver-20170620204130-0005",
    "serverSparkVersion" : "2.1.1",
    "submissionId" : "driver-20170620204130-0005",
    "success" : true
  }
  </code></pre>
  
  <p>Log from spark-worker:</p>
  
  <pre><code> 2017-06-20T20:41:30.807403232Z 17/06/20 20:41:30 INFO Worker: Asked to launch driver driver-20170620204130-0005
  2017-06-20T20:41:30.817248508Z 17/06/20 20:41:30 INFO DriverRunner: Copying user jar file:///home/me/workspace/myproj/target/scala-2.11/myproj-assembly-0.1.0.jar to /opt/spark/work/driver-20170620204130-0005/myproj-assembly-0.1.0.jar
  2017-06-20T20:41:30.883645747Z 17/06/20 20:41:30 INFO Utils: Copying /home/me/workspace/myproj/target/scala-2.11/myproj-assembly-0.1.0.jar to /opt/spark/work/driver-20170620204130-0005/myproj-assembly-0.1.0.jar
  2017-06-20T20:41:30.885217508Z 17/06/20 20:41:30 INFO DriverRunner: Killing driver process!
  2017-06-20T20:41:30.885694618Z 17/06/20 20:41:30 WARN Worker: Driver driver-20170620204130-0005 failed with unrecoverable exception: java.nio.file.NoSuchFileException: home/me/workspace/myproj/target/scala-2.11/myproj-assembly-0.1.0.jar 
  </code></pre>
  
  <p>Any idea why? Thanks</p>
  
  <p><strong>UPDATE</strong></p>
  
  <p>Is the following command right?</p>
  
  <pre><code>./spark-submit --master spark://192.168.42.80:32141 --deploy-mode cluster file:///home/me/workspace/myproj/target/scala-2.11/myproj-assembly-0.1.0.jar
  </code></pre>
  
  <p><strong>UPDATE</strong></p>
  
  <p>I think I understand a little more about the spark and why I had this problem and <a href="https://stackoverflow.com/questions/44684375/spark-submit-error-classnotfoundexception">spark-submit error: ClassNotFoundException</a>. The key point is that though the word REST used here <code>REST URL: spark://127.0.1.1:6066 (cluster mode)</code>, the application jar will not be uploaded to the cluster after submission, which is different with my understanding. so, the spark cluster cannot find the application jar, and cannot load the main class. </p>
  
  <p>I will try to find how to setup the spark cluster and use the cluster mode to submit application. No idea whether client mode will use more resources for streaming jobs. </p>