title: Apache Spark Multi Node Clustering
body: <p>I am currently working on logger analyse by using apache spark. I am new for Apache Spark. I have tried to use apache spark standalone mode. I can run my code by submitting jar with deploy-mode on the client. But I can not run with multi node cluster. I have used worker nodes are different machine.</p>
  
  <pre><code>sh spark-submit --class Spark.LogAnalyzer.App --deploy-mode cluster --master spark://rishon.server21:7077 /home/rishon/loganalyzer.jar "/home/rishon/apache-tomcat-7.0.63/LogAnalysisBackup/"
  </code></pre>
  
  <p>when i Run this command, it shows following error</p>
  
  <pre><code>15/10/20 18:04:23 ERROR ClientEndpoint: Exception from cluster was: java.io.FileNotFoundException: /home/rishon/loganalyzer.jar (No such file or directory)
  java.io.FileNotFoundException: /home/rishon/loganalyzer.jar (No such file or directory)
      at java.io.FileInputStream.open(Native Method)
      at java.io.FileInputStream.&lt;init&gt;(FileInputStream.java:146)
      at org.spark-project.guava.io.Files$FileByteSource.openStream(Files.java:124)
      at org.spark-project.guava.io.Files$FileByteSource.openStream(Files.java:114)
      at org.spark-project.guava.io.ByteSource.copyTo(ByteSource.java:202)
      at org.spark-project.guava.io.Files.copy(Files.java:436)
      at org.apache.spark.util.Utils$.org$apache$spark$util$Utils$$copyRecursive(Utils.scala:514)
      at org.apache.spark.util.Utils$.copyFile(Utils.scala:485)
      at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:562)
      at org.apache.spark.util.Utils$.fetchFile(Utils.scala:369)
      at org.apache.spark.deploy.worker.DriverRunner.org$apache$spark$deploy$worker$DriverRunner$$downloadUserJar(DriverRunner.scala:150)
      at org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:79)
  </code></pre>
  
  <p>As my understanding, The driver program sends the data and application code to worker node. I don't know my understanding is correct or not. So Please help me to run application on a cluster.</p>
  
  <p>I have tried to run jar on cluster and Now there is no exception but <strong>why the task is not assigned to worker node?</strong> </p>
  
  <p><a href="https://i.stack.imgur.com/xJLle.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/xJLle.png" alt="enter image description here"></a></p>
  
  <p>I have tried without clustering. Its working fine. shown in following figure</p>
  
  <p><a href="https://i.stack.imgur.com/8LGS0.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/8LGS0.png" alt="enter image description here"></a></p>
  
  <p>Above image shows, Task assigned to worker nodes. But I have one more problem to analyse the log file. Actually, I have log files in master node which is in a folder (ex: '/home/visva/log'). But the worker node searching the file on their own file system.</p>
  
  <p><a href="https://i.stack.imgur.com/hfSx6.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/hfSx6.png" alt="enter image description here"></a></p>