title: Apache Kafka + Spark Integration (REST API is needed?)
body: <p>Got some fundamental problems, hope someone can clear them up.</p>
  
  <p>So I want to use Apache Kafka and Apache spark for my application. I have gone through numerous tutorials and got the basic idea of what it is and how it will work.</p>
  
  <p>Use case :</p>
  
  <p>Data will be generated from a mobile device(multiple devices, lets say 1000) at an interval of 40 sec and I need to process that data and add values to the database which in turn will be reflected back in a dashboard.</p>
  
  <p>What I wanted to do is to use Apache Streams and make a post request from android itself and then those data will be processed by the spark application and that's it.</p>
  
  <p><strong>Issues:</strong> </p>
  
  <ul>
  <li>Apache Spark</li>
  </ul>
  
  <p>I am following this tutorial to get it up and running.( Am using JAVA, not scala)
  Link : <a href="https://www.santoshsrinivas.com/installing-apache-spark-on-ubuntu-16-04/" rel="nofollow noreferrer">https://www.santoshsrinivas.com/installing-apache-spark-on-ubuntu-16-04/</a></p>
  
  <p>After everything is done, I execute spark-shell and it start. I have also installed zookeeper and kafka on my server and I have started the Kafka in the background, so that's not an issue.</p>
  
  <p>When I run <code>http://161.xxx.xxx.xxx:4040/jobs/</code> I get this page <a href="https://i.stack.imgur.com/EqPm9.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/EqPm9.png" alt="enter image description here"></a></p>
  
  <p>In all the tutorial which I have gone through, there is a page like this : <a href="https://i.stack.imgur.com/gF1fN.png" rel="nofollow noreferrer">https://i.stack.imgur.com/gF1fN.png</a>  but I don't get this. Is it that spark is not properly installed?</p>
  
  <p>Now when I want to deploy a standalone jar to spark, (Using this link : <a href="http://data-scientist-in-training.blogspot.in/2015/03/apache-spark-cluster-deployment-part-1.html" rel="nofollow noreferrer">http://data-scientist-in-training.blogspot.in/2015/03/apache-spark-cluster-deployment-part-1.html</a> ) am able to run it.
  i.e with the command : <code>spark-submit --class SimpleApp.SimpleApp --master spark://http://161.xxx.xxx.xxx:7077 --name "try" /opt/spark/bin/try-0.0.1-SNAPSHOT.jar</code> , I get the output.</p>
  
  <p>Do I need to submit the application everytime if I want to use it? </p>
  
  <p>This is my Program :</p>
  
  <pre><code>package SimpleApp;
  
  /* SimpleApp.java */
  import org.apache.spark.api.java.*;
  import org.apache.spark.SparkConf;
  import org.apache.spark.api.java.function.Function;
  
  public class SimpleApp {
    public static void main(String[] args) {
      String logFile = "/opt/spark/README.md"; // Should be some file on your system
      SparkConf conf = new SparkConf().setAppName("Simple Application").setMaster("local[*]");
      JavaSparkContext sc = new JavaSparkContext(conf);
      //System.setProperty("hadoop.home.dir", "C:/winutil");
      sc.setLogLevel("ERROR"); // Don't want the INFO stuff
      JavaRDD&lt;String&gt; logData = sc.textFile(logFile).cache();
  
      long numAs = logData.filter(new Function&lt;String, Boolean&gt;() {
        public Boolean call(String s) { return s.contains("a"); }
      }).count();
  
      long numBs = logData.filter(new Function&lt;String, Boolean&gt;() {
        public Boolean call(String s) { return s.contains("b"); }
      }).count();
  
      System.out.println("Lines with a: " + numAs + ", lines with b: " + numBs);
      System.out.println("word count : "+logData.first());
      sc.stop();
    }
  }
  </code></pre>
  
  <ul>
  <li><p>Now how do I integrate Kafka into it?</p></li>
  <li><p>How to configure the app in such a way that it get executed everytime
  kafka receives a message?</p></li>
  <li><p>Moreover, do I need to make a REST API through which I need to send
  the data to kafka i.e the REST api will be used as producer?
  Something like spark Java framework? <a href="http://sparkjava.com/" rel="nofollow noreferrer">http://sparkjava.com/</a></p></li>
  <li><p>If yes, again the bottleneck will happen at REST api level i.e how
  many request it can handle or not because everywhere I read that
  Kafka has a very high throughput.</p></li>
  <li><p>Is the final structure going to be like SPARK JAVA -> KAFKA -> APACHE
  SPARK ?</p></li>
  <li><p>Lastly how to do I set up the development structure on my local
  device? I have kafka/apache spark installed. And am using Eclipse.</p></li>
  </ul>
  
  <p>Thanks</p>