title: Spark streaming (run-example vs spark-submit)
body: <p>It's a follow up question to <a href="https://stackoverflow.com/questions/33590622/spark-with-flume-configuration-classpath">Spark with Flume (configuration/classpath?)</a></p>
  
  <p>Got that problem stored after trying a few things, the problem I now have is </p>
  
  <p>$spark-submit --jars /opt/scala/spark-streaming-flume_2.10-1.5.1.jar --master local[*] /home/user/spark/FlumeStreaming.py</p>
  
  <pre><code>from pyspark import SparkContext
  from pyspark.streaming import StreamingContext
  from pyspark.streaming.flume import FlumeUtils
  
  sc = SparkContext(appName="Newapp")
  strm = StreamingContext(sc,1)
  
  flume = FlumeUtils.createStream(strm,"localhost",9999)
  
  lines = flume.map(lambda x: x[1])
  counts = lines.flatMap(lambda line: line.split(" ")) \
          .map(lambda word: (word, 1)) \
          .reduceByKey(lambda a, b: a+b)
  counts.pprint()
  
  strm.start()
  strm.awaitTermination()
  </code></pre>
  
  <hr>
  
  <pre><code>15/11/07 23:55:09 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.NoClassDefFoundError: org/apache/flume/source/avro/AvroSourceProtocol
          at org.apache.spark.streaming.flume.FlumeReceiver.responder$lzycompute(FlumeInputDStream.scala:147)
          at org.apache.spark.streaming.flume.FlumeReceiver.responder(FlumeInputDStream.scala:146)
          at org.apache.spark.streaming.flume.FlumeReceiver.initServer(FlumeInputDStream.scala:163)
          at org.apache.spark.streaming.flume.FlumeReceiver.onStart(FlumeInputDStream.scala:170)
          at org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:148)
          at org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:130)
          at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:542)
          at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:532)
          at org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:1984)
          at org.apache.spark.SparkContext$$anonfun$37.apply(SparkContext.scala:1984)
          at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
          at org.apache.spark.scheduler.Task.run(Task.scala:88)
          at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:744)
  Caused by: java.lang.ClassNotFoundException: org.apache.flume.source.avro.AvroSourceProtocol
          at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
          at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
          at java.security.AccessController.doPrivileged(Native Method)
          at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
          ... 16 more
  
  15/11/07 23:55:09 ERROR TaskSetManager: Task 0 in stage 0.0 failed 1 times; aborting job
  15/11/07 23:55:09 ERROR ReceiverTracker: Deregistered receiver for stream 0: Stopped by driver
  </code></pre>
  
  <p>The question I have is, it's the same code as the flume_wordcount.py provided in spark examples, the one in the example works fine, but my version doesn't work. The difference is the way it's run one with run-example and the other one with spark-submit, which points to the way classpath and jar files are managed. Is there something I should do?</p>