title: Why does Spark Streaming app with Kafka fail with "ClassNotFoundException: org.apache.spark.streaming.kafka.KafkaRDDPartition"?
body: <p>I use Spark Streaming with Apache Kafka.</p>
  
  <pre><code>val directKafkaStream = KafkaUtils.createDirectStream[String,String,StringDecoder,StringDecoder ](
    ssc, kafkaParams, topics)
  val events = directKafkaStream.flatMap(x=&gt;{
    val data = JSONObject.fromObject(x._2)
    Some(data)
  
  })
  val dbIndex = 1
  val clickHashKey = "app::users::click"
  
  val userClicks = events.map(x=&gt;(x.getString("userid"),x.getInt("click_count"))).reduceByKey(_+_)
  userClicks.foreachRDD(partitionOfRecords=&gt;partitionOfRecords.foreach(pair=&gt;{
    val userid = pair._1
    val clickCount = pair._2
    val jedis = RedisClient.pool.getResource
    jedis.select(dbIndex)
    jedis.hincrBy(clickHashKey, userid, clickCount)
    RedisClient.pool.returnResource(jedis)
  
  }))
  ssc.start()
  ssc.awaitTermination()
  </code></pre>
  
  <p>It fails with the follow exception:</p>
  
  <pre><code>16/12/11 14:17:20 INFO DAGScheduler: ShuffleMapStage 146 (map at UserClickCountAnalysis.scala:75) failed in 0.068 s
  16/12/11 14:17:20 INFO DAGScheduler: Job 73 failed: foreachRDD at UserClickCountAnalysis.scala:76, took 0.073045 s
  16/12/11 14:17:20 ERROR JobScheduler: Error running job streaming job 1481437040000 ms.0
  
  org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 146.0 failed 4 times, most recent failure: Lost task 0.3 in stage 146.0 (TID 295, 10.211.55.12): java.lang.ClassNotFoundException: org.apache.spark.streaming.kafka.KafkaRDDPartition
      at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
      at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
      at java.security.AccessController.doPrivileged(Native Method)
      at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
      at java.lang.Class.forName0(Native Method)
      at java.lang.Class.forName(Class.java:274)
      at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:66)
      at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612)
      at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
      at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
      at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69)
      at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:95)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:194)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)
  
  Driver stacktrace:
      at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1266)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1257)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1256)
      at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
      at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
      at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1256)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:730)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:730)
      at scala.Option.foreach(Option.scala:236)
      at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:730)
      at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1450)
      at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1411)
      at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
  16/12/11 14:17:25 INFO JobScheduler: Added jobs for time 1481437045000 ms
  16/12/11 14:17:25 INFO JobScheduler: Starting job streaming job 1481437045000 ms.0 from job set of time 1481437045000 ms
  16/12/11 14:17:25 INFO SparkContext: Starting job: foreachRDD at UserClickCountAnalysis.scala:76
  16/12/11 14:17:25 INFO DAGScheduler: Registering RDD 298 (map at UserClickCountAnalysis.scala:75)
  16/12/11 14:17:25 INFO DAGScheduler: Got job 74 (foreachRDD at UserClickCountAnalysis.scala:76) with 2 output partitions (allowLocal=false)
  16/12/11 14:17:25 INFO DAGScheduler: Final stage: ResultStage 149(foreachRDD at UserClickCountAnalysis.scala:76)
  16/12/11 14:17:25 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 148)
  16/12/11 14:17:25 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 148)
  16/12/11 14:17:25 INFO DAGScheduler: Submitting ShuffleMapStage 148 (MapPartitionsRDD[298] at map at UserClickCountAnalysis.scala:75), which has no missing parents
  16/12/11 14:17:25 INFO MemoryStore: ensureFreeSpace(3880) called with curMem=42510, maxMem=2061647216
  16/12/11 14:17:25 INFO MemoryStore: Block broadcast_74 stored as values in memory (estimated size 3.8 KB, free 1966.1 MB)
  16/12/11 14:17:25 INFO MemoryStore: ensureFreeSpace(2194) called with curMem=46390, maxMem=2061647216
  16/12/11 14:17:25 INFO MemoryStore: Block broadcast_74_piece0 stored as bytes in memory (estimated size 2.1 KB, free 1966.1 MB)
  16/12/11 14:17:25 INFO BlockManagerInfo: Added broadcast_74_piece0 in memory on 192.168.1.103:56006 (size: 2.1 KB, free: 1966.1 MB)
  16/12/11 14:17:25 INFO SparkContext: Created broadcast 74 from broadcast at DAGScheduler.scala:874
  16/12/11 14:17:25 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 148 (MapPartitionsRDD[298] at map at UserClickCountAnalysis.scala:75)
  16/12/11 14:17:25 INFO TaskSchedulerImpl: Adding task set 148.0 with 1 tasks
  16/12/11 14:17:25 INFO TaskSetManager: Starting task 0.0 in stage 148.0 (TID 296, 10.211.55.12, ANY, 1271 bytes)
  16/12/11 14:17:25 WARN TaskSetManager: Lost task 0.0 in stage 148.0 (TID 296, 10.211.55.12): java.lang.ClassNotFoundException: org.apache.spark.streaming.kafka.KafkaRDDPartition
      at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
  </code></pre>
  
  <p>my pom.xml below</p>
  
  <pre><code>    &lt;?xml version="1.0" encoding="UTF-8"?&gt;
      &lt;project xmlns="http://maven.apache.org/POM/4.0.0"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"&gt;
  &lt;modelVersion&gt;4.0.0&lt;/modelVersion&gt;
  
  &lt;groupId&gt;phnasis&lt;/groupId&gt;
  &lt;artifactId&gt;sparkstreamingUserClick&lt;/artifactId&gt;
  &lt;version&gt;1.0-SNAPSHOT&lt;/version&gt;
  
  &lt;dependencies&gt;
      &lt;!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.10 --&gt;
      &lt;!--&lt;dependency&gt;--&gt;
          &lt;!--&lt;groupId&gt;org.apache.spark&lt;/groupId&gt;--&gt;
          &lt;!--&lt;artifactId&gt;spark-core_2.10&lt;/artifactId&gt;--&gt;
          &lt;!--&lt;version&gt;1.4.0&lt;/version&gt;--&gt;
      &lt;!--&lt;/dependency&gt;--&gt;
  
      &lt;dependency&gt;
          &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
          &lt;artifactId&gt;spark-streaming-kafka_2.10&lt;/artifactId&gt;
          &lt;version&gt;1.4.0&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming_2.10 --&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
          &lt;artifactId&gt;spark-streaming_2.10&lt;/artifactId&gt;
          &lt;version&gt;1.4.0&lt;/version&gt;
      &lt;/dependency&gt;
  
      &lt;!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka_2.10 --&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;org.apache.kafka&lt;/groupId&gt;
          &lt;artifactId&gt;kafka_2.10&lt;/artifactId&gt;
          &lt;version&gt;0.8.2.1&lt;/version&gt;
      &lt;/dependency&gt;
  
  
      &lt;dependency&gt;
          &lt;groupId&gt;redis.clients&lt;/groupId&gt;
          &lt;artifactId&gt;jedis&lt;/artifactId&gt;
          &lt;version&gt;2.9.0&lt;/version&gt;
          &lt;type&gt;jar&lt;/type&gt;
          &lt;scope&gt;compile&lt;/scope&gt;
      &lt;/dependency&gt;
  
  
  
  &lt;/dependencies&gt;
  &lt;build&gt;
      &lt;sourceDirectory&gt;src/main/java&lt;/sourceDirectory&gt;
      &lt;testSourceDirectory&gt;src/test/java&lt;/testSourceDirectory&gt;
      &lt;plugins&gt;
          &lt;!--
                       Bind the maven-assembly-plugin to the package phase
            this will create a jar file without the storm dependencies
            suitable for deployment to a cluster.
           --&gt;
          &lt;plugin&gt;
              &lt;artifactId&gt;maven-assembly-plugin&lt;/artifactId&gt;
              &lt;configuration&gt;
                  &lt;descriptorRefs&gt;
                      &lt;descriptorRef&gt;jar-with-dependencies&lt;/descriptorRef&gt;
                  &lt;/descriptorRefs&gt;
                  &lt;archive&gt;
                      &lt;manifest&gt;
                          &lt;mainClass&gt;&lt;/mainClass&gt;
                      &lt;/manifest&gt;
                  &lt;/archive&gt;
              &lt;/configuration&gt;
              &lt;executions&gt;
                  &lt;execution&gt;
                      &lt;id&gt;make-assembly&lt;/id&gt;
                      &lt;phase&gt;package&lt;/phase&gt;
                      &lt;goals&gt;
                          &lt;goal&gt;single&lt;/goal&gt;
                      &lt;/goals&gt;
                  &lt;/execution&gt;
              &lt;/executions&gt;
          &lt;/plugin&gt;
      &lt;/plugins&gt;
  &lt;/build&gt;
  </code></pre>
  
  <p></p>