title: Streaming from HBase using Spark not serializable
body: <p>I am trying to stream data from HBase using Spark.  When I run the scala script, this is the error I get:</p>
  
  <pre><code>ERROR Executor: Exception in task 0.0 in stage 10.0 (TID 10)
  java.io.NotSerializableException: org.apache.hadoop.hbase.io.ImmutableBytesWritable
  </code></pre>
  
  <p>I was thinking at first that my data was formatted incorrectly, so I tried creating a very basic table with only one row:</p>
  
  <pre><code>row1 column=fam1:c1, timestamp=1422306700801, value=abc
  </code></pre>
  
  <p>Even with this row, I still get the same error. Anything obvious I am missing?  Here's the script:</p>
  
  <pre><code>def convertScanToString(scan: Scan): String = {
    val out: ByteArrayOutputStream = new ByteArrayOutputStream
    val dos: DataOutputStream = new DataOutputStream(out)
    scan.write(dos)
    Base64.encodeBytes(out.toByteArray)
  }
  
  val conf = HBaseConfiguration.create()
  val scan = new Scan()
  scan.setCaching(500)
  scan.setCacheBlocks(false)
  conf.set(TableInputFormat.INPUT_TABLE, "test_table")
  conf.set(TableInputFormat.SCAN, convertScanToString(scan))
  val rdd = sc.newAPIHadoopRDD(conf, classOf[TableInputFormat], classOf[ImmutableBytesWritable], classOf[Result])
  rdd.first
  </code></pre>
  
  <p>EDIT: As requested, here is the full stack trace</p>
  
  <pre><code>15/01/26 21:50:50 ERROR Executor: Exception in task 0.0 in stage 14.0 (TID 14)
  java.io.NotSerializableException: org.apache.hadoop.hbase.io.ImmutableBytesWritable
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183)
      at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
      at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1377)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1173)
      at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
      at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
      at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:73)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:206)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:744)
  15/01/26 21:50:50 ERROR TaskSetManager: Task 0.0 in stage 14.0 (TID 14) had a not serializable result: org.apache.hadoop.hbase.io.ImmutableBytesWritable; not retrying
  15/01/26 21:50:50 INFO TaskSchedulerImpl: Removed TaskSet 14.0, whose tasks have all completed, from pool
  15/01/26 21:50:50 INFO TaskSchedulerImpl: Cancelling stage 14
  15/01/26 21:50:50 INFO DAGScheduler: Job 14 failed: first at &lt;console&gt;:207, took 0.021506 s
  org.apache.spark.SparkException: Job aborted due to stage failure: Task 0.0 in stage 14.0 (TID 14) had a not serializable result: org.apache.hadoop.hbase.io.ImmutableBytesWritable
      at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1214)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1203)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1202)
      at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
      at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
      at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1202)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:696)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:696)
      at scala.Option.foreach(Option.scala:236)
      at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:696)
      at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1420)
      at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
      at org.apache.spark.scheduler.DAGSchedulerEventProcessActor.aroundReceive(DAGScheduler.scala:1375)
      at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
      at akka.actor.ActorCell.invoke(ActorCell.scala:487)
      at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
      at akka.dispatch.Mailbox.run(Mailbox.scala:220)
      at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
      at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
      at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
      at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
      at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
  </code></pre>