title: Persisting tweets using Spark Streaming
body: <p>To begin with, our requirement is fairly simple.  When the tweets come in, all we need to do is persist them on HDFS (at regular intervals).</p>
  
  <p>The 'checkpoint' API of JavaStreamingContext looked promising, but upon further review it seems to serve a different purpose.  (Also, I keep getting '/checkpoint/temp, error: No such file or directory (2)' error, but let's not worry about that for now).</p>
  
  <p>Question:  JavaDStream doesn't have a 'saveAsHadoopFiles' method - which kinda makes sense.  I guess saving to Hadoop from a streaming job is not a good idea.</p>
  
  <p>What's the recommended approach?  Should I write the incoming 'tweet' to a Kafka queue and then use a tool such as 'Camus'(<a href="https://github.com/linkedin/camus" rel="nofollow">https://github.com/linkedin/camus</a>) to push to HDFS?</p>