title: How should I persist my event stream to cold storage?
body: <p>I have a stream of events (we could alternatively call them "messages" or even just "data") coming from an <em>event broker</em> with time-based retension. The event broker could be <a href="http://kafka.apache.org/" rel="nofollow">Kafka</a> or <a href="http://aws.amazon.com/kinesis/" rel="nofollow">Amazon Kinesis</a> or <a href="http://azure.microsoft.com/en-us/services/event-hubs/" rel="nofollow">Microsoft Event Hubs</a>, though let's say that it's Kafka.</p>
  
  <p>My goal is take this event stream and put it into <em>cold storage</em>; that is, to store the data for future analysis via Hadoop/Spark. This means that I'd like to take this "chatty" stream of events and convert it to "chunky" files in HDFS. In a cloud environment, I'd likely use S3 or Azure Storage instead of HDFS.</p>
  
  <p>I'd also like my solution to be cost effective; for example, using serialization formats such as Avro/ORC in order to reduce my disk space cost. I'm also like an at-least-once guarentee that a given event is persited to cold storage (bonus points for once-and-only-once).</p>
  
  <p>My core questions are: </p>
  
  <ul>
  <li><strong>How are people solving this problem?</strong></li>
  <li><strong>Are there components out there already handle this scenario?</strong></li>
  <li><strong>Do I need to develop a solution myself?</strong></li>
  <li><strong>In the least, are they any recommended patterns?</strong></li>
  </ul>