title: Platform for Storing TBs of Time Stamped Data
body: <p>We have a system that creates a lot of data, up to 1.5 million time stamped records, about 24MB, per second or about 2TB per day.</p>
  
  <p>The data comes from multiple sources and has multiple formats, the one thing in common is the time stamp.</p>
  
  <p>Currently we save about 5 days of data in files and have in-house software that generates reports.</p>
  
  <p>We are contemplating creating a scalable system that can hold and query years of data.</p>
  
  <p>We're leaning towards something like what Nathan Marz describes in <a href="http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html" rel="nofollow">How to beat the CAP theorem</a>, using Hadoop/ElephantDB for long term batch storage and Storm/Cassandra for a realtime layer.</p>
  
  <p>I'm wondering if the community can point out any alternatives or suggest further reading?</p>
  
  <p>Does the fact that our data is primarily organized by time lend itself to a particular type of solution?</p>
  
  <p>Is there a better forum to ask this kind of question?</p>
  
  <p>Thanks</p>