title: What's an efficient way to store a time-series?
body: <p>I've a pretty large amount of data in a horrible format: 16.4GiB of zip files containing <code>csv</code> files.
  Each <code>csv</code> contains lines like</p>
  
  <pre><code>TYPE,2014-07-02 04:04:23.806,0.94598,0.94607
  </code></pre>
  
  <p>Basically:</p>
  
  <ul>
  <li>A <code>TYPE</code> (there are 14 different types and they could ideally grow in the future)</li>
  <li>A <code>timestamp</code> (in ms)</li>
  <li>2 float numbers (max 5 decimal places) which have usually a little spread</li>
  <li>All the float numbers of the same <code>TYPE</code> have usually very similar values when they're close in time</li>
  <li>The delta between timestamps is under 1s</li>
  <li>The data spans for 5 years</li>
  </ul>
  
  <p>In total I've 3'091'472'167 rows, so we're talking billions. 
  The main operations will be looping over part or all the data set (in order to run algorithms on the data) and one insertion of ~20 millions of records once per month (but inserting data is not something I'm interested in optimizing for).</p>
  
  <p>The ideal solution would have been storing them in a database so that I could query them easily, but some rough calculations showed me that I would need <a href="https://gist.github.com/framp/b5e9d7f7943824fd2d61" rel="noreferrer">46GiB</a> with Postgres (which MAYBE could be brought down to 18GiB with some ugly tricks).</p>
  
  <p>I've done some <a href="https://gist.github.com/framp/9103a7f1c87b0e87a4d1" rel="noreferrer">experiments</a> with the data I have and I found out that:
      Generating gzipped files containing <code>TIMESTAMP,FLOAT1,FLOAT2</code> would get me to ~14GiB
      Removing entirely the timestamps would get me down to ~5GiB</p>
  
  <p>A good way to save space could be storing the timestamp in an efficient way.
  I was thinking of creating a tree structure where every node contains one digit of the timestamp and where the leafs contain the final digit and the two floats.</p>
  
  <p>I've tried looking into solutions like druid.io but it looks like they're trying to optimize for other things my use case doesn't need.</p>
  
  <p>Is there some tool I'm missing which does exactly what I want?
  If not, what's an efficient way to store time series data?</p>
  
  <p>Thank you</p>