title: Simple (but large) read/write of time series data in python
body: <p>I'm having trouble working out how to approach my particular problem (perhaps because there are too many options!) with respect to storing simple time series data using python.</p>
  
  <p>All I want to do is store and access a really long one dimensional list of 16 bit integers. I don't even necessarily need to store an index, as it can be implicitly derived from the order of the samples. But I need a fast append operation.</p>
  
  <p>The problem is that I expect to have <em>about</em> 100GB of data which needs to be recorded to disk on the fly at 100M samples/sec (I've tested and my SSD setup can handle 400-500MB/sec writes), and with only ~14GB of RAM. My initial idea was to basically write it to individual CSV files in chunks of 100M rows, but it seems like this would waste a lot of space (a 16 bit integer can take up to 5 bytes as a string instead of 2). Other ideas included using msgpack/npz/raw binary files for chunks, and while I'm more than happy to implement this myself, I'm trying to work out if there is a solution which already exists.</p>
  
  <p>As I don't know exactly the number of samples I need beforehand, I think this means that numpy's memmap is not suitable.</p>
  
  <p>Later, I also need to slice the data between two indices for processing without loading the entire dataset into memory. This doesn't happen while the recording is occurring, only after.</p>
  
  <p>I think the pandas HDFStore is what I am after, but I am unsure of some of the best practices. </p>
  
  <p>Specifically, </p>
  
  <ol>
  <li>Given a very simple data format, what are the implications of lots of nodes (one node per chunk) vs a single node using HDFStore in pandas? If the former is better, what would be good chunk size?</li>
  <li>Are HDFStores the best way to do disk-backed high performance sequential reads and writes in pandas? Random access is not important. </li>
  <li>Is my problem in the domain of pandas? In this situation, pandas seems like overkill to me.</li>
  </ol>
  
  <p>Any other tips or recommendations would be appreciated. Really, what I need is the list object in python, but which also has support for being quickly chunked to and from the disk.</p>