title: Storing HDF5 subsets as datasets (in python)
body: <p>I have only limited knowledge of HDF5 but I would like to understand something about HDF selections.</p>
  
  <p>To give some context I'm interested in using HDF5 for applications in machine learning. Suppose you have a data matrix with <code>n</code> rows and <code>p</code> columns. In a typical <code>k</code>-folds cross-validation setting, you will split the matrix into <code>k</code> samples (each one of size <code>(n/k, p)</code>) and repetitively use <code>k-1</code> for training and 1 for testing. Of course, storing all the training and testing sets will use a lot of space. This is where HDF5 selections could help.</p>
  
  <p>If I understand correctly, a selection can refer to any subset of a dataset. Moreover a selection can be stored into a dataset. Therefore, starting from a <code>(n, p)</code> dataset in HDF5, I could create <code>k</code> groups (one for each fold), containing a training dataset (a subset of the rows in the original dataset) and a testing dataset (the remainder). As it's only references, it won't use too much space.</p>
  
  <p>I have found some documentation about selections but it's not very clear. The code examples are in C which is a bit harsh to modify and I mainly use Python. I haven't found anything related to this in <code>PyTables</code>. I have found <a href="http://docs.h5py.org/en/latest/refs.html" rel="nofollow">some examples in <code>h5py</code></a> but I couldn't figure how to put data in the subset.</p>
  
  <p>Can anybody confirm that this is an interesting approach and provide some python code for storing a subset of rows from a dataset as another dataset?</p>