title: How to efficiently randomly select a subset of data from an h5py dataset
body: <p>I have a very very big dataset in h5py and this leads to memory problem when loaded in full and subsequent processing. I need to randomly select a subset and work with it. This is doing "boosting" in the context in machine learning. </p>
  
  <pre><code>dataset = h5py.File(h5_file, 'r')
  
  train_set_x_all = dataset['train_set_x'][:]
  train_set_y_all = dataset['train_set_y'][:]
  
  dataset.close()
  
  p = np.random.permutation(len(train_set_x_all))[:2000]   # rand select 2000
  train_set_x = train_set_x_all[p]   
  train_set_y = train_set_y_all[p]
  </code></pre>
  
  <p>I still somehow need to get the full set and slice it with index array p. This works for me as subsequently training only worked on the smaller set. But I wonder if there's still a better way to let me do this without even keeping the full dataset in memory at all.</p>