title: iterating through hdf5 database slow in python
body: <p>I created a hdf5 file with 5000 groups each containing 1000 datasets with h5py. The datasets are 1-D arrays of integers, averaging about 10,000 elements, though this can vary from dataset to dataset. The datasets were created without the chunked storage option. The total size of the dataset is 281 GB. I want to iterate through all datasets to create a dictionary mapping the dataset name to the length of the dataset. I will use this dictionary in an algorithm later.</p>
  
  <p>Here's what I've tried. </p>
  
  <pre><code>import h5py
  
  f = h5py.File('/my/path/to/database.hdf5', 'r')
  lens = {}
  for group in f.itervalues():
      for dataset in group.itervalues():
          lens[dataset.name] = dataset.len()
  </code></pre>
  
  <p>This is too slow for my purposes and I'm looking for ways to speed up this procedure. I'm aware that it's possible to parallelize operations with h5py, but wanted to see if there was another approach before going down that road. I'd be open to re-creating the database with different options/structure if it would speed things up.</p>