title: Concatenate a large number of HDF5 files
body: <p>I have about 500 HDF5 files each of about 1.5 GB.</p>
  
  <p>Each of the files has the same exact structure, which is 7 compound (int,double,double) datasets and variable number of samples.</p>
  
  <p>Now I want to concatenate all this files by concatenating each of the datasets so that at the end I have a single 750 GB file with my 7 datasets.</p>
  
  <p>Currently I am running a h5py script which:</p>
  
  <ul>
  <li>creates a HDF5 file with the right datasets of unlimited max</li>
  <li>open in sequence all the files</li>
  <li>check what is the number of samples (as it is variable)</li>
  <li>resize the global file</li>
  <li>append the data</li>
  </ul>
  
  <p>this obviously takes many hours,
  would you have a suggestion about improving this?</p>
  
  <p>I am working on a cluster, so I could use HDF5 in parallel, but I am not good enough in <strong>C</strong> programming to implement something myself, I would need a tool already written.</p>