title: How to append data to one specific dataset in a hdf5 file with h5py
body: <p>I am looking for a possibility to append data to an existing dataset inside a <code>.h5</code> file using Python (<code>h5py</code>).</p>
  
  <p>A short intro to my project: I try to train a CNN using medical image data. Because of the huge amount of data and heavy memory usage during the transformation of the data to NumPy arrays, I needed to split the "transformation" into a few data chunks: load and preprocess the first 100 medical images and save the NumPy arrays to hdf5 file, then load the next 100 datasets and append the existing <code>.h5</code> file, and so on.</p>
  
  <p>Now, I tried to store the first 100 transformed NumPy arrays as follows:</p>
  
  <pre><code>import h5py
  from LoadIPV import LoadIPV
  
  X_train_data, Y_train_data, X_test_data, Y_test_data = LoadIPV()
  
  with h5py.File('.\PreprocessedData.h5', 'w') as hf:
      hf.create_dataset("X_train", data=X_train_data, maxshape=(None, 512, 512, 9))
      hf.create_dataset("X_test", data=X_test_data, maxshape=(None, 512, 512, 9))
      hf.create_dataset("Y_train", data=Y_train_data, maxshape=(None, 512, 512, 1))
      hf.create_dataset("Y_test", data=Y_test_data, maxshape=(None, 512, 512, 1))
  </code></pre>
  
  <p>As can be seen, the transformed NumPy arrays are splitted into four different "groups" that are stored into the four <code>hdf5</code> datasets<code>[X_train, X_test, Y_train, Y_test]</code>.
  The <code>LoadIPV()</code> function performs the preprocessing of the medical image data.</p>
  
  <p>My problem is that I would like to store the next 100 NumPy arrays into the same <code>.h5</code> file into the existing datasets: that means that I would like to append to, for example, the existing <code>X_train</code> dataset of shape <code>[100, 512, 512, 9]</code> with the next 100 NumPy arrays, such that <code>X_train</code> becomes of shape <code>[200, 512, 512, 9]</code>. The same should work for the other three datasets <code>X_test</code>, <code>Y_train</code> and <code>Y_test</code>.</p>