title: How do I train a neural network in Keras on data stored in HDF5 files?
body: <p>I have two fairly large PyTables EArrays which contain the inputs and labels for a regression task. The input array is 4d (55k x 128 x 128 x 3) and the label array is 1d (55k). I have a NN architecture specified in Keras which I want to train on this data, but there are two problems.</p>
  
  <ol>
  <li>The input array at least is too large to fit in memory at once.</li>
  <li>I only want to train on some random subset of the full data, since I want to take train, test, and validation splits. I select the splits by slicing on random subsets of the indices.</li>
  </ol>
  
  <p>How can I select subsets of the HDF5 arrays (input and output) according to train/test indices and train on the training subsets, without reading them into memory all at once? Is there some way to create a "view" of the on-disk array that can be sliced and that Keras will see as a regular NumPy ndarray?</p>
  
  <p>What I've tried so far is to convert my arrays to Keras HDF5Matrix objects (with e.g. <code>X = keras.utils.io_utils.HDF5Matrix(X)</code>), but when I then slice this to get a training split, the full slice (80% of the full array) gets put into memory, which gives me a <code>MemoryError</code>.</p>