title: Training a Keras model from batches of .npy files using generator?
body: <p>Currently I am dealing with a big data issue when training Image data using Keras. I have directory which has batch of .npy file. Each batch contain 512 images. Each batch has its corresponding label file as .npy. So it looks like: {image_file_1.npy, label_file_1.npy, ..., image_file_37.npy, label_file_37}. Each image file has dimension <code>(512, 199, 199, 3)</code>, each label file has dimension <code>(512, 1)</code>(eather 1 or 0) . If I load all the images in one ndarray it will be 35+ GB. So far reading all the Keras Doc. I am still not able to find how I will be able to train using custom generator. I have read about <code>flow_from_dict</code> and <code>ImageDataGenerator(...).flow()</code> but they are not ideal in that case or I do not know how to customized them.Here what I have done. </p>
  
  <pre><code>import numpy as np
  import keras
  from keras.models import Sequential
  from keras.layers import Dense, Dropout, Flatten
  from keras.layers import Conv2D, MaxPooling2D
  from keras.optimizers import SGD
  from keras.preprocessing.image import ImageDataGenerator
  
  val_gen = ImageDataGenerator(rescale=1./255)
  x_test = np.load("../data/val_file.npy")
  y_test = np.load("../data/val_label.npy")
  val_gen.fit(x_test)
  
  model = Sequential()
  ...
  model_1.add(layers.Dense(512, activation='relu'))
  model_1.add(layers.Dense(1, activation='sigmoid'))
  
  model.compile(loss='categorical_crossentropy', 
                optimizer=sgd, 
                 metrics=['acc'])
  
  model.fit_generator(generate_batch_from_directory() # should give 1 image file and 1 label file
                      validation_data=val_gen.flow(x_test, 
                                                   y_test, 
                                                   batch_size=64),
                      validation_steps=32)
  </code></pre>
  
  <p>So here <code>generate_batch_from_directory()</code> should take <code>image_file_i.npy</code> and <code>label_file_i.npy</code> every time and optimise the weight until there is no batch left. Each image array in the .npy files has already been processed with augmentation, rotation and scaling. Each <code>.npy</code> file is properly mixed with data from class 1 and 0 (50/50).</p>
  
  <p>If I append all the batch and create a big file such as:</p>
  
  <pre><code>X_train = np.append([image_file_1, ..., image_file_37])
  y_train = np.append([label_file_1, ..., label_file_37])
  </code></pre>
  
  <p>It does not fit in the memory. Otherwise I could use <code>.flow()</code> to generate image sets to train the model.</p>
  
  <p>Thanks for any advise.</p>