title: How to deal with thousands of images for CNN training Keras
body: <p>I have ~10000k images that cannot fit in memory. So for now I can only read 1000 images and train on it... </p>
  
  <p>My code is here :</p>
  
  <pre><code>img_dir = "TrainingSet" # Enter Directory of all images 
  image_path = os.path.join(img_dir+"/images",'*.bmp')
  files = glob.glob(image_path)
  images = []
  masks = []
  contours = []
  indexes = []
  files_names = []
  
  for f1 in np.sort(files):
    img = cv2.imread(f1)
    result = re.search('original_cropped_(.*).bmp', str(f1))
    idx = result.group(1)
    mask_path = img_dir+"/masks/mask_cropped_"+str(idx)+".bmp"
    mask = cv2.imread(mask_path,0)
    contour_path = img_dir+"/contours/contour_cropped_"+str(idx)+".bmp"
    contour = cv2.imread(contour_path,0)
  
    indexes.append(idx)
    images.append(img)
    masks.append(mask)
    contours.append(contour)
  
  train_df = pd.DataFrame({"id":indexes,"masks": masks, "images": images,"contours": contours })
  train_df.sort_values(by="id",ascending=True,inplace=True)
  print(train_df.shape)
  
  img_size_target = (256,256)
  
  ids_train, ids_valid, x_train, x_valid, y_train, y_valid, c_train, c_valid = train_test_split(
      train_df.index.values,
      np.array(train_df.images.apply(lambda x: cv2.resize(x,img_size_target).reshape(img_size_target[0],img_size_target[1],3))), 
      np.array(train_df.masks.apply(lambda x: cv2.resize(x,img_size_target).reshape(img_size_target[0],img_size_target[1],1))), 
      np.array(train_df.contours.apply(lambda x: cv2.resize(x,img_size_target).reshape(img_size_target[0],img_size_target[1],1))), 
      test_size=0.2, random_state=1337)
  
  #Here we define the model architecture... 
  #.....
  #End of model definition
  
  # Training 
  optimizer = Adam(lr=1e-3,decay=1e-10)    
  model.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=["accuracy"])
  
  early_stopping = EarlyStopping(patience=10, verbose=1)
  model_checkpoint = ModelCheckpoint("./keras.model", save_best_only=True, verbose=1)
  reduce_lr = ReduceLROnPlateau(factor=0.5, patience=5, min_lr=0.00001, verbose=1)
  
  epochs = 200
  batch_size = 32
  
  history = model.fit(x_train, y_train,
                  validation_data=[x_valid, y_valid], 
                  epochs=epochs,
                  batch_size=batch_size,
                  callbacks=[early_stopping, model_checkpoint, reduce_lr])
  </code></pre>
  
  <p>What I would like to know is how can I modify my code in order to do batches of a small set of images without loading all the other 10000 into memory ? which means that the algorithm will read X images each epoch from directory and train on it and after that goes for the next X until the last one.  </p>
  
  <p>X here would be a reasonable amount of images that can fit into memory. </p>