title: Keras: batch training for multiple large datasets
body: <p>this question regards the common problem of training on multiple large files in Keras which are jointly too large to fit on GPU memory. 
  I am using Keras 1.0.5 and I would like a solution that does not require 1.0.6. 
  One way to do this was described by fchollet
  <a href="https://github.com/fchollet/keras/issues/107" rel="noreferrer">here</a> and
  <a href="https://github.com/fchollet/keras/issues/68" rel="noreferrer">here</a>:</p>
  
  <pre><code># Create generator that yields (current features X, current labels y)
  def BatchGenerator(files):
      for file in files:
          current_data = pickle.load(open("file", "rb"))
          X_train = current_data[:,:-1]
          y_train = current_data[:,-1]
          yield (X_train, y_train)
  
  # train model on each dataset
  for epoch in range(n_epochs):
      for (X_train, y_train) in BatchGenerator(files):
          model.fit(X_train, y_train, batch_size = 32, nb_epoch = 1)
  </code></pre>
  
  <p>However I fear that the state of the model is not saved, rather that the model is reinitialized not only between epochs but also between datasets. Each "Epoch 1/1" represents training on a different dataset below:</p>
  
  <p>~~~~~ Epoch 0 ~~~~~~</p>
  
  <p>Epoch 1/1
  295806/295806 [==============================] - 13s - loss: 15.7517<br>
  Epoch 1/1
  407890/407890 [==============================] - 19s - loss: 15.8036<br>
  Epoch 1/1
  383188/383188 [==============================] - 19s - loss: 15.8130<br>
  ~~~~~ Epoch 1 ~~~~~~</p>
  
  <p>Epoch 1/1
  295806/295806 [==============================] - 14s - loss: 15.7517<br>
  Epoch 1/1
  407890/407890 [==============================] - 20s - loss: 15.8036<br>
  Epoch 1/1
  383188/383188 [==============================] - 15s - loss: 15.8130</p>
  
  <p>I am aware that one can use model.fit_generator but as the method above was repeatedly suggested as a way of batch training I would like to know what I am doing wrong.</p>
  
  <p>Thanks for your help,</p>
  
  <p>Max</p>