title: Keras fit_generator: Unexpected usage of __getitem__ method
body: <p>I am using the fit_generator function to train my models and want to verify that my data is constructed and used as intended. My class derived from keras.utils.Sequence() implements the methods <code>__getitem__</code>, <code>__len__</code> and <code>on_epoch_end</code> and looks like this: </p>
  
  <pre><code>class PairwiseSequence(Sequence):
  """Generator that returns a combination of simulations (over a
  parametrizable amount of timesteps) and the corresponding metric distance.
  
  pair_list: List of pairwise combinations of simulations
  results: dictionary with results for the metric distance between
           simulation pairs
  sim_files: List of filenames representing single timesteps
  batch_size: number of samples to process in a single interference run
  """
  
  def __init__(self, pair_list, results, mean, std, train=False, sim_files=None,
               batch_size=1):
      self.pair_list = pair_list
      self.results = results
      self.batch_size = batch_size
      self.sim_files = sim_files
      self.mean = mean
      self.std = std
      self.train = train
  
  def __len__(self):
      return math.ceil(len(self.pair_list) / self.batch_size)
  
  def __getitem__(self, idx):
      dummy = LOADING_METHOD(self.pair_list[0][0], self.sim_files)
      x_1 = np.zeros(tuple([self.batch_size]) + dummy.shape)
      x_2 = np.zeros(tuple([self.batch_size]) + dummy.shape)
      y = np.zeros((self.batch_size, 1))
  
      if self.train:
          #print((idx * self.batch_size + i) % len(self.pair_list), ',')
          print("training idx:", idx)
      else:
          print("validation idx:", idx)
  
          for i in range(0, self.batch_size):
          (sim1, sim2) = self.pair_list[(idx * self.batch_size + i) %
                                        len(self.pair_list)]
          x_1[i] = LOADING_METHOD(sim1, self.sim_files) - self.mean[0]
          x_1[i] /= self.std[0]
          x_2[i] = LOADING_METHOD(sim2, self.sim_files) - self.mean[1]
          x_2[i] /= self.std[1]
          y[i] = self.results[frozenset((sim1.ensemble, sim2.ensemble))]
      return [x_1, x_2], y
  
  def on_epoch_end(self):
      if self.train:
          print("training generator: epoch end")
      else:
          print("validation generator: epoch end")
      #random.shuffle(self.pair_list)
  </code></pre>
  
  <p>This class is used as generator for training and validation data (two separate instances).</p>
  
  <p>As you can see I am printing the <code>idx</code> parameter of <code>__getitem__</code> and some notifcation when an epoch ends to the console. I am invoking fit_generator as followed (with turned of multiprocessing):</p>
  
  <pre><code>history_callback = model.fit_generator(
      generator=train_gen,
      steps_per_epoch=len(train_gen),
      epochs=epochs,
      verbose=0,
      callbacks=callbacks,
      validation_data=valid_gen,
      validation_steps=len(valid_gen),
      workers=1,
      use_multiprocessing=False,
      shuffle=False
  )
  </code></pre>
  
  <p>Also I turned of shuffling of the data. With this configuration I would have expected that <code>idx</code> goes from 0 to len(generator) and after that <code>on_epoch_end</code> is invoked. I have 372 samples for training and 93 for validation with batch_size 12 <code>idx</code> should go from 0 to 30 (training data) respectively 0 to 7 (validation data). But <code>__getitem__</code> gets invoked more often than I expected and also <code>on_epoch_end</code> gets called inbetween! Here is how the console output looks like:</p>
  
  <pre><code>batch_size: 12
  len(train_gen): 31
  len(valid_gen): 8
  2018-02-14 08:45:09.041929: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX
  training idx: 0
  training idx: 1
  training idx: 2
  training idx: 3
  training idx: 4
  training idx: 5
  training idx: 6
  training idx: 7
  training idx: 8
  training idx: 9
  training idx: 10
  training idx: 11
  training idx: 12
  training idx: 13
  training idx: 14
  training idx: 15
  training idx: 16
  training idx: 17
  training idx: 18
  training idx: 19
  training idx: 20
  training generator: epoch end
  training idx: 21
  training idx: 22
  training idx: 23
  training idx: 24
  training idx: 25
  training idx: 26
  training idx: 27
  training idx: 28
  training idx: 29
  training idx: 30
  training idx: 0
  validation generator: epoch end
  validation idx: 0
  training idx: 1
  validation idx: 1
  training idx: 2
  validation idx: 2
  training idx: 3
  validation idx: 3
  training idx: 4
  validation idx: 4
  training idx: 5
  validation idx: 5
  validation generator: epoch end
  training idx: 6
  validation idx: 6
  training idx: 7
  validation idx: 7
  training idx: 8
  validation idx: 0
  training idx: 9
  validation idx: 1
  training idx: 10
  validation idx: 2
  validation idx: 3
  validation idx: 4
  validation idx: 5
  validation idx: 6
  validation idx: 7
  validation idx: 0
  validation idx: 1
  validation idx: 2
  Epoch 00000: val_loss improved from inf to 10512.69922, saving model to /home/stefan/vcs/MA/code/results/test/TB_dummy_distance_10513.hdf5
  training idx: 11
  training idx: 12
  training idx: 13
  training idx: 14
  training idx: 15
  training idx: 16
  training idx: 17
  training idx: 18
  training idx: 19
  training idx: 20
  training generator: epoch end
  training idx: 21
  training idx: 22
  training idx: 23
  training idx: 24
  training idx: 25
  training idx: 26
  training idx: 27
  training idx: 28
  training idx: 29
  training idx: 30
  training idx: 0
  validation generator: epoch end
  validation idx: 0
  training idx: 1
  validation idx: 1
  training idx: 2
  validation idx: 2
  training idx: 3
  validation idx: 3
  training idx: 4
  validation idx: 4
  training idx: 5
  validation idx: 5
  validation generator: epoch end
  training idx: 6
  validation idx: 6
  training idx: 7
  validation idx: 7
  validation idx: 0
  training idx: 8
  validation idx: 1
  training idx: 9
  validation idx: 2
  training idx: 10
  validation idx: 3
  validation idx: 4
  validation idx: 5
  validation idx: 6
  validation idx: 7
  validation idx: 0
  validation idx: 1
  validation idx: 2
  Epoch 00001: val_loss improved from 10512.69922 to 5905.95929, saving model to /home/stefan/vcs/MA/code/results/test/TB_dummy_distance_5906.hdf5
  </code></pre>
  
  <p>How does fit_generator use the <code>__getitem__</code> and <code>on_epoch_end</code> methods? Does it also call these methods to get some sample data for weight initialization before the first epoch starts? Is this behaviour caused by some kind of caching?</p>
  
  <p>Any help is appreciated very much! Thank you in advance!</p>
  
  <h1>Update:</h1>
  
  <p>For testing purposes I changed the <code>max_queue_size</code> argument of <code>fit_generator</code> to 1. This is the resulting terminal output:</p>
  
  <pre><code>batch_size: 12
  len(train_gen): 31
  len(valid_gen): 8
  2018-02-14 10:10:40.001065: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX
  training idx: 0
  training idx: 1
  training idx: 2
  training idx: 3
  training idx: 4
  training idx: 5
  training idx: 6
  training idx: 7
  training idx: 8
  training idx: 9
  training idx: 10
  training idx: 11
  training idx: 12
  training idx: 13
  training idx: 14
  training idx: 15
  training idx: 16
  training idx: 17
  training idx: 18
  training idx: 19
  training idx: 20
  training idx: 21
  training idx: 22
  training idx: 23
  training idx: 24
  training idx: 25
  training idx: 26
  training idx: 27
  training idx: 28
  training idx: 29
  training idx: 30
  training generator: epoch end
  training idx: 0
  training idx: 1
  validation idx: 0
  validation idx: 1
  validation idx: 2
  validation idx: 3
  validation idx: 4
  validation idx: 5
  validation idx: 6
  validation generator: epoch end
  validation idx: 7
  validation idx: 0
  validation idx: 1
  Epoch 00000: val_loss improved from inf to 18090.34473, saving model to /home/stefan/vcs/MA/code/results/test/TB_dummy_distance_18090.hdf5
  training idx: 2
  training idx: 3
  training idx: 4
  training idx: 5
  training idx: 6
  training idx: 7
  training idx: 8
  training idx: 9
  training idx: 10
  training idx: 11
  training idx: 12
  training idx: 13
  training idx: 14
  training idx: 15
  training idx: 16
  training idx: 17
  training idx: 18
  training idx: 19
  training idx: 20
  training idx: 21
  training idx: 22
  training idx: 23
  training idx: 24
  training idx: 25
  training idx: 26
  training idx: 27
  training idx: 28
  training idx: 29
  training idx: 30
  training generator: epoch end
  training idx: 0
  training idx: 1
  validation idx: 0
  validation idx: 1
  validation idx: 2
  validation idx: 3
  validation idx: 4
  validation idx: 5
  validation idx: 6
  validation generator: epoch end
  validation idx: 7
  validation idx: 0
  validation idx: 1
  Epoch 00001: val_loss did not improve
  </code></pre>
  
  <p>Now at least in the very first epoch all training samples get queried. But for the validation data and training data in the second epoch, <code>on_epoch_end</code> still gets called to early.</p>