title: Why does shuffling my validation set in Keras change my model's performance?
body: <p><strong>Why I'm confused:</strong></p>
  
  <p>If I test my model on examples [A, B, C], it will obtain a certain accuracy. If I test the same model on examples [C, B, A], it should obtain the same accuracy. In other words, shuffling the examples shouldn't change my model's accuracy. But that's what seems to be happening below:</p>
  
  <p><strong>Step-by-step:</strong></p>
  
  <p>Here is where I train the model:</p>
  
  <pre><code>model.fit_generator(batches, batches.nb_sample, nb_epoch=1, verbose=2,
                      validation_data=val_batches,
                      nb_val_samples=val_batches.nb_sample)
  </code></pre>
  
  <p>Here is where I test the model, without shuffling the validation set:</p>
  
  <pre><code>gen = ImageDataGenerator()
  results = []
  for _ in range(3):
      val_batches = gen.flow_from_directory(path+"valid", batch_size=batch_size*2,
                                            target_size=target_size, shuffle=False)
      result = model.evaluate_generator(val_batches, val_batches.nb_sample)
      results.append(result)
  </code></pre>
  
  <p>Here are the results (val_loss, val_acc):</p>
  
  <pre><code>[2.8174608421325682, 0.17300000002980231]
  [2.8174608421325682, 0.17300000002980231]
  [2.8174608421325682, 0.17300000002980231]
  </code></pre>
  
  <p>Notice that the validation accuracies are the same.</p>
  
  <p>Here is where I test the model, with a <strong>shuffled</strong> validation set:</p>
  
  <pre><code>results = []
  for _ in range(3):
      val_batches = gen.flow_from_directory(path+"valid", batch_size=batch_size*2,
                                            target_size=target_size, shuffle=True)
      result = model.evaluate_generator(val_batches, val_batches.nb_sample)
      results.append(result)
  </code></pre>
  
  <p>Here are the results (val_loss, val_acc):</p>
  
  <pre><code>[2.8174608802795409, 0.17299999999999999]
  [2.8174608554840086, 0.1730000001192093]
  [2.8174608268737793, 0.17300000059604645]
  </code></pre>
  
  <p>Notice that <strong>the validation accuracies are inconsistent</strong>, despite an unchanged validation set and an unchanged model. What's going on?</p>
  
  <hr>
  
  <p><strong>Note</strong>:</p>
  
  <p>I'm evaluating on the entire validation set each time. <a href="https://keras.io/models/sequential/#evaluate_generator" rel="noreferrer">model.evaluate_generator</a> returns after evaluating the model on the number of examples equal to <code>val_batches.nb_sample</code>, which is the number of examples in the validation set.</p>