title: Incorrect Val Loss from Keras during training
body: <p>Keras seems to be reporting an incorrect val loss during training. When I train the model, it reports a val MSE loss of around 0.88. When I immediately reload the model and run the .predict method on the exact same validation set, I get a val MSE loss of around 1.2. I have checked the weights to ensure that the post-training model is identical to the reloaded model. Does anyone have any idea what might be going on? Thanks!</p>
  
  <p>Code:</p>
  
  <pre><code>training_data = data[:10000]
  training_norm_factor = np.std(training_data)
  training_mean = np.mean(training_data)
  training_inputs = training_data[:,:-1].reshape((-1, look_back, 1))
  training_inputs = (training_inputs - training_mean)/training_norm_factor
  training_labels = (training_data[:,-1] - training_mean)/training_norm_factor
  
  val_data = data[10000:]
  val_inputs = val_data[:, :-1].reshape((-1, look_back, 1))
  val_inputs = (val_inputs - training_mean)/training_norm_factor
  val_labels = (val_data[:,-1] - training_mean)/training_norm_factor
  
  # Build Model
  model = Sequential()
  model.add(layers.GRU(256, input_shape=(5, 1), return_sequences=True, 
  kernel_regularizer=regularizers.l2(0.01)))
  model.add(layers.GRU(256, kernel_regularizer=regularizers.l2(0.01)))
  model.add(layers.Dense(256, activation='relu', 
  kernel_regularizer=regularizers.l2(0.01)))
  model.add(layers.Dense(1))
  
  
  # Compile and Train
  model.summary()
  
  model.compile(optimizer=RMSprop(lr=1e-3, clipnorm=1.),
                loss='mean_squared_error',
                )
  
  callbacks_list = [
      K.callbacks.EarlyStopping(monitor='val_loss', mode='min', patience=0), K.callbacks.ModelCheckpoint('/Users/rickblickstead/Documents/GitHub/Volatility-
                  Forecasting/MyModel.5df5', 
                  monitor='val_loss', verbose=1, save_best_only=True, 
                  mode='min', period=1)
  ]
  
  history = model.fit(training_inputs[:, -5:], training_labels,
                      epochs=8,
                      batch_size=256,
                      shuffle=True,
                      callbacks=callbacks_list,
                      validation_data = (val_inputs[:, -5:], val_labels)
                     )
  
  # Re-Test validation set
  
  best_model = load_model('MyModel.5df5')
  best_predictions = best_model.predict(val_inputs[:, -5:], batch_size=256, verbose=0)
  print np.mean((best_predictions-val_labels)**2)
  </code></pre>