# Epoch consists of a single batch, yet model.fit and model.evaluate give different results on Keras. Model.fit is incorrect

My earlier question has unfortunately not been answered satisfactorily (as shown below if an epoch consists of a single batch the same strange behavior is observed), and this issue is driving me crazy, so please forgive the reposting of the question with new, more targeted, very simple code. Here is the link to my previous version of essentially the same question:

Seems like the Keras .fit and .evaluate methods give different training accuracy (yet the same validation accuracy). Same thing on loss.?

(I tried to comment to the proposed answer to that topic, but the allowed space is too short to enable posting of the new code).

Note that in the new example below an epoch has a single batch (2 examples) and the code is trained for a single epoch, so 1 epoch = 1 batch. Therefore what model.fit prints should match what model.evaluate prints. (This has been motivated by Andrej Karpathy's recommendation to start with fitting a single batch)

import numpy as np
np.random.seed(42)
import keras
from keras import layers
from keras.regularizers import l2
import tensorflow as tf
tf.random.set_seed(42)
import keras.backend as K

#def mse_custom(y_true, y_pred): #Tried defining the loss, got same results
#    custom = K.mean(K.square((y_pred - y_true)))
#    return custom

#Below I use 20 examples because the issue becomes much clearer than with a few examples
x_train = np.array([0.10526316, 0. ,0.26315789,0.73684211,0.94736842,1., \
0.52631579,0.84210526,0.21052632, 0.63157895,0.42105263,0.31578947,\
0.36842105,0.78947368,0.15789474,0.68421053, 0.47368421,0.05263158,\
0.57894737,0.89473684]).reshape(20,1,1,1)
y_train = np.array([0.31438135, 0. ,0.48843161,0.83670357,0.86771850,1., \
0.70921682,0.85732503,0.48475008,0.79387635,0.66044467,0.46540228,\
0.57612215,0.97176724,0.43534982,0.83881293,0.61425264,0.30733072,\
0.68508229,0.87707974]).reshape(20,1)
input_shape = (None,None,1)  #(1,1,1) gives the same results

seedbase = 1
model = keras.Sequential(
[
keras.Input(shape=input_shape),
layers.Dense(100, activation="sigmoid", use_bias=True,
kernel_initializer=keras.initializers.RandomNormal(mean=0., stddev=1./np.sqrt(1),seed=seedbase+0), #/np.sqrt(30)
bias_initializer=keras.initializers.RandomNormal(mean=0., stddev=1.,seed=seedbase+1),
kernel_regularizer=l2(0),
bias_regularizer=None),
layers.Dense(1, activation="sigmoid", use_bias=True,
kernel_initializer=keras.initializers.RandomNormal(mean=0., stddev=1./np.sqrt(1),seed=seedbase+12), #/np.sqrt(10)
bias_initializer=keras.initializers.RandomNormal(mean=0., stddev=1.,seed=seedbase+13),
kernel_regularizer=l2(0),
bias_regularizer=None),
]
)

model.summary()

"""
## Train the model
"""
n_val = 2
n_tr =20-n_val
val_split = n_val/20
batch_size = 2
epochs = 1
opt = keras.optimizers.SGD(learning_rate=0.1)
#opt = keras.optimizers.RMSprop(0.00099)
model.compile(loss='mean_squared_error', optimizer=opt)
ann=model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs, validation_split=val_split, verbose = 1)
"""
## Evaluate the trained model
"""
print("\n")
score = model.evaluate(x_train[:n_tr], y_train[:n_tr], batch_size=1, verbose=0)
print("Training loss:", score)
score = model.evaluate(x_train[-n_val:], y_train[-n_val:], batch_size=1, verbose=0)
print("Validation loss:", score)


And if you increase the epochs to say 100, the model.fit training loss goes to 0 (exactly matching only one of the two examples but very far from the other example, so clearly incorrect, and this is a clue to what is wrong) and training stops. The model.evaluate training loss stabilizes at the correct value. And the validation model.fit loss is very close to the validation model.evaluate loss.

Also, if I increase the number of examples in a single batch, the mismatch becomes much smaller.

I know I must be doing something wrong, but what? Your help would be very much appreciated.

• Doing a quick check it seems that the batch_size changes the reported score when using model.evaluate(), which seems really weird to me. Apr 21 at 18:13
• For " 1 epoch = 1 batch", you should set batch_size=20 not 2. Doing so will give you a consistent result. Apr 22 at 7:39
• @Oxbowerce This is what I explained in the answer to the linked question. Apr 22 at 7:40

You are setting input_shape=(None, None, 1) but your x_train has an input shape of (20,1,1,1). Firstly, these should have the same dimensions, with each None in the input shape indicating a variable batch dimension. The simple approach would be to have input_shape=(None, 1) and x_train have an input shape of (20,1) but alternatively having an input_shape=(None, None, None, 1) and x_train having an input shape of (20, 1, 1, 1) would also work. However, you must also ensure that y_train has the same batch dimensions as x_train. So in the simple case this would be (20, 1), but if x_train has a shape (20, 1, 1, 1) then y_train should also have a shape of (20, 1, 1, 1).
{"extraction_info": {"found_math": true, "script_math_tex": 0, "script_math_asciimath": 0, "math_annotations": 0, "math_alttext": 0, "mathml": 0, "mathjax_tag": 0, "mathjax_inline_tex": 0, "mathjax_display_tex": 0, "mathjax_asciimath": 1, "img_math": 0, "codecogs_latex": 0, "wp_latex": 0, "mimetex.cgi": 0, "/images/math/codecogs": 0, "mathtex.cgi": 0, "katex": 0, "math-container": 0, "wp-katex-eq": 0, "align": 0, "equation": 0, "x-ck12": 0, "texerror": 0, "math_score": 0.31947943568229675, "perplexity": 3566.5317434071167}, "config": {"markdown_headings": true, "markdown_code": true, "boilerplate_config": {"ratio_threshold": 0.18, "absolute_threshold": 10, "end_threshold": 15, "enable": true}, "remove_buttons": true, "remove_image_figures": true, "remove_link_clusters": true, "table_config": {"min_rows": 2, "min_cols": 3, "format": "plain"}, "remove_chinese": true, "remove_edit_buttons": true, "extract_latex": true}, "warc_path": "s3://commoncrawl/crawl-data/CC-MAIN-2021-39/segments/1631780057347.80/warc/CC-MAIN-20210922102402-20210922132402-00230.warc.gz"}