model.predict() accuracy extremely low on training dataset This question is similar to this.
I'm new to ML, and I am trying to classify breast cancer histology images using EfficientNets with Transfer Learning. The dataset is small (400 images in total - there are 4 classes and all classes are equally balanced) and I am using ImageNet weights, and fine-tuning the model by freezing the first two blocks.
I've implemented a model with Keras that reaches a training accuracy of ~90% after 30 epochs.
When trying to use model.predict on the training dataset (to understand the results of the predict), I expect the results to be good since the prediction is being done on data that the model has already seen but the results I get are extremely low. The prediction accuracy in the report created by sklearn.metrics.classification_report is 27.81%.
After trying various methods to change this, I still have not been able to improve or understand the result.
The labels I use:
{0: 'Benign', 1: 'InSitu', 2: 'Invasive', 3: 'Normal'}
After playing around with code, I found that the expected labels and the results were as follows:
expected:
[0, 3, 1, 0, 0, 3, 2, 0, 2, 2, 1, 0, 3, 1, 2, 3, 2, 1, 3, 2, 3, 3, 1, 0, 3, 1, 3, 2, 2, 1, 1, 3, 0, 1, 3, 0, 1, 2, 0, 3, 2, 3, 2, 3, 1, 0, 1, 0, 1, 1, 2, 1, 2, 2, 2, 1, 1, 0, 3, 3, 0, 3, 1, 1, 3, 1, 1, 0, 2, 1, 0, 0, 1, 1, 1, 2, 2, 1, 3, 1, 1, 3, 3, 3, 0, 1, 1, 0, 2, 3, 0, 3, 1, 2, 2, 3, 0, 3, 3, 0, 0, 0, 2, 0, 1, 0, 2, 1, 3, 1, 0, 2, 2, 0, 1, 3, 1, 1, 3, 2, 3, 0, 2, 1, 2, 3, 3, 0, 0, 1, 3, 0, 0, 3, 2, 3, 1, 2, 0, 0, 0, 3, 3, 3, 0, 0, 0, 0, 3, 0, 0, 3, 2, 1, 3, 2, 1, 1, 2, 2, 3, 3, 1, 2, 1, 0, 3, 3, 0, 1, 2, 2, 1, 3, 1, 3, 0, 1, 3, 2, 3, 3, 0, 2, 1, 3, 2, 0, 2, 0, 3, 3, 2, 3, 0, 2, 2, 0, 3, 2, 2, 1, 1, 3, 2, 3, 0, 2, 0, 3, 1, 0, 1, 2, 0, 2, 0, 0, 2, 2, 2, 1, 1, 0, 3, 0, 1, 1, 2, 2, 0, 0, 3, 0, 1, 0, 1, 1, 1, 3, 2, 2, 3, 2, 1, 2, 0, 1, 2, 1, 1, 1, 3, 1, 2, 3, 1, 2, 0, 2, 3, 1, 2, 1, 2, 1, 2, 2, 2, 0, 3, 3, 1, 3, 3, 3, 0, 1, 1, 2, 2, 3, 3, 2, 0, 0, 2, 1, 0, 1, 3, 0, 2, 0, 0, 3, 2, 1, 2, 2, 1, 0, 0, 0, 2, 3, 3, 0, 2, 3, 1, 0, 0, 0, 0, 3, 0, 3, 2, 2]
result:
[3 0 3 0 3 3 3 1 0 2 3 3 2 1 3 2 1 1 3 1 2 3 2 1 0 3 3 0 2 1 2 2 2 1 3 3 0 2 0 2 1 1 2 0 0 0 0 0 2 3 0 1 3 0 1 1 2 0 0 2 3 3 0 2 2 1 2 3 2 2 1 2 1 3 3 3 1 0 1 3 0 1 2 3 2 0 1 3 0 3 3 0 0 3 3 3 1 3 0 2 3 3 3 0 3 0 0 1 3 1 1 3 0 0 0 3 1 1 0 1 3 2 0 3 2 3 0 1 0 2 1 1 2 3 2 2 1 2 1 2 1 1 0 1 1 1 3 2 2 1 3 1 1 0 2 1 1 1 0 1 0 0 0 0 2 2 0 0 3 2 3 3 2 3 3 3 2 1 2 3 2 3 2 2 1 1 0 1 0 2 3 2 1 3 0 1 0 1 2 0 3 1 2 0 1 1 2 0 3 1 3 2 1 2 2 1 3 0 1 2 3 3 1 3 2 1 2 3 0 3 0 2 2 3 0 1 3 1 0 2 2 1 0 2 3 1 0 1 3 3 2 2 0 0 2 1 0 0 3 2 2 3 1 0 2 1 0 0 0 2 0 2 1 3 1 0 3 1 1 1 1 1 2 0 3 3 0 3 0 0 2 0 3 1 1 3 1 0 2 2 3 2 2 2 2 2 1 0 3 3 2 0 0 0 1 2 3 2 1 2]
The code I use to predict:
training_data.reset()
predicts = model.predict(training_data, steps=20, verbose=1)
predict_class = np.argmax(predicts, axis = 1) # could this be the problem?
errors = np.where(predict_class != truth)[0]
print(&quot;No of errors = {}/{}&quot;.format(len(errors),training_data.samples))

Where training_data is an image generator defined as follows:
training_gen = ImageDataGenerator(
    rescale=1./255,
    width_shift_range=0.1,
    height_shift_range=0.1,
    zoom_range=0.15,
    horizontal_flip=True,
    vertical_flip=True,
    preprocessing_function = preprocess_img
    )

training_data = training_gen.flow_from_dataframe(
    dataframe=df_train,
    directory = train_dir,
    x_col=&quot;ID&quot;,
    y_col=&quot;Truth&quot;,
    target_size=(height,width),
    batch_size=batch_size,
    # shuffle=True,
    class_mode=&quot;categorical&quot;,
    seed=seed_augment
)

Am I misunderstanding when I assume that the predict function will give me good accuracy on training data? Does this imply that my model is actually really bad?
Any direction would be appreciated.
The problem with the way the prediction is done in the code found in the question is that the generator is shuffling the data (even though the shuffle=True is commented out, True is the default). Additionally, the batch_size in the code is set to 16. Setting batch_size = 1 produced the correct output for me, and the resulting accuracy increased to what I was expecting.
Oh, and don't use the same generator to test the predict function!
Hope this helps someone! :)