Seemingly good results with training a CNN but bad when testing I have an image classification task, and I am using Keras for a network with CNN layers, with what seems like good results in training, translates to poor performance in testing. Upon training, I quickly see an increase in accuracy and validation accuracy to around the following level:

4678/4678 [==============================] - 2s 427us/step - loss: 0.0607 - acc: 0.9795 - val_loss: 0.1605 - val_acc: 0.9590


Now my first reaction was that this might be too good to be true, turns it was. In testing the model, it had very bad performance.

Bashar Haddad suggests that there may be a data imbalance problem, but with the following data specifications of data, before being split into 25% validation, I don't think that's my problem:

(data is being fit consecutively in a for loop)
average number of datapoints in training: 4500
average number of datapoints in testing: 1500
average number of classes: 46 (max 49)


and most of the classes had a good amount of datapoints in each.

Based on this, I see the user found increasing batch size and decreasing learning rate helped, and I fit the model in the following way... 

varying batch size from 100 -&gt; 500
varying epochs size from 50 -&gt; 250
cnn_model.fit(X_train, y_train,
                        batch_size=500,
                        epochs=50,
                        verbose=1,
                        shuffle=True,
                        validation_data=(X_test, y_test)
            )


Varying batch_size and epochs had little to no effect. 

With both accuracy and validation accuracy high, and also very little difference between them while training, I am confused as to why using this model on test data results in such bad performance.

What could be reasons for this? And possible solutions to solve it?
You model might just overfit. This is a classical case with neural networks. It means that your model is very good on training data but performs poorly on testing data, i.e. it is very bad at generalizing...

You can monitor the performance on a validation set and use dropout to circumvent overfitting...

[EDIT] As @Jan van der Vegt said, overfitting on validation set is unlikely. Something similar happened to me in the past and it turned out that I was using a very small validation set. I'm still not sure why but I had very similar performance on both train and validation sets. As soon as I increased the size of the validation set, this behaviour disappeared. Although data leakage was indeed my first guess, I could not find any error in the way datasets were created. So my advice is: make sure that you don't have data leakage between training and validation set and make sure that your validation scores can be trusted by having a large enough validation size...