performances evaluation of image classification with different distribution for train and test set I am implementing a CNN to do image classification of 4 classes representing different weathers : Haze, Sunny, Rainy, Snowy. I have as training set 3200 images, and as test set 3038 images. 

The problem is that the test and training set are different set of pictures(so to be clear, I didn't split the dataset, but I have been given a training set and a test set in different folders) and they have so a different distribution one with respect to the other. 

After training and evaluating, I have that my accuracy on the test set is 0.475313 and the loss always on the test set is 3.060471.

Also if I plot the hihstory of accuracy and of the loss I have the following:



what I don't understand is if I have the accuracy so low because I am overfitting, or because the distribution is different. And also, how do I interpret the plots I have posted about the histories of loss and accuracy? Is it normal that the gap between test and train accuracy is so large?

Thanks in advance.

[EDIT] By searching online, I have found that to cope with this kind of problems, it is possible to use domain adaptation, but honestly I have no idea how to implement it because I have just read about it.
I'm afraid you have too little data. That's why optimizer can't converge to any reasonable place. I'd suggest you try doing transfer learning first. Few links for you to check out:


https://machinelearningmastery.com/transfer-learning-for-deep-learning/
https://www.pyimagesearch.com/2019/05/20/transfer-learning-with-keras-and-deep-learning/
https://www.pyimagesearch.com/2019/06/24/change-input-shape-dimensions-for-fine-tuning-with-keras/
https://www.pyimagesearch.com/2019/06/03/fine-tuning-with-keras-and-deep-learning/


Consider also heavier augmentation. Albumentations is a nice repo, used in many kaggle competitions.