Transfer Learning on Resnets/VGGs -- Validation accuracy can never be over 75%
There was a Contest in Kaggle - SIIM-ISIC Melanoma Classification.  In this contest also tasks for participants to classify two class: malignant &amp; benign. 
But in the dataset, there were the same images in both the class. And this produces the Dataleakage. 
Dataleakage,  also can produce in the same class images both in the training and in the validation set.
And to solve this problem Chris Deotte(Kaggle Grandmaster), Applied RAPIDS cuML kNN using CNN embeddings for finding the similar images and then remove those image.
Here are link of the Chris Deotte Kaggle Notebook.
This is the one solution to detect similar images for classes if the model does not work well. There are obviously other factors to improving model accuracy. Best of luck.