Transfer Learning on Resnets/VGGs -- Validation accuracy can never be over 75% I am trying to classify skin cancer images into two categories -- malignant and benign. Literatures suggest that using pre-trained resnet/vgg network achieves more than 90% accuracy. However, with my dataset, whatever I try, the validation accuracy can never be greater than 75%.
I am using a well-balanced dataset where there are 500 malignant and 500 benign images in the training dataset. The number of images is on a smaller end; however, the fact that I am using transfer learning and other literatures showing that with only ~1000 images they could achieve more than 90% accuracy, I believe there's something wrong with the transfer learning logic than the data itself.
I additionally have tried the followings:

Used different number of batches (8/16/32/64/128) and different learning rates (from 1e-7 to 0.1)
&quot;unfreezed (change require_grad=False to True)&quot; the pretrained resnet/vgg after the validation error platues and updated the weights for smaller epochs (a.k.a. fine tuning the pretrained network after setting require_grad=True after validation accuracy platues)
Decayed the learning rate using scheduler
used other nets: inception v3, GoogleNet
rather than having one fully connected layer, used dropout-FC-relu-FC

This is the core logic:
model_ft = models.resnet18(pretrained=True) # some other place in my code, I set all of resnet18 param to require_grad=False
num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, 2)

model_ft = model_ft.to(device)

criterion = nn.CrossEntropyLoss()

optimizer_ft = optim.SGD(model_ft.parameters(), lr=0.001, momentum=0.9)

and this is how I normalize my data (I have tried cropping, rotating, and horizontal-flipping; none of these boosted the performance. For now, in order to preserve some relevant clinical features, I decided not to touch the data):
data_transforms = {
    'train': transforms.Compose([
        transforms.Resize(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Resize(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

Can someone please let me know what I could do to improve the model performance? Without too much of a data pre-processing, we think the model should achieve well-above 85% accuracy in the validation set.
There was a Contest in Kaggle - SIIM-ISIC Melanoma Classification.  In this contest also tasks for participants to classify two class: malignant &amp; benign. 
But in the dataset, there were the same images in both the class. And this produces the Dataleakage. 
Dataleakage,  also can produce in the same class images both in the training and in the validation set.
And to solve this problem Chris Deotte(Kaggle Grandmaster), Applied RAPIDS cuML kNN using CNN embeddings for finding the similar images and then remove those image.
Here are link of the Chris Deotte Kaggle Notebook.
This is the one solution to detect similar images for classes if the model does not work well. There are obviously other factors to improving model accuracy. Best of luck.