Confusing results of transfer learning accuracy
I have just started an image classification project according to the tutorial from the documentation on the pytorch website(this).In the tutorial, there is a part of code like this:

model_ft = models.resnet50(pretrained=True)
num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, 20)


And I have known the reason why the fc layer is supposed to be changed .Since my project needs to classify 20 classes , so I just changed the parameters from 2 to 20. But , I just get the accuracy of around 60%. When I dont change the fc layer like this:

model_ft = se_resnet50(pretrained = True)


It turned out that the accuracy reaches 93.75% which is much better compared with the former results.

I just couldnt figure out why I get worse classification results when I modify the fc layer. Shouldn't it be modified?

It's probably more difficult for the network to find the matching class between 20 classes than between two classes. 

For example if you give it a dog image and it need to classify it between cat, dog and horse it could send 60% cat, 30% dog 10% horse and then be wrong
while if it needs to classify it only between dog and horse it would give may be 75% dog, 25% horse and then be wright. 

The finetunnig will also be longer so you could have better result if you train it longer with the 20 classes if you haven't stop it after convergence but after a fix number of epochs.
