Doing a fine tuning after a transfer learning I read about fine tuning and transfer learning for CNNs and was wondering if we can do fine tuning after using transfer learning on the same CNN? If so, will this increase the performance of the model or decrease it?
Generally, it will increase the performance of the model, but you'll have to experiment with it and check for your case. Try it, till you get a better result. If after many trials also you are not able to increase the performance of the model then it means that you are limited by the architecture of the pre-trained model.
Refer to this http://cs231n.github.io/transfer-learning/ for more details.