Doing a fine tuning after a transfer learning
Generally, it will increase the performance of the model, but you'll have to experiment with it and check for your case. Try it, till you get a better result. If after many trials also you are not able to increase the performance of the model then it means that you are limited by the architecture of the pre-trained model.
Refer to this http://cs231n.github.io/transfer-learning/ for more details.