Validation Loss Fluctuates then Decrease alongside Validation Accuracy Increases
This question has been asked a year ago when I faced this problem I searched and it hasn't any answers on it, I tried different ways and finally, data augmentation helped me.
I used data augmentation and a very small learning rate. If the fluctuations are big, the batch size should be increased, and the learning rate should be decreased. After all, using more epochs helps you to have an almost smooth plot.