Overfitting while fine-tuning pre-trained transformer
What makes you think your model is overfitting? Are you concerned about the difference between the training loss and validation loss?
If so, this is not overfitting. Overfitting is when the weights learned from training fail to generalize to data unseen during model training.
In the case of the plot shown here, your validation loss continues to go down, so your model continues to improve its ability to generalize to unseen data.
Once your validation loss starts creeping upward, then you have begun to overfit.
See chapter 5 of Jeremy Howard's Deep Learning for Coders with fastai and PyTorch for more details. https://www.amazon.com/Deep-Learning-Coders-fastai-PyTorch/dp/1492045527