Overfitting while fine-tuning pre-trained transformer Pretrained transformers (GPT2, Bert, XLNET) is popular and useful because of their transfer learning capabilities.
Just to remind: The goal of Transfer learning is is to transfer knowledge gained from one domain/task and use that transfer/use that knowledge to solve some related tasks. This is done by training a model on a huge amount of labelled data (which we already have and probably easy to get), then remove last few layers and fine-tune the model for the new related task with task-related dataset.
I took a recent pretrained transformer published by Google called XLNET, and just add classification layer of the top of that and fine-tune the whole network. (Which is the main intention of this kind of model, correct me if I am wrong )
Problem is, the model is hugely overfitting. I have 1200 example to train and each has 350 words on average.
To overcome the overfitting, I set the dropout of each layer of the transformer from 0.1 to 0.5. This did not work. So I  decreased the number of trainable parameters (since the transformer has a huge number of parameters), by freezing first 10 layers (11 layers + 1 classification layer in total). Even that does not work. So I counted the number of trainable parameters in the last layer. There are 7680000 parameters which are very high compared to my dataset (around 1200*350= 420000 words). So, this high number of tunable parameters is the most possible reason for overfitting.
Here is the loss graph:

My questions are:
Do you see any flaw in my analysis?
Is there anything I can do to decrease overfitting? (tried with low learning rate and large batch size)
If my analysis is correct, then the claim that &quot;fine-tune pre-trained transformers with small dataset&quot; is bit misleading and datasets should not be so small. Am I correct?
What makes you think your model is overfitting? Are you concerned about the difference between the training loss and validation loss?
If so, this is not overfitting. Overfitting is when the weights learned from training fail to generalize to data unseen during model training.
In the case of the plot shown here, your validation loss continues to go down, so your model continues to improve its ability to generalize to unseen data.
Once your validation loss starts creeping upward, then you have begun to overfit.
See chapter 5 of Jeremy Howard's Deep Learning for Coders with fastai and PyTorch for more details. https://www.amazon.com/Deep-Learning-Coders-fastai-PyTorch/dp/1492045527