Trained BERT models perform unpredictably on test set
BERT-style finetuning is known for its instability. Some aspects to take into account when having this kind of issues are:

The number of epochs typically used to finetune BERT models is normally around 3.
The main source of instability is that the authors of the original BERT article suggested using the Adam optimizer but disabling the bias compensation (such a variant became known as &quot;BertAdam&quot;).
Currently, practitioners have shifted from Adam to AdamW as optimizer.
It is typical to do multiple &quot;restarts&quot;, that is, train the model multiple times and choose the best performing one on the validation data.
Model checkpoints are normally saved after each epoch. The model we chose is the checkpoint with best validation loss among all epoch of every restart we tried.

There are two main articles that study BERT-like finetuning instabilities that may be of use to you. They describe in detail most of the aspects I mentioned before:

On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines
Revisiting Few-sample BERT Fine-tuning