Overfitting with text classification using Transformers
Your model is overfitting. You should try standard methods people use to prevent overfitting:


Larger dropout (up to 0.5), in low-resource setups word dropout (i.e., randomly masking input tokens) also sometimes help (0.1-0.3 might be reasonable values).
If you have many input classes, label smoothing can help.
You can try a smaller model dimension.


If you use a pre-trained Transformer (such as BERT), you, of course, cannot change the model dimension. In that case, you can try to set a much smaller learning rate for fine-tuning BERT than you use for training the actual classifier.