Dataset too small for multiclass text classification
For pretrained models in NLP, look at BERT and RoBERTa. If you can find a language model trained on your data's superset on Huggingface, then, use that pretrained model.
In order to multiclass classification, since your data is less, look at augmentations in NLP (most notably, backtranslation amongst others). Use focal loss (to handle class imbalance).
Since, you are going to finetune use small learning, 1e-5. But, you will be adding your own layers also, so keep 1e-5 for the pretrained model and 1e-3 for the new layer you put.