Bert and SVM classification
BERT is a transformer-based model. The pre-trained BERT can be used for two purposes:

Fine-tuning
Extracting embedding


You don't need to use an SVM once you're keyed into a BERT architecture. Your BERT model will generate embeddings and can be fine-tuned (ala ULMfit last layer) to perform a specific task. You could potentially just use the embeddings and then perform the task with another model but the performance would likely not be better.

So, how you want to use BERT still remains a choice. But if you can fine-tune the BERT model, it would generally yield higher performance. But you'll have to validate it based on the experiments.