How to use pretrained checkpoints of BERT model on semantic text similarity task?
Have a look at https://medium.com/the-artificial-impostor/news-topic-similarity-measure-using-pretrained-bert-model-1dbfe6a66f1d

You can have the two sentences as first and second use the next sentence score as a similarity measure. You can further fine-tune your model on some semantic similarity tasks like Sent-Eval or your own dataset if you have one