Is it possible to pre-train a CNN in a self-supervised way so that it can later be used to solve an instance segmentation task?
Is it possible to use SSL to pre-train e.g. a faster R-CNN on a pretext task (for example, rotation), then use this pre-trained model for instance segmentation with the aim to get better accuracy?

Yes, it's possible and this has already been done. I don't know the details (because I have not yet read those papers), but I will provide you with some links to some potentially useful papers (based on their titles and abstracts) and associated code.

Learning Instance Segmentation by Interaction (code, 2018)
Self-supervised Transfer Learning for Instance Segmentation through Physical Interaction (code, 2019)
White Matter Tract Segmentation with Self-supervised Learning (code, 2020)
Self-supervised Depth Estimation to Regularise Semantic Segmentation in Knee Arthroscopy (code, 2020)
Simulation of Brain Resection for Cavity Segmentation Using Self-supervised and Semi-supervised Learning (code, 2020)

You can probably find more relevant papers here, where I also found some of the just cited papers.
The pre-text tasks designed in these papers could be useful in your case, but it may also turn out that you need to develop other pre-text tasks or combine multiple of them.
Maybe you can start from some pre-trained faster R-CNN or some appropriate model for instance segmentation (that you can find on the web, for example, here), which has been pre-trained on some imagery data similar to yours (either with SSL or by other means), then try to fine-tune this model with your labeled dataset for instance segmentation, and see if you get better results than just training a faster R-CNN from scratch. Eventually, if this pre-trained model does not lead to higher performance, you could pre-train it yourself with some SSL technique that you can come up with or one that is described in the literature. Of course, you should probably use a pre-trained model that has been pre-trained with data that is relevant for your downstream task (i.e. the instance segmentation task). You didn't describe the details of your unlabelled and labeled data, so I cannot be more specific (and I wouldn't currently be able to, in any case, because I didn't fully read those papers, and my experience with SSL techniques is mostly theoretical).
For more info about SSL, take a look at this and this answers.