Using trained BERT Model and Data Preprocessing
When using a pre-trained BERT embeddings from pytorch (which are then fine-tuned), should the text data fed into the model be pre-processed like in any standard NLP task?
For instance, should  stemming, removing low frequency words, de-captilisation, be performed or should the raw text simply be passed to `transformers.BertTokenizer'?

I think preprocessing will not change your output predictions. I will try to explain for each case you mentioned -

stemming or lemmatization :
Bert uses BPE (Byte- Pair Encoding to shrink its vocab size), so words like run and running will ultimately be decoded to run + ##ing.
So it's better not to convert running into run because, in some NLP problems, you need that information.
De-Capitalization - Bert provides two models (lowercase and uncased). One converts your sentence into lowercase, and others will not change related to the capitalization of your sentence. So you don't have to do any changes here just select the model for your use case.
Removing high-frequency words -
Bert uses the Transformer model, which works on the attention principal.
So when you finetune it on any problem, it will look only on those words which will impact the output and not on words which are common in all data.

