Will BERT embedding be always same for a given document when used as a feature extractor
BERT is deterministic. There is no variation unless you parse your tokens differently in succeeding runs. Here is the original paper the model architecture is based off of Transformer Paper. Note that in every layer, the only operations used for the most part are matrix multiplications, concatenations, basic ops, and layer normalizations, all of which are deterministic.