BERT has a non deterministic behaviour
I had the exact same problem using the pytorch implementation, until I realised I didn't set the model in eval mode. Hence, dropout was still activated. I guess this is also the source of your undeterministic behavior.

Fix using pytorch:

model.eval()


Using tensorflow with an Estimator, ensure you call estimator.evaluate or estimator.predict during testing. If you exported the model in some ways, then you should check the tensorflow documentation.