What is the typical accuracy of masked language models during BERT pretraining?
It is very hard to tell and researchers usually don't measure it because it's not a comparable number such as GLUE score. It depends on size of your vocabulary and also the masking strategy (for example newer BERT successors use span-masking). BERT is a subword language model (uses WordPiece tokenizer) so if you use large vocab then sentences are usually tokenized to &quot;smaller&quot; subword units that are easier to predict. I trained a BERT-like model for Czech language, which is morphologically more complex than English, and got top@1 test accuracy ~0.55, top@3 test accuracy ~0.68 after 1.5M steps with 30K vocab size.