What is the typical accuracy of masked language models during BERT pretraining? I was reading the BERT paper but I didn't find any tables concerning the performance of the masked language models during pretraining. Does anyone know the accuracy of BERT's masked language model?
It is very hard to tell and researchers usually don't measure it because it's not a comparable number such as GLUE score. It depends on size of your vocabulary and also the masking strategy (for example newer BERT successors use span-masking). BERT is a subword language model (uses WordPiece tokenizer) so if you use large vocab then sentences are usually tokenized to &quot;smaller&quot; subword units that are easier to predict. I trained a BERT-like model for Czech language, which is morphologically more complex than English, and got top@1 test accuracy ~0.55, top@3 test accuracy ~0.68 after 1.5M steps with 30K vocab size.