Pre-training language models on text sequences can be accomplished using a combination of tokenization and mask prediction. In tokenization, commonly occurring sequences are used to generate a vocabulary for the model . This vocabulary is used to map text to a sequence of integer token IDs which are used as input to the model. For mask prediction, tokens are randomly masked and the model is trained to reproduce the original sequence based on context. Therefore, for a given masked token, the model predicts a probability that each token in the vocabulary will occur at that location.