recent advances in deep learning corpus of text data (largely available over the internetis used for training language representation models nextthe model is trained and fine-tuned in supervised fashion for the specific use case of interest an example would the sentiment classification use case we explored in you can find more details about bert at com/ / /open-sourcing-bert-state-of-art-pre html bert uses two strategies for trainingmasked language modeling and next sentence prediction for masked language modelingwhile feeding word sequences into bertroughly of the words in each sequence are replaced with [masktoken the model then attempts to predict the original value of the masked words based on the context provided by the othernon-maskedwords in the sequence allennlp has released fun tool that uses bert in the backend (see simple demothe model predicted the word in [maskas car with probability figure - allennlp demo