Pre-training models, like BERT have demonstrated great promise in the field of representation learning. BERT is a language model based on the transformer architecture and is pre-trained on a large corpus of text. Throughout the pre-training phase, BERT learns to recognize patterns and connections among words by engaging in self-supervised tasks, such as masked language modeling and predicting the next sentence. The resulting contextualized word embeddings can be fine-tuned for downstream tasks such as sentiment analysis, question answering, and named entity recognition.