recent advances in deep learning transformer models in late google published its findings about the transformer networka groundbreaking deep learning model for nlp the paper "attention is all you need(shift in the research community for language models for whilernns were the best choice to process sequential data howeverthe sequential processing and comparatively poor performance on long-term dependencies brought various challenges to large nlp tasks the transformer network plays vital role in outperforming in such use cases transformer networks can train in parallelreducing the compute time by huge margin they are based on self-attention mechanism and dispense the recurrence and convolutions entirely (thusa faster computethe transformer model achieves bilingual evaluation understudy (bleuscore on the wmt english to-german translation taskimproving over the existing best resultsincluding ensemblesby more than two bleu idirectional encoder representations from transformers in year after publishing transformer networksresearchers at google ai language open sourced new technique for nlp called bidirectional encoder representations from transformers (bertbert relies on transformerbut with some variations vanilla transformer consists of an encoder and decoder architecturethe encoder reads the text inputwhile the decoder produces the prediction berthoweverleverages only the encoder part because bert' goal is to generate language representation modelthis is ideal one of unique differentiators of bert is its semi-supervised setting in this settingthe process first focuses on pretraining (unsupervised)where large