text: Since their introduction in 2017, Transformers 
  (Vaswani et al., 2017) have taken NLP by storm, 
  offering enhanced parallelization and better model- 
  ing of long-range dependencies. The best known 
  Transformer-based model is BERT (Devlin et al., 
  2019); it obtained state-of-the-art results in numer- 
  ous benchmarks and is still a must-have baseline. 
  While it is clear that BERT works remarkably 
  well, it is less clear why, which limits further 
  hypothesis-driven improvement of the architecture. 
  Unlike CNNs, the Transformers have little cogni- 
  tive motivation, and the size of these models limits 
  our ability to experiment with pre-training and per- 
  form ablation studies. This explains a large number 
  of studies over the past year that attempted to un- 
  derstand the reasons behind BERT’s performance. 
  In this paper, we provide an overview of what 
  has been learned to date, highlighting the questions 
  which are still unresolved. We ﬁrst consider the 
  linguistic aspects of it, i.e., the current evidence 
  regarding the types of linguistic and world knowl- 
  edge learned by BERT, as well as where and how 
  this knowledge may be stored in the model. We 
  then turn to the technical aspects of the model and 
  provide an overview of the current proposals to 