text: Abstract 
  
  Transformer-based models have pushed state 
  of the art in many areas of NLP, but our un- 
  derstanding of what is behind their success 
  is still limited. This paper is the ﬁrst sur- 
  vey of over 150 studies of the popular BERT 
  model. We review the current state of knowl- 
  edge about how BERT works, what kind 
  of information it learns and how it is repre- 
  sented, common modiﬁcations to its training 
  objectives and architecture, the overparame- 
  terization issue and approaches to compres- 
  sion. We then outline directions for future 
  research. 