text: Since BERT embeddings are contextualized, an 
  interesting question is to what extent they cap- 
  ture phenomena like polysemy and homonymy. 
  There is indeed evidence that BERT’s contextu- 
  alized embeddings form distinct clusters corre- 
  sponding to word senses (Wiedemann et al., 2019; 
  Schmidt and Hofmann, 2020), making BERT suc- 
  cessful at word sense disambiguation task. How- 
  ever, Mickus et al. (2019) note that the representa- 
  tions of the same word depend on the position 
  of the sentence in which it occurs, likely due to 
  the NSP objective. This is not desirable from the 
  linguistic point of view, and could be a promising 
  
  3Voita et al. (2019a) look at the evolution of token embed- 
  dings, showing that in the earlier Transformer layers, MLM 
  forces the acquisition of contextual information at the expense 
  of the token identity, which gets recreated in later layers. 