text: ies) insufﬁcient (Warstadt et al., 2019). A given 
  method might also favor one model over another, 
  e.g., RoBERTa trails BERT with one tree extraction 
  method, but leads with another (Htut et al., 2019). 
  The choice of linguistic formalism also matters 
  (Kuznetsov and Gurevych, 2020). 
  
  Several studies reported that distilled contex- 
  tualized embeddings better encode lexical se- 
  mantic information (i.e. 
  they are better at tra- 
  ditional word-level tasks such as word similarity). 
  The methods to distill a contextualized represen- 
  tation into static include aggregating the informa- 
  tion across multiple contexts (Akbik et al., 2019; 
  Bommasani et al., 2020), encoding "semantically 
  bleached" sentences that rely almost exclusively on 
  the meaning of a given word (e.g. "This is <>") 
  (May et al., 2019), and even using contextualized 
  embeddings to train static embeddings (Wang et al., 
  2020d). 
  
  In view of all that, the alternative is to focus on 
  identifying what BERT actually relies on at infer- 
  ence time. This direction is currently pursued both 
  at the level of architecture blocks (to be discussed 
  in detail in subsection 6.3), and at the level of in- 
  formation encoded in model weights. Amnesic 
  probing (Elazar et al., 2020) aims to speciﬁcally 
  remove certain information from the model and see 
  how it changes performance, ﬁnding, for example, 
  that language modeling does rely on part-of-speech 
  information. 