text: inference time. 
  Learning what happens at 
  Most BERT analysis papers focus on different 
  probes of the model, with the goal to ﬁnd what 
  the language model "knows". However, probing 
  studies have limitations (subsection 3.4), and to this 
  point, far fewer papers have focused on discovering 
  what knowledge actually gets used. Several promis- 
  ing directions are the "amnesic probing" (Elazar 
  et al., 2020), identifying features important for pre- 
  diction for a given task (Arkhangelskaia and Dutta, 
  2019), and pruning the model to remove the non- 
  important components (Voita et al., 2019b; Michel 
  et al., 2019; Prasanna et al., 2020). 
  
  7 Directions for further research 
  
  BERTology has clearly come a long way, but it 
  is fair to say we still have more questions than 
  answers about how BERT works. In this section, 
  we list what we believe to be the most promising 
  directions for further research. 
  
  Benchmarks that require verbal reasoning. 
  While BERT enabled breakthroughs on many NLP 
  benchmarks, a growing list of analysis papers are 
  showing that its language skills are not as impres- 
  sive as it seems. In particular, it was shown to rely 
  on shallow heuristics in natural language inference 
  (McCoy et al., 2019b; Zellers et al., 2019; Jin et al., 
  2020), reading comprehension (Si et al., 2019a; 
  Rogers et al., 2020; Sugawara et al., 2020; Si et al., 
  2019b; Yogatama et al., 2019), argument reason- 
  ing comprehension (Niven and Kao, 2019), and 
  text classiﬁcation (Jin et al., 2020). Such heuristics 
  can even be used to reconstruct a non-publicly- 
  available model (Krishna et al., 2020). As with 
  any optimization method, if there is a shortcut in 
  
  8 Conclusion 