text: The models and methodology in these studies 
  differ, so the evidence is inconclusive. In particular, 
  Voita et al. (2019b) ﬁnd that before pruning the 
  majority of heads are syntactic, and Prasanna et al. 
  (2020) – that the majority of heads do not have 
  potentially non-trivial attention patterns. 
  
  Developing methods 
  to "teach" reasoning. 
  While large pre-trained models have a lot of knowl- 
  edge, they often fail if any reasoning needs to be 
  performed on top of the facts they possess (Tal- 
  mor et al., 2019, see also subsection 3.3). For in- 
  stance, Richardson et al. (2020) propose a method 
  to "teach" BERT quantiﬁcation, conditionals, com- 
  paratives, and boolean coordination. 
  
  An important limitation of the current head and 
  layer ablation studies (Michel et al., 2019; Koval- 
  eva et al., 2019) is that they inherently assume 
  that certain knowledge is contained in heads/layers. 
  However, there is evidence of more diffuse rep- 
  resentations spread across the full network, such 
  as the gradual increase in accuracy on difﬁcult se- 
  mantic parsing tasks (Tenney et al., 2019a) or the 
  absence of heads that would perform parsing "in 
  general" (Clark et al., 2019; Htut et al., 2019). If so, 
  ablating individual components harms the weight- 
  sharing mechanism. Conclusions from component 
  ablations are also problematic if the same informa- 
  tion is duplicated elsewhere in the network. 