text: More recently, Kobayashi et al. (2020) showed 
  that the norms of attention-weighted input vec- 
  tors, which yield a more intuitive interpretation 
  of self-attention, reduce the attention to special to- 
  kens. However, even when the attention weights 
  are normed, it is still not the case that most heads 
  that do the "heavy lifting" are even potentially in- 
  terpretable (Prasanna et al., 2020). 
  
  One methodological choice in in many studies 
  of attention is to focus on inter-word attention and 
  simply exclude special tokens (e.g. Lin et al. (2019) 
  and Htut et al. (2019)). However, if attention to 
  special tokens actually matters at inference time, 
  drawing conclusions purely from inter-word atten- 
  tion patterns does not seem warranted. 
  
  Figure 4: BERT layer transferability (columns corre- 
  spond to probing tasks, Liu et al. (2019a). 
  
  of hierarchical sentence structure, as detected by 
  the probing tasks of predicting the token index, the 
  main auxiliary verb and the sentence subject. 
  
  There is a wide consensus in studies with differ- 
  ent tasks, datasets and methodologies that syntac- 
  tic information is most prominent in the middle 
  layers of BERT.4 Hewitt and Manning (2019) had 
  the most success reconstructing syntactic tree depth 
  from the middle BERT layers (6-9 for base-BERT, 
  14-19 for BERT-large). Goldberg (2019) reports 
  the best subject-verb agreement around layers 8- 
  9, and the performance on syntactic probing tasks 
  used by Jawahar et al. (2019) also seems to peak 
  around the middle of the model. The prominence 
  of syntactic information in the middle BERT layers 
  is related to Liu et al. (2019a)’s observation that the 
  middle layers of Transformers are best-performing 
  overall and the most transferable across tasks (see 
  Figure 4). 