text: 4.2.2 Attention to special tokens 
  Kovaleva et al. (2019) show that most self- 
  attention heads do not directly encode any non- 
  trivial linguistic information, at least when ﬁne- 
  tuned on GLUE (Wang et al., 2018), since only less 
  than 50% of heads exhibit the "heterogeneous" pat- 
  tern. Much of the model produced the vertical pat- 
  tern (attention to [CLS], [SEP], and punctuation 
  tokens), consistent with the observations by Clark 
  et al. (2019). This redundancy is likely related to 
  the overparameterization issue (see section 6).  Both Clark et al. (2019) and Htut et al. (2019) 
  conclude that no single head has the complete 
  syntactic tree information, in line with evidence 
  of partial knowledge of syntax (cf. subsection 3.1). 
  However, Clark et al. (2019) identify a BERT head 