and in a multi-layer model where attention is fol-
lowed by non-linear transformations, the patterns
in individual heads do not provide a full picture.
Also, while many current papers are accompanied
by attention visualizations, and there is a growing
number of visualization tools (Vig, 2019; Hoover
et al., 2019), the visualization is typically limited
to qualitative analysis (often with cherry-picked
examples) (Belinkov and Glass, 2019), and should
not be interpreted as deﬁnitive evidence.
4.2.2 Attention to special tokens
Kovaleva et al. (2019) show that most self-
attention heads do not directly encode any non-
trivial linguistic information, at least when ﬁne-
tuned on GLUE (Wang et al., 2018), since only less
than 50% of heads exhibit the "heterogeneous" pat-
tern. Much of the model produced the vertical pat-
tern (attention to [CLS], [SEP], and punctuation
tokens), consistent with the observations by Clark
et al. (2019). This redundancy is likely related to
the overparameterization issue (see section 6).