avenue for future work.
The above discussion concerns token embed-
dings, but BERT is typically used as a sentence or
text encoder. The standard way to generate sen-
tence or text representations for classiﬁcation is
to use the [CLS] token, but alternatives are also
being discussed, including concatenation of token
representations (Tanaka et al., 2020), normalized
mean (Tanaka et al., 2020), and layer activations
(Ma et al., 2019). See Toshniwal et al. (2020) for a
systematic comparison of several methods across
tasks and sentence encoders.
4.2 Self-attention heads
Several studies proposed classiﬁcation of attention
head types. Raganato and Tiedemann (2018) dis-
cuss attending to the token itself, previous/next
tokens and the sentence end. Clark et al. (2019)
distinguish between attending to previous/next to-
kens, [CLS], [SEP], punctuation, and "attending
broadly" over the sequence. Kovaleva et al. (2019)
propose 5 patterns shown in Figure 3.
4.2.1 Heads with linguistic functions
The "heterogeneous" attention pattern shown in
Figure 3 could potentially be linguistically inter-
pretable, and a number of studies focused on iden-
tifying the functions of self-attention heads. In
particular, some BERT heads seem to specialize
in certain types of syntactic relations. Htut et al.
(2019) and Clark et al. (2019) report that there
are BERT heads that attended signiﬁcantly more
than a random baseline to words in certain syntac-
tic positions. The datasets and methods used in
these studies differ, but they both ﬁnd that there are
heads that attend to words in obj role more than
the positional baseline. The evidence for nsubj,
advmod, and amod varies between these two stud-
ies. The overall conclusion is also supported by
V oita et al. (2019b)’s study of the base Transformer
in machine translation context. Hoover et al. (2019)
hypothesize that even complex dependencies like
dobj are encoded by a combination of heads
rather than a single head, but this work is limited
to qualitative analysis. Zhao and Bethard (2020)
looked speciﬁcally for the heads encoding negation
scope.
Both Clark et al. (2019) and Htut et al. (2019)
conclude that no single head has the complete
syntactic tree information, in line with evidence
of partial knowledge of syntax (cf. subsection 3.1).
However, Clark et al. (2019) identify a BERT head
that can be directly used as a classiﬁer to perform
coreference resolution on par with a rule-based
system, which by itself would seem to require quite
a lot of syntactic knowledge.
Lin et al. (2019) present evidence that atten-
tion weights are weak indicators of subject-
verb agreement and reﬂexive anaphora. Instead
of serving as strong pointers between tokens that
should be related, BERT’s self-attention weights
were close to a uniform attention baseline, but there
was some sensitivity to different types of distrac-
tors coherent with psycholinguistic data. This is
consistent with conclusions by Ettinger (2019).
To our knowledge, morphological information
in BERT heads has not been addressed, but with
the sparse attention variant by Correia et al. (2019)
in the base Transformer, some attention heads ap-
pear to merge BPE-tokenized words. For semantic
relations, there are reports of self-attention heads
encoding core frame-semantic relations (Kovaleva
et al., 2019), as well as lexicographic and common-
sense relations (Cui et al., 2020).
The overall popularity of self-attention as an in-
terpretability mechanism is due to the idea that
"attention weight has a clear meaning: how much
a particular word will be weighted when comput-
ing the next representation for the current word"
(Clark et al., 2019). This view is currently debated
(Jain and Wallace, 2019; Serrano and Smith, 2019;
Wiegreffe and Pinter, 2019; Brunner et al., 2020),
and in a multi-layer model where attention is fol-
lowed by non-linear transformations, the patterns
in individual heads do not provide a full picture.
Also, while many current papers are accompanied