Diagonal Heterogeneous
Vertical Vertical + diagonal Block
[CLS] [CLS] [SEP] [SEP] [SEP] [SEP] [SEP] [SEP] [CLS] [CLS] [SEP] [SEP] [SEP] [SEP] [CLS] 
Figure 3: Attention patterns in BERT (Kovaleva et al., 2019)
ies) insufﬁcient (Warstadt et al., 2019). A given
method might also favor one model over another,
e.g., RoBERTa trails BERT with one tree extraction
method, but leads with another (Htut et al., 2019).
The choice of linguistic formalism also matters
(Kuznetsov and Gurevych, 2020).
In view of all that, the alternative is to focus on
identifying what BERT actually relies on at infer-
ence time. This direction is currently pursued both
at the level of architecture blocks (to be discussed
in detail in subsection 6.3), and at the level of in-
formation encoded in model weights. Amnesic
probing (Elazar et al., 2020) aims to speciﬁcally
remove certain information from the model and see
how it changes performance, ﬁnding, for example,
that language modeling does rely on part-of-speech
information.
Another direction is information-theoretic prob-
ing. Pimentel et al. (2020) operationalize prob-
ing as estimating mutual information between the
learned representation and a given linguistic prop-
erty, which highlights that the focus should be not
on the amount of information contained in a rep-
resentation, but rather on how easily it can be ex-
tracted from it. V oita and Titov (2020) quantify
the amount of effort needed to extract information
from a given representation as minimum descrip-
tion length needed to communicate both the probe
size and the amount of data required for it to do
well on a task.
4 Localizing linguistic knowledge
4.1 BERT embeddings
In studies of BERT, the term "embedding" refers
to the output of a Transformer layer (typically, the
ﬁnal one). Both conventional static embeddings
(Mikolov et al., 2013) and BERT-style embeddings
can be viewed in terms of mutual information max-
imization (Kong et al., 2019), but the latter are
contextualized. Every token is represented by a
vector dependent on the particular context of occur-
rence, and contains at least some information about
that context (Miaschi and Dell’Orletta, 2020).
Several studies reported that distilled contex-
tualized embeddings better encode lexical se-
mantic information (i.e. they are better at tra-
ditional word-level tasks such as word similarity).
The methods to distill a contextualized represen-
tation into static include aggregating the informa-
tion across multiple contexts (Akbik et al., 2019;
Bommasani et al., 2020), encoding "semantically
bleached" sentences that rely almost exclusively on
the meaning of a given word (e.g. "This is <>")
(May et al., 2019), and even using contextualized
embeddings to train static embeddings (Wang et al.,
2020d).
But this is not to say that there is no room for
improvement. Ethayarajh (2019) measure how
similar the embeddings for identical words are in
every layer, reporting that later BERT layers pro-
duce more context-speciﬁc representations3. They
also ﬁnd that BERT embeddings occupy a narrow
cone in the vector space, and this effect increases
from the earlier to later layers. That is, two ran-
dom words will on average have a much higher
cosine similarity than expected if embeddings
were directionally uniform (isotropic) . Since
isotropy was shown to be beneﬁcial for static word
embeddings (Mu and Viswanath, 2018), this might
be a fruitful direction to explore for BERT.
Since BERT embeddings are contextualized, an
interesting question is to what extent they cap-
ture phenomena like polysemy and homonymy.
There is indeed evidence that BERT’s contextu-
alized embeddings form distinct clusters corre-
sponding to word senses(Wiedemann et al., 2019;
Schmidt and Hofmann, 2020), making BERT suc-
cessful at word sense disambiguation task. How-
ever, Mickus et al. (2019) note thatthe representa-
tions of the same word depend on the position
of the sentence in which it occurs , likely due to