then check which of them survive the pruning, ﬁnd-
ing that the syntactic and positional heads are the
last ones to go. For BERT, Prasanna et al. (2020)
go in the opposite direction: pruning on the basis of
importance scores, and interpreting the remaining
"good" subnetwork. With respect to self-attention
heads speciﬁcally, it does not seem to be the case
that only the heads that potentially encode non-
trivial linguistic patterns survive the pruning.
The models and methodology in these studies
differ, so the evidence is inconclusive. In particular,
V oita et al. (2019b) ﬁnd that before pruning the
majority of heads are syntactic, and Prasanna et al.
(2020) – that the majority of heads do not have
potentially non-trivial attention patterns.
An important limitation of the current head and
layer ablation studies (Michel et al., 2019; Koval-
eva et al., 2019) is that they inherently assume
that certain knowledge is contained in heads/layers.
However, there is evidence of more diffuse rep-
resentations spread across the full network, such
as the gradual increase in accuracy on difﬁcult se-
mantic parsing tasks (Tenney et al., 2019a) or the
absence of heads that would perform parsing "in
general" (Clark et al., 2019; Htut et al., 2019). If so,
ablating individual components harms the weight-
sharing mechanism. Conclusions from component
ablations are also problematic if the same informa-
tion is duplicated elsewhere in the network.
7 Directions for further research
BERTology has clearly come a long way, but it
is fair to say we still have more questions than
answers about how BERT works. In this section,
we list what we believe to be the most promising
directions for further research.
Benchmarks that require verbal reasoning.
While BERT enabled breakthroughs on many NLP
benchmarks, a growing list of analysis papers are
showing that its language skills are not as impres-
sive as it seems. In particular, it was shown to rely
on shallow heuristics in natural language inference
(McCoy et al., 2019b; Zellers et al., 2019; Jin et al.,
2020), reading comprehension (Si et al., 2019a;
Rogers et al., 2020; Sugawara et al., 2020; Si et al.,
2019b; Yogatama et al., 2019), argument reason-
ing comprehension (Niven and Kao, 2019), and
text classiﬁcation (Jin et al., 2020). Such heuristics
can even be used to reconstruct a non-publicly-
available model (Krishna et al., 2020). As with
any optimization method, if there is a shortcut in
the data, we have no reason to expect BERT to not
learn it. But harder datasets that cannot be resolved
with shallow heuristics are unlikely to emerge if
their development is not as valued as modeling
work.
Benchmarks for the full range of linguistic
competence. While the language models seem to
acquire a great deal of knowledge about language,
we do not currently have comprehensive stress tests
for different aspects of linguistic knowledge. A
step in this direction is the "Checklist" behavioral
testing (Ribeiro et al., 2020), the best paper at ACL
2020. Ideally, such tests would measure not only
errors, but also sensitivity (Ettinger, 2019).
Developing methods to "teach" reasoning.
While large pre-trained models have a lot of knowl-
edge, they often fail if any reasoning needs to be
performed on top of the facts they possess (Tal-
mor et al., 2019, see also subsection 3.3). For in-
stance, Richardson et al. (2020) propose a method
to "teach" BERT quantiﬁcation, conditionals, com-
paratives, and boolean coordination.
Learning what happens at inference time.
Most BERT analysis papers focus on different
probes of the model, with the goal to ﬁnd what
the language model "knows". However, probing
studies have limitations (subsection 3.4), and to this
point, far fewer papers have focused on discovering
what knowledge actually gets used. Several promis-
ing directions are the "amnesic probing" (Elazar
et al., 2020), identifying features important for pre-
diction for a given task (Arkhangelskaia and Dutta,