prisingly strong ability of these models to re-
call factual knowledge without any ﬁne-tuning
demonstrates their potential as unsupervised
open-domain QA systems. The code to re-
produce our analysis is available at https:
//github.com/facebookresearch/LAMA.
1 Introduction
Recently, pretrained high-capacity language mod-
els such as ELMo (Peters et al., 2018a) and BERT
(Devlin et al. , 2018a) have become increasingly
important in NLP. They are optimised to either
predict the next word in a sequence or some
masked word anywhere in a given sequence ( e.g.
“Dante was born in [M ask] in the year 1265.”).
The parameters of these models appear to store
Memory Query Answer
Symbolic
Memory Access
Neural LM
Memory Access
(Dante, born-in, X)
“Dante was born in[Mask].”
Dante
Florence
born-in
Florence
Florence
KG
LM
e.g. ELMo/BERT
Figure 1: Querying knowledge bases (KB) and lan-
guage models (LM) for factual knowledge.
vast amounts of linguistic knowledge (Peters et al.,
2018b; Goldberg, 2019; Tenney et al., 2019) use-
ful for downstream tasks. This knowledge is
usually accessed either by conditioning on latent
context representations produced by the original
model or by using the original model weights to
initialize a task-speciﬁc model which is then fur-
ther ﬁne-tuned. This type of knowledge transfer
is crucial for current state-of-the-art results on a
wide range of tasks.
In contrast, knowledge bases are e ﬀective so-
lutions for accessing annotated gold-standard re-
lational data by enabling queries such as (D ante,
born-in, X). However, in practice we often need
to extract relational data from text or other modal-
ities to populate these knowledge bases. This
requires complex NLP pipelines involving entity
extraction, coreference resolution, entity linking
and relation extraction (Surdeanu and Ji, 2014)—
components that often need supervised data and
ﬁxed schemas. Moreover, errors can easily prop-
agate and accumulate throughout the pipeline. In-
stead, we could attempt to query neural language
models for relational data by asking them to ﬁll in
masked tokens in sequences like “Dante was born
arXiv:1909.01066v2  [cs.CL]  4 Sep 2019
Figure 2: BERT world knowledge (Petroni et al., 2019)
blanks (e.g. "Cats like to chase [___]"). Petroni
et al. (2019) showed that, for some relation types,
vanilla BERT is competitive with methods rely-
ing on knowledge bases (Figure 2), and Roberts
et al. (2020) show the same for open-domain QA
using T5 model (Raffel et al., 2019). Davison et al.
(2019) suggest that it generalizes better to unseen
data. In order to retrieve BERT’s knowledge, we
need good template sentences, and there is work
on their automatic extraction and augmentation
(Bouraoui et al., 2019; Jiang et al., 2019b).
However, BERT cannot reason based on its
world knowledge. Forbes et al. (2019) show that
BERT can "guess" the affordances and properties of
many objects, but can not reason about the relation-
ship between properties and affordances. For ex-
ample, it “knows" that people can walk into houses,
and that houses are big, but it cannot infer that
houses are bigger than people. Zhou et al. (2020)
and Richardson and Sabharwal (2019) also show
that the performance drops with the number of nec-
essary inference steps. Some of BERT’s world
knowledge success comes from learning stereotypi-
cal associations (Poerner et al., 2019), e.g., a person
with an Italian-sounding name is predicted to be
Italian, even when it is incorrect.
3.4 Limitations
Multiple probing studies in section 3 and section 4
report that BERT possesses a surprising amount of
syntactic, semantic, and world knowledge. How-
ever, Tenney et al. (2019a) remarks, “the fact that
a linguistic pattern is not observed by our probing
classiﬁer does not guarantee that it is not there, and
the observation of a pattern does not tell us how it
is used." There is also the issue of how complex a
probe should be allowed to be (Liu et al., 2019a). If