report that an intermediate ﬁne-tuning step with
supervised parsing does not make much difference
for downstream task performance.
3.2 Semantic knowledge
To date, more studies have been devoted to BERT’s
knowledge of syntactic rather than semantic phe-
nomena. However, we do have evidence from an
MLM probing study that BERT has some knowl-
edge of semantic roles (Ettinger, 2019). BERT
even displays some preference for the incorrect
ﬁllers for semantic roles that are semantically re-
lated to the correct ones, as opposed to those that
are unrelated (e.g. "to tip a chef" is better than "to
tip a robin", but worse than "to tip a waiter").
Tenney et al. (2019b) showed that BERT en-
codes information about entity types, relations,
semantic roles, and proto-roles, since this infor-
mation can be detected with probing classiﬁers.
BERT struggles with representations of num-
bers. Addition and number decoding tasks showed
that BERT does not form good representations for
ﬂoating point numbers and fails to generalize away
from the training data (Wallace et al., 2019b). A
part of the problem is BERT’s wordpiece tokeniza-
tion, since numbers of similar values can be divided
up into substantially different word chunks.
Out-of-the-box BERT is surprisingly brittle to
named entity replacements: e.g. replacing names
in the coreference task changes 85% of predictions
(Balasubramanian et al., 2020). This suggests that
the model does not actually form a generic idea of
named entities, although its F1 scores on NER prob-
ing tasks are high (Tenney et al., 2019a). Broscheit
(2019) ﬁnd that ﬁne-tuning BERT on Wikipedia
entity linking "teaches" it additional entity knowl-
edge, which would suggest that it did not absorb all
the relevant entity information during pre-training
on Wikipedia.
3.3 World knowledge
The bulk of evidence about commonsense knowl-
edge captured in BERT comes from practitioners
using it to extract such knowledge. One direct prob-
ing study of BERT reports that BERT struggles
with pragmatic inference and role-based event
knowledge (Ettinger, 2019). BERT also struggles
with abstract attributes of objects, as well as visual
and perceptual properties that are likely to be as-
sumed rather than mentioned (Da and Kasai, 2019).
The MLM component of BERT is easy to
adapt for knowledge induction by ﬁlling in the
Language Models as Knowledge Bases?
Fabio Petroni1 Tim Rockt¨aschel1,2 Patrick Lewis1,2 Anton Bakhtin1
Yuxiang Wu1,2 Alexander H. Miller1 Sebastian Riedel1,2
1Facebook AI Research
2University College London
{fabiopetroni, rockt, plewis, yolo, yuxiangwu, ahm, sriedel}@fb.com
Abstract
Recent progress in pretraining language mod-
els on large textual corpora led to a surge
of improvements for downstream NLP tasks.
Whilst learning linguistic knowledge, these
models may also be storing relational knowl-
edge present in the training data, and may
be able to answer queries structured as “ﬁll-
in-the-blank” cloze statements. Language
models have many advantages over structured
knowledge bases: they require no schema en-
gineering, allow practitioners to query about
an open class of relations, are easy to extend to
more data, and require no human supervision
to train. We present an in-depth analysis of the
relational knowledge already present (without
ﬁne-tuning) in a wide range of state-of-the-
art pretrained language models. We ﬁnd that
(i) without ﬁne-tuning, BERT contains rela-
tional knowledge competitive with traditional
NLP methods that have some access to ora-
cle knowledge, (ii) BERT also does remark-
ably well on open-domain question answer-
ing against a supervised baseline, and (iii) cer-
tain types of factual knowledge are learned
much more readily than others by standard lan-
guage model pretraining approaches. The sur-
prisingly strong ability of these models to re-
call factual knowledge without any ﬁne-tuning
demonstrates their potential as unsupervised
open-domain QA systems. The code to re-