A Primer in BERTology: What We Know About How BERT Works
Anna Rogers
Center for Social Data Science
University of Copenhagen
arogers@sodas.ku.dk
Olga Kovaleva
Dept. of Computer Science
University of Massachusetts Lowell
okovalev@cs.uml.edu
Anna Rumshisky
Dept. of Computer Science
University of Massachusetts Lowell
arum@cs.uml.edu
Abstract
Transformer-based models have pushed state
of the art in many areas of NLP, but our un-
derstanding of what is behind their success
is still limited. This paper is the ﬁrst sur-
vey of over 150 studies of the popular BERT
model. We review the current state of knowl-
edge about how BERT works, what kind
of information it learns and how it is repre-
sented, common modiﬁcations to its training
objectives and architecture, the overparame-
terization issue and approaches to compres-
sion. We then outline directions for future
research.
1 Introduction
Since their introduction in 2017, Transformers
(Vaswani et al., 2017) have taken NLP by storm,
offering enhanced parallelization and better model-
ing of long-range dependencies. The best known
Transformer-based model is BERT (Devlin et al.,
2019); it obtained state-of-the-art results in numer-
ous benchmarks and is still a must-have baseline.
While it is clear that BERT works remarkably
well, it is less clear why, which limits further
hypothesis-driven improvement of the architecture.
Unlike CNNs, the Transformers have little cogni-
tive motivation, and the size of these models limits
our ability to experiment with pre-training and per-
form ablation studies. This explains a large number
of studies over the past year that attempted to un-
derstand the reasons behind BERT’s performance.
In this paper, we provide an overview of what
has been learned to date, highlighting the questions
which are still unresolved. We ﬁrst consider the
linguistic aspects of it, i.e., the current evidence
regarding the types of linguistic and world knowl-
edge learned by BERT, as well as where and how
this knowledge may be stored in the model. We
then turn to the technical aspects of the model and
provide an overview of the current proposals to
improve BERT’s architecture, pre-training and ﬁne-
tuning. We conclude by discussing the issue of
overparameterization, the approaches to compress-
ing BERT, and the nascent area of pruning as a
model analysis technique.
2 Overview of BERT architecture
Fundamentally, BERT is a stack of Transformer
encoder layers (Vaswani et al., 2017) which consist
of multiple self-attention "heads". For every input
token in a sequence, each head computes key, value
and query vectors, used to create a weighted repre-
sentation. The outputs of all heads in the same layer
are combined and run through a fully-connected
layer. Each layer is wrapped with a skip connection
and followed by layer normalization.
The conventional workﬂow for BERT consists
of two stages: pre-training and ﬁne-tuning. Pre-
training uses two self-supervised tasks: masked
language modeling (MLM, prediction of randomly
masked input tokens) and next sentence prediction
(NSP, predicting if two input sentences are adjacent
to each other). In ﬁne-tuning for downstream ap-
plications, one or more fully-connected layers are
typically added on top of the ﬁnal encoder layer.
The input representations are computed as fol-
lows: each word in the input is ﬁrst tokenized into
wordpieces (Wu et al., 2016), and then three em-
bedding layers (token, position, and segment) are
combined to obtain a ﬁxed-length vector. Special
token [CLS] is used for classiﬁcation predictions,
and [SEP] separates input segments.
Google1 and HuggingFace (Wolf et al., 2020)
provide many variants of BERT, including the orig-
inal "base" and "large" versions. They vary in the
number of heads, layers, and hidden state size.
1https://github.com/
google-research/bert
arXiv:2002.12327v3  [cs.CL]  9 Nov 2020