layers are more transferable (Liu et al., 2019a). In
ﬁne-tuning, it explains why the ﬁnal layers change
the most (Kovaleva et al., 2019), and why restoring
the weights of lower layers of ﬁne-tuned BERT to
their original values does not dramatically hurt the
model performance (Hao et al., 2019).
Tenney et al. (2019a) suggest that while syntactic
information appears early in the model and can be
localized, semantics is spread across the entire
model, which explains why certain non-trivial ex-
amples get solved incorrectly at ﬁrst but correctly
at the later layers. This is rather to be expected:
semantics permeates all language, and linguists de-
bate whether meaningless structures can exist at
all (Goldberg, 2006, p.166-182). But this raises
the question of what stacking more Transformer
layers in BERT actually achieves in terms of the
spread of semantic knowledge, and whether that
is beneﬁcial. Tenney et al. compared BERT-base
and BERT-large, and found that the overall pattern
of cumulative score gains is the same, only more
spread out in the larger model.
Note that Tenney et al. (2019a)’s experiments
concern sentence-level semantic relations; Cui et al.
(2020) report that the encoding of ConceptNet se-
mantic relations is the worst in the early layers and
increases towards the top. Jawahar et al. (2019)
place "surface features in lower layers, syntactic
features in middle layers and semantic features in
higher layers", but their conclusion is surprising,
given that only one semantic task in this study actu-
ally topped at the last layer, and three others peaked
around the middle and then considerably degraded
by the ﬁnal layers.
5 Training BERT
This section reviews the proposals to optimize the
training and architecture of the original BERT.
5.1 Model architecture choices
To date, the most systematic study of BERT archi-
tecture was performed by Wang et al. (2019b), who
experimented with the number of layers, heads, and
model parameters, varying one option and freez-
ing the others. They concluded that the number
of heads was not as signiﬁcant as the number
of layers . That is consistent with the ﬁndings
of V oita et al. (2019b) and Michel et al. (2019)
(section 6), and also the observation by Liu et al.
(2019a) that the middle layers were the most trans-
ferable. Larger hidden representation size was con-
sistently better, but the gains varied by setting.
All in all, changes in the number of heads
and layers appear to perform different func-
tions. The issue of model depth must be related to
the information ﬂow from the most task-speciﬁc
layers closer to the classiﬁer (Liu et al., 2019a),
to the initial layers which appear to be the most
task-invariant (Hao et al., 2019), and where the
tokens resemble the input tokens the most (Brun-
ner et al., 2020) (see subsection 4.3). If that is the
case, a deeper model has more capacity to encode
information that is not task-speciﬁc.
On the other head, many self-attention heads
in vanilla BERT seem to naturally learn the same
patterns (Kovaleva et al., 2019). This explains
why pruning them does not have too much impact.
The question that arises from this is how far we
could get with intentionally encouraging diverse
self-attention patterns: theoretically, this would
mean increasing the amount of information in the
model with the same number of weights. Raganato
et al. (2020) show for Transformer-based machine
translation we can simply pre-set the patterns that
we already know the model would learn, instead of
learning them from scratch.
Vanilla BERT is symmetric and balanced in
terms of self-attention and feed-forward layers, but
it may not have to be. For the base Transformer,
Press et al. (2020) report beneﬁts from more self-
attention sublayers at the bottom and more feedfor-
ward sublayers at the top.
5.2 Improvements to the training regime
Liu et al. (2019b) demonstrate the beneﬁts of
large-batch training: with 8k examples both the
language model perplexity and downstream task