Figure 5: Pre-trained weights help BERT ﬁnd wider
optima in ﬁne-tuning on MRPC (right) than training
from scratch (left) (Hao et al., 2019)
beddings as input for training BERT, while Po-
erner et al. (2019) adapt entity vectors to BERT
representations. As mentioned above, Wang et al.
(2020c) integrate knowledge not through entity em-
beddings, but through additional pre-training ob-
jective of knowledge base completion. Sun et al.
(2019b,c) modify the standard MLM task to mask
named entities rather than random words, and Yin
et al. (2020) train with MLM objective over both
text and linearized table data. Wang et al. (2020a)
enhance RoBERTa with both linguistic and factual
knowledge with task-speciﬁc adapters.
Pre-training is the most expensive part of train-
ing BERT, and it would be informative to know
how much beneﬁt it provides. On some tasks, a
randomly initialized and ﬁne-tuned BERT obtains
competitive or higher results than the pre-trained
BERT with the task classiﬁer and frozen weights
(Kovaleva et al., 2019). The consensus in the com-
munity is that pre-training does help in most situa-
tions, but the degree and its exact contribution re-
quires further investigation. Prasanna et al. (2020)
found that most weights of pre-trained BERT are
useful in ﬁne-tuning, although there are "better"
and "worse" subnetworks. One explanation is that
pre-trained weights help the ﬁne-tuned BERT ﬁnd
wider and ﬂatter areas with smaller generalization
error, which makes the model more robust to over-
ﬁtting (see Figure 5 from Hao et al. (2019)).
Given the large number and variety of proposed
modiﬁcations, one would wish to know how much
impact each of them has. However, due to the
overall trend towards large model sizes, systematic
ablations have become expensive. Most new mod-
els claim superiority on standard benchmarks, but
gains are often marginal, and estimates of model
stability and signiﬁcance testing are very rare.
5.4 Fine-tuning BERT
Pre-training + ﬁne-tuning workﬂow is a crucial
part of BERT. The former is supposed to provide
task-independent knowledge, and the latter would
presumably teach the model to rely more on the
representations useful for the task at hand.
Kovaleva et al. (2019) did not ﬁnd that to be the
case for BERT ﬁne-tuned on GLUE tasks 5: dur-
ing ﬁne-tuning, the most changes for 3 epochs oc-
curred in the last two layers of the models, but those
changes caused self-attention to focus on [SEP]
rather than on linguistically interpretable patterns.
It is understandable why ﬁne-tuning would increase
the attention to [CLS], but not [SEP]. If Clark
et al. (2019) are correct that [SEP] serves as "no-
op" indicator, ﬁne-tuning basically tells BERT what
to ignore.
Several studies explored the possibilities of im-
proving the ﬁne-tuning of BERT:
• Taking more layers into account: learning
a complementary representation of the infor-
mation in deep and output layers (Yang and
Zhao, 2019), using a weighted combination
of all layers instead of the ﬁnal one (Su and
Cheng, 2019; Kondratyuk and Straka, 2019),
and layer dropout (Kondratyuk and Straka,
2019).
• Two-stage ﬁne-tuning introduces an inter-
mediate supervised training stage between
pre-training and ﬁne-tuning (Phang et al.,
2019; Garg et al., 2020; Arase and Tsujii,
2019; Pruksachatkun et al., 2020; Glavaš and
Vuli´c, 2020). Ben-David et al. (2020) propose
a pivot-based variant of MLM to ﬁne-tune
BERT for domain adaptation.
• Adversarial token perturbations improve
robustness of the model (Zhu et al., 2019).
• Adversarial regularization in combination
with Bregman Proximal Point Optimization
helps alleviate pre-trained knowledge forget-
ting and therefore prevents BERT from overﬁt-
ting to downstream tasks (Jiang et al., 2019a).
• Mixout regularization improves the stability
of BERT ﬁne-tuning even for a small number
of training examples (Lee et al., 2019).
With large models, even ﬁne-tuning becomes ex-
pensive, but Houlsby et al. (2019) show that it can