be successfully approximated with adapter mod-
ules. They achieve competitive performance on
26 classiﬁcation tasks at a fraction of the computa-
tional cost. Adapters in BERT were also used for
multi-task learning (Stickland and Murray, 2019)
and cross-lingual transfer (Artetxe et al., 2019). An
alternative to ﬁne-tuning is extracting features from
frozen representations, but ﬁne-tuning works better
for BERT (Peters et al., 2019b).
A big methodological challenge in the current
NLP is that the reported performance improve-
ments of new models may well be within varia-
tion induced by environment factors (Crane, 2018).
BERT is not an exception. Dodge et al. (2020)
report signiﬁcant variation for BERT ﬁne-tuned
on GLUE tasks due to both weight initialization
and training data order. They also propose early
stopping on the less-promising seeds.
Although we hope that the above observations
may be useful for the practitioners, this section
does not exhaust the current research on ﬁne-tuning
and its alternatives. For example, we do not cover
such topics as Siamese architectures, policy gradi-
ent training, automated curriculum learning, and
others.
6 How big should BERT be?
6.1 Overparameterization
Transformer-based models keep growing by or-
ders of magnitude: the 110M parameters of base
BERT are now dwarfed by 17B parameters of
Turing-NLG (Microsoft, 2020), which is dwarfed
by 175B of GPT-3 (Brown et al., 2020). This trend
raises concerns about computational complexity
of self-attention (Wu et al., 2019a), environmental
issues (Strubell et al., 2019; Schwartz et al., 2019),
fair comparison of architectures (Aßenmacher and
Heumann, 2020), and reproducibility.
Human language is incredibly complex, and
would perhaps take many more parameters to de-
scribe fully, but the current models do not make
good use of the parameters they already have. V oita
et al. (2019b) showed that all but a few Trans-
former heads could be pruned without signif-
icant losses in performance . For BERT, Clark
et al. (2019) observe that most heads in the same
layer show similar self-attention patterns (perhaps
related to the fact that the output of all self-attention
heads in a layer is passed through the same MLP),
which explains why Michel et al. (2019) were able
to reduce most layers to a single head.
Depending on the task, some BERT heads/layers
are not only redundant (Kao et al., 2020), but also
harmful to the downstream task performance. Pos-
itive effect from head disabling was reported for
machine translation (Michel et al., 2019), abstrac-
tive summarization (Baan et al., 2019), and GLUE
tasks (Kovaleva et al., 2019). Additionally, Ten-
ney et al. (2019a) examine the cumulative gains of
their structural probing classiﬁer, observing that in
5 out of 8 probing tasks some layers cause a drop
in scores (typically in the ﬁnal layers). Gordon
et al. (2020) ﬁnd that 30–40% of the weights can
be pruned without impact on downstream tasks.
In general, larger BERT models perform better
(Liu et al., 2019a; Roberts et al., 2020), but not
always: BERT-base outperformed BERT-large on
subject-verb agreement (Goldberg, 2019) and sen-
tence subject detection (Lin et al., 2019). Given
the complexity of language, and amounts of pre-
training data, it is not clear why BERT ends up with
redundant heads and layers. Clark et al. (2019) sug-
gest that one possible reason is the use of attention
dropouts, which causes some attention weights to
be zeroed-out during training.
6.2 Compression techniques
Given the above evidence of overparameteriza-
tion, it does not come as a surprise that BERT
can be efﬁciently compressed with minimal ac-
curacy loss, which would be highly desirable for
real-world applications. Such efforts to date are
summarized in Table 1. The main approaches are
knowledge distillation, quantization, and pruning.
The studies in the knowledge distillation
framework (Hinton et al., 2014) use a smaller
student-network trained to mimic the behavior of