Compression Performance Speedup Model Evaluation
BERT-base (Devlin et al., 2019) ×1 100% ×1 BERT 12 All GLUE tasks, SQuAD
BERT-small ×3.8 91% - BERT 4† All GLUE tasks
Distillation
DistilBERT (Sanh et al., 2019a) ×1.5 90% § ×1.6 BERT 6 All GLUE tasks, SQuAD
BERT6-PKD (Sun et al., 2019a) ×1.6 98% ×1.9 BERT 6 No WNLI, CoLA, STS-B; RACE
BERT3-PKD (Sun et al., 2019a) ×2.4 92% ×3.7 BERT 3 No WNLI, CoLA, STS-B; RACE
Aguilar et al. (2019), Exp. 3 ×1.6 93% - BERT 6 CoLA, MRPC, QQP, RTE
BERT-48 (Zhao et al., 2019) ×62 87% ×77 BERT 12∗† MNLI, MRPC, SST-2
BERT-192 (Zhao et al., 2019) ×5.7 93% ×22 BERT 12∗† MNLI, MRPC, SST-2
TinyBERT (Jiao et al., 2019) ×7.5 96% ×9.4 BERT 4† No WNLI; SQuAD
MobileBERT (Sun et al., 2020) ×4.3 100% ×4 BERT 24† No WNLI; SQuAD
PD (Turc et al., 2019) ×1.6 98% ×2.5‡ BERT6† No WNLI, CoLA and STS-B
WaLDORf (Tian et al., 2019) ×4.4 93% ×9 BERT 8†∥ SQuAD
MiniLM (Wang et al., 2020b) ×1.65 99% ×2 BERT 6 No WNLI, STS-B, MNLImm; SQuAD
MiniBERT(Tsai et al., 2019) ×6∗∗ 98% ×27∗∗ mBERT3† CoNLL-18 POS and morphology
BiLSTM-soft (Tang et al., 2019) ×110 91% ×434‡ BiLSTM1 MNLI, QQP, SST-2
Quanti-
zation
Q-BERT-MP (Shen et al., 2019) ×13 98% ¶ - BERT 12 MNLI, SST-2, CoNLL-03, SQuAD
BERT-QAT (Zafrir et al., 2019) ×4 99% - BERT 12 No WNLI, MNLI; SQuAD
GOBO(Zadeh and Moshovos, 2020) ×9.8 99% - BERT 12 MNLI
Pruning
McCarley et al. (2020), ff2 ×2.2‡ 98%‡ ×1.9‡ BERT24 SQuAD, Natural Questions
RPP (Guo et al., 2019) ×1.7‡ 99%‡ - BERT 24 No WNLI, STS-B; SQuAD
Soft MvP (Sanh et al., 2020) ×33 94% ¶ - BERT 12 MNLI, QQP, SQuAD
IMP (Chen et al., 2020), rewind 50% ×1.4–2.5 94–100% - BERT 12 No MNLI-mm; SQuAD
Other
ALBERT-base (Lan et al., 2020b) ×9 97% - BERT 12† MNLI, SST-2
ALBERT-xxlarge (Lan et al., 2020b) ×0.47 107% - BERT 12† MNLI, SST-2
BERT-of-Theseus (Xu et al., 2020) ×1.6 98% ×1.9 BERT 6 No WNLI
PoWER-BERT (Goyal et al., 2020) N/A 99% ×2–4.5 BERT 12 No WNLI; RACE
Table 1: Comparison of BERT compression studies. Compression, performance retention, inference time speedup
ﬁgures are given with respect to BERT base, unless indicated otherwise. Performance retention is measured as
a ratio of average scores achieved by a given model and by BERT base. The subscript in the model description
reﬂects the number of layers used. ∗Smaller vocabulary used. †The dimensionality of the hidden layers is reduced.
∥Convolutional layers used. ‡Compared to BERTlarge. ∗∗Compared to mBERT.§As reported in (Jiao et al., 2019).¶In
comparison to the dev set.
this strategy often requires compatible hardware.
As discussed in section 6, individual self-
attention heads and BERT layers can be disabled
without signiﬁcant drop in performance (Michel
et al., 2019; Kovaleva et al., 2019; Baan et al.,
2019). Pruning is a compression technique that
takes advantage of that fact, typically reducing the
amount of computation via zeroing out of certain
parts of the large model. In structured pruning,
architecture blocks are dropped, as in LayerDrop
(Fan et al., 2019). In unstructured, the weights in
the entire model are pruned irrespective of their lo-
cation, as in magnitude pruning (Chen et al., 2020)
or movement pruning (Sanh et al., 2020).
Prasanna et al. (2020) and Chen et al. (2020)
explore BERT from the perspective of the lottery
ticket hypothesis (Frankle and Carbin, 2019), look-
ing speciﬁcally at the "winning" subnetworks in
pre-trained BERT. They independently ﬁnd that
such subnetworks do exist, and that transferability
between subnetworks for different tasks varies.
If the ultimate goal of training BERT is compres-
sion, Li et al. (2020) recommend training larger
models and compressing them heavily rather than
compressing smaller models lightly.
Other techniques include decomposing BERT’s
embedding matrix into smaller matrices (Lan et al.,
2020a), progressive module replacing (Xu et al.,
2020) and dynamic elimination of intermediate en-
coder outputs (Goyal et al., 2020). See Ganesh et al.
(2020) for a more detailed discussion of compres-
sion methods.