knowledge distillation, quantization, and pruning.
The studies in the knowledge distillation
framework (Hinton et al., 2014) use a smaller
student-network trained to mimic the behavior of
a larger teacher-network. For BERT, this has been
achieved through experiments with loss functions
(Sanh et al., 2019b; Jiao et al., 2019), mimicking
the activation patterns of individual portions of the
teacher network (Sun et al., 2019a), and knowledge
transfer at the pre-training (Turc et al., 2019; Jiao
et al., 2019; Sun et al., 2020) or ﬁne-tuning stage
(Jiao et al., 2019). McCarley et al. (2020) suggest
that distillation has so far worked better for GLUE
than for reading comprehension, and report good
results for QA from a combination of structured
pruning and task-speciﬁc distillation.
Quantization decreases BERT’s memory foot-
print through lowering the precision of its weights
(Shen et al., 2019; Zafrir et al., 2019). Note that