text: Given the above evidence of overparameteriza- 
  tion, it does not come as a surprise that BERT 
  can be efﬁciently compressed with minimal ac- 
  curacy loss, which would be highly desirable for 
  real-world applications. Such efforts to date are 
  summarized in Table 1. The main approaches are 
  knowledge distillation, quantization, and pruning. 
  The studies in the knowledge distillation 
  framework (Hinton et al., 2014) use a smaller 
  student-network trained to mimic the behavior of 
  a larger teacher-network. For BERT, this has been 
  achieved through experiments with loss functions 
  (Sanh et al., 2019b; Jiao et al., 2019), mimicking 
  the activation patterns of individual portions of the 
  teacher network (Sun et al., 2019a), and knowledge 
  transfer at the pre-training (Turc et al., 2019; Jiao 
  et al., 2019; Sun et al., 2020) or ﬁne-tuning stage 
  (Jiao et al., 2019). McCarley et al. (2020) suggest 
  that distillation has so far worked better for GLUE 
  than for reading comprehension, and report good 
  results for QA from a combination of structured 
  pruning and task-speciﬁc distillation. 