text: models and compressing them heavily rather than 
  compressing smaller models lightly. 
  
  Other techniques include decomposing BERT’s 
  embedding matrix into smaller matrices (Lan et al., 
  2020a), progressive module replacing (Xu et al., 
  2020) and dynamic elimination of intermediate en- 
  coder outputs (Goyal et al., 2020). See Ganesh et al. 
  (2020) for a more detailed discussion of compres- 
  sion methods. 
  
  6.3 Pruning and model analysis 
  
  There is a nascent discussion around pruning as a 
  model analysis technique. The basic idea is that 
  a compressed model a priori consists of elements 
  that are useful for prediction; therefore by ﬁnding 
  out what they do we may ﬁnd out what the whole 
  network does. For instance, BERT has heads that 
  seem to encode frame-semantic relations, but dis- 
  abling them might not hurt downstream task per- 
  formance Kovaleva et al. (2019); this suggests that 
  this knowledge is not actually used. 
  
  If the ultimate goal of training BERT is compres- 
  sion, Li et al. (2020) recommend training larger  For the base Transformer, Voita et al. (2019b) 
  identify the functions of self-attention heads and 