text: this strategy often requires compatible hardware. 
  
  individual self- 
  attention heads and BERT layers can be disabled 
  without signiﬁcant drop in performance (Michel 
  et al., 2019; Kovaleva et al., 2019; Baan et al., 
  2019). Pruning is a compression technique that 
  takes advantage of that fact, typically reducing the 
  amount of computation via zeroing out of certain 
  parts of the large model. In structured pruning, 
  architecture blocks are dropped, as in LayerDrop 
  (Fan et al., 2019). In unstructured, the weights in 
  the entire model are pruned irrespective of their lo- 
  cation, as in magnitude pruning (Chen et al., 2020) 
  or movement pruning (Sanh et al., 2020). 
  
  As discussed in section 6, 
  
  Prasanna et al. (2020) and Chen et al. (2020) 
  explore BERT from the perspective of the lottery 
  ticket hypothesis (Frankle and Carbin, 2019), look- 
  ing speciﬁcally at the "winning" subnetworks in 
  pre-trained BERT. They independently ﬁnd that 
  such subnetworks do exist, and that transferability 
  between subnetworks for different tasks varies. 