text: 6.1 Overparameterization 
  
  Transformer-based models keep growing by or- 
  ders of magnitude: the 110M parameters of base 
  BERT are now dwarfed by 17B parameters of 
  Turing-NLG (Microsoft, 2020), which is dwarfed 
  by 175B of GPT-3 (Brown et al., 2020). This trend 
  raises concerns about computational complexity 
  of self-attention (Wu et al., 2019a), environmental 
  issues (Strubell et al., 2019; Schwartz et al., 2019), 
  fair comparison of architectures (Aßenmacher and 
  Heumann, 2020), and reproducibility. 
  
  Human language is incredibly complex, and 
  would perhaps take many more parameters to de- 
  scribe fully, but the current models do not make 
  good use of the parameters they already have. Voita 
  et al. (2019b) showed that all but a few Trans- 
  former heads could be pruned without signif- 
  icant losses in performance. For BERT, Clark 
  et al. (2019) observe that most heads in the same 
  layer show similar self-attention patterns (perhaps 
  related to the fact that the output of all self-attention 
  heads in a layer is passed through the same MLP), 
  which explains why Michel et al. (2019) were able 
  to reduce most layers to a single head. 