text: Stochastic Parrots 
  
  mBERT across 29 tasks. Either way, these models do not address 
  the inclusion problems raised by [65], who note that over 90% of 
  the world’s languages used by more than a billion people currently 
  have little to no support in terms of language technology. 
  
  Alongside work investigating what information the models re- 
  tain from the data, we see a trend in reducing the size of these 
  models using various techniques such as knowledge distillation 
  [26, 58], quantization [118, 153], factorized embedding parame- 
  terization and cross-layer parameter sharing [70], and progressive 
  module replacing [146]. Rogers et al. [110] provide a comprehensive 
  comparison of models derived from BERT using these techniques, 
  such as DistilBERT [113] and ALBERT [70]. While these models 
  maintain and sometimes exceed the performance of the original 
  BERT model, despite their much smaller size, they ultimately still 
  rely on large quantities of data and significant processing and stor- 
  age capabilities to both hold and reduce the model. 