text: 3 ENVIRONMENTAL AND FINANCIAL COST 
  Strubell et al. recently benchmarked model training and develop- 
  ment costs in terms of dollars and estimated 𝐶𝑂2 emissions [129]. 
  While the average human is responsible for an estimated 5t 𝐶𝑂2𝑒 
  per year,2 the authors trained a Transformer (big) model [136] with 
  neural architecture search and estimated that the training procedure 
  emitted 284t of 𝐶𝑂2. Training a single BERT base model (without 
  hyperparameter tuning) on GPUs was estimated to require as much 
  energy as a trans-American flight. 
  
  While some of this energy comes from renewable sources, or 
  cloud compute companies’ use of carbon credit-offset sources, the 
  authors note that the majority of cloud compute providers’ energy is 
  not sourced from renewable sources and many energy sources in the 
  world are not carbon neutral. In addition, renewable energy sources 
  are still costly to the environment,3 and data centers with increasing 
  computation requirements take away from other potential uses of 