text: Transformer models, on the other hand, have been able to con- 
  tinuously benefit from larger architectures and larger quantities of 
  data. Devlin et al. [39] in particular noted that training on a large 
  dataset and fine-tuning for specific tasks leads to strictly increasing 
  results on the GLUE tasks [138] for English as the hyperparameters 
  of the model were increased. Initially developed as Chinese LMs, the 
  ERNIE family [130, 131, 145] produced ERNIE-Gen, which was also 
  trained on the original (English) BERT dataset, joining the ranks 
  of very large LMs. NVIDIA released the MegatronLM which has 
  8.3B parameters and was trained on 174GB of text from the English 
  Wikipedia, OpenWebText, RealNews and CC-Stories datasets [122]. 
  Trained on the same dataset, Microsoft released T-NLG,1 an LM 
  with 17B parameters. OpenAI’s GPT-3 [25] and Google’s GShard 
  [73] and Switch-C [43] have increased the definition of large LM by 
  orders of magnitude in terms of parameters at 175B, 600B, and 1.6T 
  parameters, respectively. Table 1 summarizes a selection of these 
  LMs in terms of training data size and parameters. As increasingly 
  large amounts of text are collected from the web in datasets such 
  as the Colossal Clean Crawled Corpus [107] and the Pile [51], this 
  trend of increasingly large LMs can be expected to continue as long 
  as they correlate with an increase in performance. 