text: 2 BACKGROUND 
  Similar to [14], we understand the term language model (LM) to 
  refer to systems which are trained on string prediction tasks: that is, 
  predicting the likelihood of a token (character, word or string) given 
  either its preceding context or (in bidirectional and masked LMs) 
  its surrounding context. Such systems are unsupervised and when 
  deployed, take a text as input, commonly outputting scores or string 
  predictions. Initially proposed by Shannon in 1949 [117], some of 
  the earliest implemented LMs date to the early 1980s and were used 
  as components in systems for automatic speech recognition (ASR), 
  machine translation (MT), document classification, and more [111]. 
  In this section, we provide a brief overview of the general trend of 
  language modeling in recent years. For a more in-depth survey of 
  pretrained LMs, see [105]. 
  
  Before neural models, n-gram models also used large amounts 
  of data [20, 87]. In addition to ASR, these large n-gram models of 
  English were developed in the context of machine translation from 
  another source language with far fewer direct translation examples. 
  For example, [20] developed an n-gram model for English with 
  a total of 1.8T n-grams and noted steady improvements in BLEU 
  score on the test set of 1797 Arabic translations as the training data 
  was increased from 13M tokens. 