model with a maximum-likelihood model for each document to smooth the model. n-gram In an n-gram model, the probability of observing the sentence is approximated as It is assumed that the probability of observing the ith word wi in the context history of the preceding i − 1 words can be approximated by the probability of observing it in the shortened context history of the preceding n − 1 words (nth-order Markov property). To clarify, for n = 3 and i = 2 we have . The conditional probability can be calculated from n-gram model frequency counts: The terms bigram and trigram language models denote n-gram models with n = 2 and n = 3, respectively. Typically, the n-gram model probabilities are not derived directly from frequency counts, because models derived this way have severe problems when confronted with any n-grams that have not been explicitly seen before. Instead, some form of smoothing is necessary, assigning some of the total probability mass to unseen words or n-grams. Various methods are used, from simple "add-one" smoothing (assign a count of 1 to unseen n-grams, as an uninformative prior) to more sophisticated models, such as Good–Turing discounting or back-off models. Bidirectional Bidirectional representations condition on both pre- and post- context (e.g., words) in all layers. Example In a bigram (n = 2) language model, the probability of the sentence I saw the red house is approximated as whereas in a trigram (n = 3) language model, the approximation is Note that the context of the first n – 1 n-grams is filled with start-of-sentence markers, typically denoted <s>. Additionally, without an end-of-sentence marker, the probability of an ungrammatical sequence *I saw the would always be higher than that of the longer sentence I saw the red house. Exponential Maximum entropy language models encode the relationship between a word and the n-gram history using feature functions. The equation is where is the partition function, is the parameter vector, and is the feature function. In the simplest case, the feature function is just an indicator of the presence of a certain n-gram. It is helpful to use a prior on or some form of regularization. The log-bilinear model is another example of an exponential language model. Neural network Neural language models (or continuous space language models) use continuous representations or embeddings of words to make their predictions. These models make use of neural networks.