 Bidirectional representations condition on both pre- and post- context (e.g., words) in all layers.In a bigram ("n" = 2) language model, the probability of the sentence "I saw the red house" is approximated as

whereas in a trigram ("n" = 3) language model, the approximation is

Note that the context of the first "n" – 1 "n"-grams is filled with start-of-sentence markers, typically denoted .Additionally, without an end-of-sentence marker, the probability of an ungrammatical sequence "*I saw the" would always be higher than that of the longer sentence "I saw the red house."