So, having observed previous t -1 tokens, we would like to predict the next token s t , 1 ≤ t ≤ T , via conditional probability, from left to right. How should we calculate the probabilities? First, note that lengths of different sequences may vary, and we can rationally assume that it is sufficient to observe only tokens s t-τ , . . . , s t-1 in some time span τ < t a to predict s t . This way, probabilities can be estimated by counting all occurrences of tokens divided by the vocabulary dimension and additionally using some smoothing techniques. This is how classic n-gram language modeling works: P (s t |s 1 , . . . , s t-1 ) = P (s t |s t-τ , . . . , s t-1 ), P (s t |s 1 , . . . , s t-1 ) = #(s 1 , . . . , s t-1 , s t ) + #(s 1 , . . . , s t-1 ) + |V| .