Given a corpus of texts written in some natural or formal language, we seek to learn the grammar and semantics of this language by analyzing the texts via probabilistic modeling. Texts are composed of basic units called tokens, each of which is derived from some vocabulary, a set of available tokens. Suppose that we have a text s. To begin our analysis, we first need to tokenize the text, i.e. establish the set of rules to divide s into tokens s 1 , s 2 , . . . , s T (e.g. partition an English text to obtain words and punctuation marks). Depending on tokenization method, our vocabulary V will comprise unique tokens from processed s. Now, we have the text sequence s = (s 1 , s 2 , . . . , s T ) composed of T ordered tokens s t at time steps t. The goal of a language model is to estimate the joint probability of the entire sequence s: