The model architecture used is similar to the one used in and is illustrated in Figure are fed token by token to an embedding layer of 𝑚 dimensions, followed by 𝑙 layers of LSTM / GRU cell size 𝑤 . To prevent squeezing the encoded input, the embedding dimensions should be 𝑚 ≤ 𝑤. Between the inner RNN layers there can be dropout layers with a probability 𝑑. The output from the cells is squeezed to the vocabulary size 𝑣 by a linear transformation layer and a softmax is performed to obtain the probabilities of sampling each token in the next position. This is repeated for each token in the entire sequence.