Recurrent neural networks are neural networks with latent variables called hidden states. As mentioned in Section II, latent variable models are more preferable compared to n-gram models at least because the number of parameters of the latter is O(|V| n ) and it grows exponentially with n. Using RNNs as an encoder block in SMILESLM gives SMILESRNN model. Assume that X t ∈ R ×d is a minibatch of embedding features at time step t ≤ τ , H t ∈ R ×h a hidden state at t, and O t ∈ R ×|V| an output at t. The hidden state at t depends on the current inputs and the previous state, while the output at t on the current hidden state: This is how vanilla RNNs are constructed. At each time step during forward propagation, they summarize the previous context in the state variable and perform recurrent computation of the next state to determine the next output distribution. Backward propagation requires unfolding the computational graph of depth τ (backpropagation through time or BPTT ). The longer the input sequences are, the more chances to experience gradient vanishing or exploding, which results in numerical instability. Typically, it means that earlier sequence fragments in the context are more or less relevant for future observations. To address such peculiarities, various gating mechanisms such as in long short-term memory (LSTM) and gated recurrent units (GRU) were proposed. For example, GRUs attempt to capture information importance, irrelevance, and logical break by incorporating reset gates R t ∈ (0, 1) ×h for short-term dependencies and update gates U t ∈ (0, 1) ×h for long-term dependencies: