Although multi-layer LSTM and GRUs demonstrate appropriate and promising performance, one can also build an encoder block of SMILESLM using convolutional layers and attention mechanisms. Depending on data, task, and network architecture, one might benefit from parallelization and interpretability of such encoders. The following two subsections describe CNN and Transformer encoders, which have found significant success in NLP tasks. However, based on our experience, they are inferior in terms of validity and capturing of long-term SMILES fragments. As both models need only constant minimum number of sequential operations and have shorter maximum path lengths, this seems counterintuitive; still, we hypothesize that the main reason is the absence of an explicit state/memory mechanism. We suggest experimenting with more refined models like in or and comparing their performances with our baseline or the results from .