Structured state-space sequence models (S4s) are a recent member of the fast-growing family of statespace architectures , which are gathering increasing attention in the deep learning community . S4s showed outstanding performance in audio, image, and text generation and have a 'dual nature': they (a) are trained over the entire input sequences to learn complex global properties and (b) generate one string element at a time -thereby combining some respective strengths of Transformers and LSTMs. Motivated by such 'best of two worlds' behavior, here we ask the following question: Can S4 advance the current state-of-the-art in chemical language modeling? We find evidence that it can.