We experimented with three different model architectures in Chemformer, so-called model variants, including the original Chemformer architecture , the Perceiver architecture and the Switch Transoformer . The original Chemformer architecture is a BART (Bidirectional and Auto-Regressive Transformer) model with encoder-decoder architecture , using sinusoidal positional embedding, pre-norm layer-normalization and GELU activation functions. A Perceiver variation was used , keeping the original Chemformer architecture with the exception of an initial cross-attention layer in the Transformer encoder. This cross attention is combined with a learnable projection tensor, which was used for the query and the embedded source sequence was used as the key-value pair, projecting the internal representation to a smaller dimension. Finally, to experiment with induced sparsity, we used the Switch Transformer layer implementation of Varuna Jayasiri . In the Switch transformer, the traditional feed forward was replaced with eight experts with feed forward layers of one-eight the size of the original model.