The Chemformer model was implemented using the PyTorch and PyTorch Lightning frameworks. We used the Transformer in the pre-norm layout -Layer Normalisation is applied before the attention and feedforward blocks -and the GELU activation function throughout. A comparison of the size of the Chemformer and Chemformer-Large models is shown in Table .