Each model was pre-trained for 1,000,000 steps using 4 NVIDIA V100 GPUs with a batch size of 128 molecules per GPU. The original Transformer learning rate schedule was used, along with 8000 linear warm-up steps. Pre-training took approximately 2.5 days for Chemformer and 6 days for Chemformer-Large. The one-cycle learning rate schedule was used for fine-tuning, for both sequence-to-sequence and discriminative tasks. Additionally, we used the Adam optimiser with parameters β 1 = 0.9 and β 2 = 0.999 for both pre-training and fine-tuning on all tasks.