In addition to our novel pre-training tasks, we also introduce a novel SMILES augmentation scheme for downstream tasks which uses a tunable augmentation probability. Given a canonical input-output pair of SMILES from the training set, (s in , s out ) ∈ D train , we randomly augment s in and s out independently with probability p aug . For sequence-to-sequence tasks we use p aug = 0.5 throughout, unless stated otherwise. Since the augmentations do not need to be pre-computed, we can augment on-the-fly, similarly to a previous study . Thus, this approach has three key advantages. Firstly, the augmentation probability can be tuned. Secondly, only the canonical data needs to be stored, rather than every augmented version of the dataset. And, thirdly, the model sees a different form of the same data every epoch, regardless of the number of epochs; we conjecture that this could improve the model's ability to generalise to unseen data.