Schwaller et al. replaced the LSTM with the popular Transformer architecture for the task of forward synthesis prediction. Concurrently, Karpov et al. introduced the Transformer for retrosynthesis which was seen to improve drastically over the LSTM for the top-k accuracy evaluation metric. Karpov et al. employed popular training strategies (e.g. cyclic learning rate scheduling), to optimise the large parameter space of the Transformer. Other common strategies for model training consist of dataset augmentation and model pre-training. As a molecule has several valid (non-canonical) SMILES, dataset augmentation is achieved by adding the randomised SMILES to the existing dataset . The augmentation can be complemented by pre-training the model on a masking task . Irwin et al. investigated the combined effects of pre-training and augmentation for retrosynthesis in their Chemformer framework. They showed that both methods improve the current evaluation metric (top-k accuracy) over the vanilla Transformer. Nevertheless, it is unclear if this translates to a lower degree of invalid SMILES and diverse predictions.