From the fine-tuning results we presented three key conclusions can be drawn. Firstly, the Chemformer model can be applied to a wide variety of downstream tasks, including both sequenceto-sequence and discriminative tasks, fairly easily. Secondly, self-supervised pre-training can improve convergence of the Chemformer model on downstream Cheminformatics tasks, and can therefore significantly improve results on these tasks when training time is limited. Finally, a combination of transfer learning and our novel augmentation strategy is able to produce state-ofthe-art top-1 results on all downstream sequence-to-sequence tasks we examined.