Downstream results on sequence-to-sequence datasets show that our pre-trained Chemformer models outperform not only their randomly initialised (no transfer learning) counterparts, but also the current state-of-the-art models for a number of tasks. Specifically, Chemformer is able to beat the existing state-of-the-art on top-1 prediction for direct synthesis and retrosynthesis prediction, and is able to produce more desirable molecules than existing approaches on the molecular optimisation task. However, for molecular optimisation, our Chemformer models used beam search (with a beam width of 10) to generate output molecules while the reaction informatics tasks, the model pre-trained with only the masking task performs best on the molecular optimisation dataset. This is a surprising result since, for the combined task, the model is required to solve both the masking and the augmentation pre-training tasks. One possible explanation for this is that the combined task model overfits quickly on the molecular optimisation task. This is supported by the lower performance of the Chemformer-Large model on the same task. We therefore conjecture that with further hyperparameter tuning the performance of the combined task model could be improved. In particular, more work is needed to determine the optimal number of epochs required for fine-tuning.