Previous works have also attempted to use Transformers for molecular property prediction. Many of these works publish stronger results than Chemformer on the three MoleculeNet tasks we investigated. For example, MolBERT publishes RMSEs of 0.531, 0.948 and 0.561 on ESOL, Free Solvation and Lipophilicity, respectively. However, we do not produce a direct comparison of these models with Chemformer for three reasons. Firstly, since we are constrained by the use of simultaneous fine-tuning, our dataset splits are different. Secondly, we produce a single model for all tasks, while most existing baselines fine-tune on each task separately. And, finally, many existing models contain significantly more learnable parameters than Chemformer; the Mol-BERT model contains approximately 85 million parameters. Further improvements can be made to the Chemformer model by scaling up the size of the models and the size of the pre-training dataset. This is something we intend to investigate in future work.