In order to investigate the importance of the number of learnable model parameters, we pretrained both a base model, Chemformer, and a larger model, Chemformer-Large. The Chemformer model uses the same hyperparameters as the original Transformer and contains approximately 45 million learnable weights, whereas the Chemformer-Large model expands this to 230 million weights. Full details of the models can be found in Section 2.4.