While the three pre-trained Chemformer models perform comparably on both sets of discriminative tasks, the randomly initialised model performs significantly worse. A possible explanation for this is that, without transfer learning, Chemformer overfits quickly on small datasets -the number of molecules per biological activity prediction task varies from 1241 to 5830, and the number of molecules per molecular property is no more than 4200. This explanation is supported by the observation that, for biological activity prediction, the optimal architecture found for the randomly initialised model used only 6 million learnable parameters, in comparison to almost 20 million for the pre-trained models. The randomly initialised model for property prediction uses even fewer parameters. Larger randomly initialised models were found to perform worse. These results suggests that, in small-data regimes, pre-training is crucial for strong performance with Transformer models on discriminative tasks.