Firstly, we observe the overall model quality generated by the hyper-parameter optimization. By optimizing either on high validation accuracy or RCS, models were obtained that were both fast and accurate. However, the Pareto front shows that no model achieves a top-1 accuracy above, or near, the original Chemformer model of 53.5%. The highest validation accuracy was achieved using the optimal parameters from randomly initialized models, 46.7% (Table , RandomInit). Overall, most configurations were dominated by KD using the pretrained Zinc model (Figure ), with the highest accuracy models on the Pareto front coming from the randomly initialized and KD Chemformer models.