Finally, specific hyper-parameters that contributed to the highest validation accuracy and RCS models are identified (Table ). All models had the maximum number of six encoder layers and less than six decoder layers, with the exception of the slowest, best performing randomly initialized model. Furthermore, all chosen models, with the exception of the randomly initialized models, selected the largest possible feed forward and embedding dimension of 2048 and 512, respectively. With respect to KD, models distilling from the Zinc models selected high values for all possible KD values, whereas models distilling from Chemformer prioritized embedding and decoder feature-based KD over encoder feature-based KD and response-KD, with all preferring training knowledge distillation in combination with standard training. In conclusion, hyper-parameter optimization in general showed that on average KD based on the pretrained Zinc model are closest to the Pareto front, however, all training methods have models on the Pareto front.