Hyper-parameter optimization During hyper-parameter optimization, we found that the Pareto front contained mostly smaller decoders and larger encoders. This is expected behaviour as the inference calculations were based on auto-regressive greedy search, which iteratively calls the decoder to make its predictions. Using Optuna, we trained 150 models, including three training settings from random initialization, KD using Chemformer and KD using the pretrained Zinc model. However, the Optuna search allowed for duplicate parameter suggestions, resulting in on average seven to eight duplicate model settings. This is likely because some models had especially bad validation averages (around the 0% accuracy margin). Additionally, no model reached a high accuracy, none higher than 46.7%. This is likely due to the theoretical maximum of the model parameters, where we the original implementation parameters were set as the maximum. Finally, the Pareto front mostly included pretrained KD models.