Model training was performed using PyTorch (version 2.2.2) and Lightning (version 2.2.1) on a single GPU. Most of the original training parameters from the Chemformer paper were used to allow for direct comparisons (Table ). Using Optuna , hyper-parameter optimization was carried out using the Bayesian Tree-structured Parzen Estimator (TPE) . Inspired by earlier work on hyper-parameter optimization for multi-step retrosynthesis , we set the objectives for the optimization to the validation greedy search accuracy, and a adapted version of the rate-correct score (RCS) . It was obtained by dividing the correct samples by the cumulative time ( Correct Samples Total Time ). The choices of the model parameters are given in Table . Distillation was performed using implementations from Textbrewer (version 0.2.1 -post1) , which was adapted to Chemformer. Cross entropy losses were computed on the output values (response-based KD) or internal representations using direct comparison (feature-based representations), with or without standard cross entropy loss, also named hard-label loss. All losses were combined using a weighted average based on weight hyper-parameters, normally set to 1.0, which were subsequently normalized to sum to one. Feature-based representations were compared between the output of the embedding, encoder or decoder modules.