Knowledge distillation (KD) exploits previously trained teacher models to train smaller and more efficient student models. The information from a teacher model can be transferred to a student model in three ways: 1) response-based KD , 2) feature-based KD and 3) relation-based KD . These approaches distill knowledge from the output, internal representations or the angles/distances in-between batches of internal representations, respectively (Figure ). Additionally, more efficient Transformer architectures have been pursued to increase inference speed of encoder-decoder architectures. Efficient alternative Transformer architectures is a large area of research and includes models with induced sparsity for faster compute times , and memory efficient transformers with lower complexity . Previous work specific to retrosynthesis also included increasing efficiency of the sampling algorithms using speculative decoding .