Overall in the fully trained models, KD generally leads to lower accuracy compared to standard training methods, which aligns with previous research , possibly due to capacity issues . Relation-based KD could play a bigger role possibly increasing accuracy in future research , especially when considering alternative architecture. Additionally, using top-10 accuracy in USPTO-50k might be a misleading benchmark, as we observed that top-1 accuracy was inversely related to top-10 accuracy in USPTO-50k but not in USPTO-full. This indicates that the long training on USPTO-50k does not translate well to beam search accuracy, as the model becomes overconfident to optimize for low-diversity high-accuracy responses over diversity.