To understand the decision-making process of retrosynthesis frameworks, a novel interpretability study is carried out. In particular, the interpretability of semi-template and template-free architectures is investigated. Template-based frameworks are considered inherently interpretable since one can link their prediction directly to literature precedent. Furthermore, model interpretability has been investigated and reported for the template-based category . The aim of this study is to uncover whether the other two framework categories can capture chemically important functional groups, sterics and charge transfers in the reaction -similar to a chemist.