In recent works, Torren-Peraire et al. benchmarked single-step retrosynthesis models in a multi-step fashion. Instead of evaluating the model on single reactions via the traditional top-k accuracy, their benchmark considers the full synthesis route from building blocks to the target molecule. While this is a reasonable idea, the authors utilised the top-k route accuracy and number of solved routes as evaluation metrics. Similar to the top-k accuracy, the route accuracy biasedly rewards models that recall existing routes from the test database. Additionally, a permissive model may propose a larger number of solved routes due to chemically unrealistic and unfeasible reaction steps . Maziarz et al. improved upon previous shortcomings by counting the number of diverse routes predicted by a model within a time window. Nonetheless, the authors acknowledged the difficulty of validating synthesis route feasibility i.e. if the route would likely be experimentally successful. We argue that models should most importantly predict (experimentally) feasible reactions. Consequently, our pipeline (Section 2.2.2) evaluates models based on their ability to propose feasible, diverse and unique reactions.