To streamline the benchmarking results, a summary of all evaluation metrics is provided within Table . Analysing and optimising each metric individually is unfeasible and direct comparison between frameworks becomes difficult. This is because no framework category performs best on all evaluation metrics. Instead, we propose that optimising round-trip accuracy provides the best performance measure of a retrosynthesis framework. Nonetheless, the round-trip accuracy is flawed in certain ways. First, it does not take into account the diversity of the predictions e.g. the simple nucleophilic substitution C2H5X + NH3 --→ C2H5NH2 + HX is chemically feasible for various nucleophiles (e.g. Br -, Cl -). Trivial changes in chemistry therefore boost the round-trip accuracy. Second, the round-trip accuracy provides no measure of the number of duplicate reactions. If a framework proposes a large number of duplicates of a feasible reaction, it will inherently boost the round-trip accuracy, too. Third, no measure on invalid molecules (SMILES) is provided. It is postulated that a retrosynthesis framework that has "learnt" chemistry, should not generate invalid molecules. Therefore, the benchmarking is formulated as optimising the round-trip accuracy while holding "soft" constraints on all other metrics introduced in Section 2.2.2. These constraints are user-selected in a reasonable fashion. As such, the set of constraint is given as Div ≥ 2.5 with SJS ≥ 0.9, Dup ≥ 0.8, ∆SC ≥ 0.35, Inv10 ≤ 10%. In other words, for a set of 10 reactions per given target, the frameworks should propose on average at least 2.5 reaction classes with no more than 2 duplicate reactions and 1 invalid reaction (SMILES). The similarity SJS of at least 90% ensures that a framework proposes all 10 reaction superclasses (see ESI for more detailed information). The constraint for the SCScore difference was chosen according to ∆SCgt of the ground-truth test dataset, which is equal to ≈ 0.48. The reactions contained in the USPTO dataset were most likely patented due to an efficient synthesis route. Thus, it would be unreasonable to expect the frameworks to outperform the dataset. The constraint on ∆SC is therefore chosen to be within 30% of ∆SCgt. From Table , it can be seen that template-based models perform overall the best for the round-trip accuracy with few duplicate and invalid predictions. In fact, LocalRetro is the only model that satisfies all soft constraints with a high rt-accuracy of 81%. Considering that templates contain data on chemically viable reactions from prior literature and experimental studies, this outcome may not be particularly unexpected. In contrast, the other two categories exhibit lower performance in suggesting feasible reactions, possibly because they lack chemically meaningful descriptors. For the template-free models, it can be observed that they generate less diverse reactions than the other two categories. An extreme example is Chemformer, which predicts a large number of duplicates of the ground-truth reaction, thereby leading to a large inflation of the rt-accuracy. Nonetheless, efforts made by the community have led to a smaller degree of invalid SMILES (1-8% on average) compared to the 20% of Liu et al. 's initial work on sequence models. Furthermore, template-free models are observed to be best at the synthetic complexity (SCScore) metric. However, it is hypothesised that template-free models experience the common problem of length-based overfitting on the training dataset. This means that the models prefer to generate shorter SMILES sequences due to the appearance of short reactant SMILES in the training database. On the other hand, the semi-template category (with the exception of GraphRetro) is observed to produce a diverse set of reactants with few duplicate reactions. This is thanks to the nature of the two-step approach as seen in Figure . During the reaction centre detection step, the algorithm can sample different reaction centres (pertaining to different reaction chemistry). This ensures a larger number of diverse and unique predictions as seen for G 2 Retro. Nevertheless, while diverse reactions with few duplicates are important, they slightly decrease the rt-accuracy (e.g. G 2 Retro). Interestingly, the semi-template models propose the largest degree of invalid molecules. The preceding literature has not highlighted this issue. Therefore, this challenge is yet to be tackled.