SDDBench: A Benchmark for Synthesizable Drug Design
Drug Discovery
We believe that some reviewers may lack sufficient background in retrosynthetic planning, current evaluation methods for retrosynthetic algorithms, and the definition of synthesizability. We recommend that reviewers refer to [1] for a more precise evaluation of submissions within this field in the future. It appears that some reviewers continue to rely on metrics such as search success rate, which, as we have explained, can lead to imprecise evaluations.

Additionally, synthesizability metrics like the SA score and SCScore do not ensure that a feasible synthetic route can be identified for a molecule, as we have emphasized throughout this paper. The true goal of synthesizability is to find an actual feasible synthetic route. Regardless of how high these metric scores may be, they cannot guarantee the discovery of a feasible synthesis route. Therefore, we propose that future metrics should be aligned with the retrosynthetic planner—a direction to which this paper is dedicated.

[1] FusionRetro: Molecule Representation Fusion via In-Context Learning for Retrosynthetic Planning, ICML 2023.
W1: The analysis and discussion in Section 5, which presents the benchmark, lacks depth.

A1: We already provided molecule property analysis in Table 8 in Appendix.

W2: Lastly, the paper does not provide guidelines on how the synthesizability objective could be balanced with other (potentially opposing) objectives.

A2: Yes, our findings show that molecules generated by pocket2mol have the highest likelihood of being solved in a single synthesis step. Smaller molecules tend to be more easily solved within a shorter number of steps. The average synthetic route length for molecules proposed by Pockert2Mol is 2.93. 14.7% of them proposed Pocket2Mol by are building blocks.

W3: Other metric candidates compared with SA score.

A3: Firstly, the SA score is the most widely used indicator of synthesizability, and it is utilized by the SBDD model and all drug design models for this purpose. Our goal is to replace the SA score, and the experiment clearly demonstrates the adequacy of our approach.

W4: Inference Speed

A4: We have provided time analysis in Line 498-501.

W5: Presentation

A5: With the current 10-page limit, we can only present a limited amount of information. Many concepts are drawn from previous papers and don't fit well into the main text. While we considered placing them in the appendix, this still requires readers to jump back and forth several times.

Q1: Why is the search success rate not a reliable metric (line 252)? This is an interesting point but the paper could discuss this claim more and ideally provide empirical evidence.

A6: We have provided short analysis in Section 2.6. Reviewers are encouraged to refer to [1] for further details.

Q2: The definition of the round-trip score using Tanimoto similarity...

A7: Eq. 1 represents the joint probability: the probability that the drug design model first generates a molecule and the probability that the retrosynthetic planner then generates a synthesis pathway for this molecule. It is unrelated to the round-trip score.

Q3: Have you also experimented with established tools (e.g. AIZynthFinder) or pretrained models?

A8: Since a retrosynthesis model must be trained to generate a feasible synthesis route, other tools rely on the search success rate, which, as we have already explained, is flawed.

Q4: What is the typical run time to compute this score? How does it compare to other synthesizability scores?

A9: We have reported the time in Lines 498-502, utilizing the KV cache, which allows for very fast processing of a single molecule. Its runtime matches the time required for the retrosynthetic planner to predict synthesis routes for a single molecule.

[1] FusionRetro: Molecule Representation Fusion via In-Context Learning for Retrosynthetic Planning. ICML 2023.
We believe the primary challenge in synthesis is identifying a feasible synthetic route for the molecule. It appears this reviewer may lack sufficient background knowledge on the topic.

W1: Difference from previous works

A1: Previous work [3] has primarily followed a top-down approach to drug design and synthesis planning. In contrast, our method begins with the desired product molecule and uses a retrosynthetic planner to build a synthesis route in a top-down manner. We then employ a forward reaction model to evaluate the feasibility of the route. In this regard, we are indeed the first to adopt this approach. Additionally, when it comes to evaluating the feasibility of the synthetic route, [1,2] were the pioneers in this area. We have directly incorporated the findings from these two ICML-accepted papers into our methodology, which is novel we believe.  

SA, etc. score do not ensure that a viable synthetic route for a molecule can be identified, as we have discussed in this paper

Q1: When classifying the test set molecules as “successfully” solved...

A2: This is a challenge that remains unsolved within the community. Previous retrosynthetic planners have relied on search success rate to evaluate retrosynthetic methods, but many of the routes identified are not feasible, meaning the target molecule cannot actually be synthesized. This issue is discussed in [1,2]. In [1], a more robust metric was introduced that matches the predicted starting materials with those in a reaction database. However, this approach has its own limitations, as the reaction database may not cover all potential routes. Nevertheless, compared to search success rate, the matching-based method represents a significant improvement.

Q2: It would be informative for context to include molecule size/number of heavy atoms as another metric.

A3: We agree with the reviewer’s observation. Our findings show that molecules generated by pocket2mol have the highest likelihood of being solved in a single synthesis step. Smaller molecules tend to be more easily solved within a shorter number of steps.

Q3: A specific question about Pocket2Mol is in Table 1, the authors show that the Search Success Rate is highest. What is the route length distribution of these "successful" molecules? Are any directly in the building blocks?

A4: The average length is 2.93. 14.7% of the molecules proposed Pocket2Mol by are building blocks.

Q4: How many molecules out of the 10,000 generated molecules do not have a solved retrosynthesis route according to NeuralSym? 

A5: We have already reported the search success rate.

Q5: What are the building blocks used?

A6: They ares to determine the molecules in the leaf nodes of retrosynthetic routes are building blocks or not.



[1] FusionRetro: Molecule Representation Fusion via In-Context Learning for Retrosynthetic Planning. ICML 2023.  
[2] Preference Optimization for Molecule Synthesis with Residual Conditional Energy-based Models. ICML 2024.  
[3] Amortized tree generation for bottom-up synthesis planning and synthesizable molecular design. ICLR 2022.   
[4] Planning chemical syntheses with deep neural networks and symbolic AI, 2018 Nature.
W1: It seems to me that the "wrong" target is set for that metric

A1: Thank you for your suggestion. Currently, the reliability of our metric relies on the amount of reaction data used by our model. In theory, the more reaction data available, the more dependable the metric becomes. As discussed in the introduction, while certain natural molecules are theoretically synthesizable, feasible synthetic routes may remain undiscovered by humans. Based on this, we define a synthesizable molecule as one with a feasible synthetic route identifiable by a retrosynthetic planner. Feasibility is assessed by comparing the synthetic route predicted by the retrosynthetic planner with a reference route in the reaction database.

Our goal is to predict the synthetic routes of molecules generated by various drug design models directly through the retrosynthetic planner. We then evaluate the reliability of the round-trip score. In the first category of molecules, you observed that 99.5% achieved a round-trip score of 1, indicating that our round-trip score correctly identifies 99.5% of synthesizable molecules. We agree on the importance of evaluating the accuracy of the round-trip score for negative examples (molecules known to be unsynthesizable). However, defining unsynthesizable molecules poses significant challenges. Unsynthesizable could mean that the molecular structure cannot exist at room temperature and pressure, or that a synthesis route cannot be found with current human knowledge. Each definition has its limitations.

However, our primary goal is to encourage drug design models to use our metric to identify molecules for which the retrosynthetic planner can find synthetic routes. Once identified, these molecules will indeed have findable synthesis routes. From this perspective, our metric is accurate and highly reliable. Other metrics, such as the SA score, cannot guarantee the discovery of synthetic pathways, even at high values.


W2: Some claims made in the first paragraph on the introduction should be supported by appropriate citations.

A2: We will cite more papers.

W3: the claim on lines 74-76

A3: We agree with your point and will make the necessary corrections to this statement.

W4: Related Work

A4: We will introduce more.

W5: Guo's work

A5: We have thoroughly reviewed the details of this work. However, in our view, the synthetic metric used in this paper is flawed. The SA score does not ensure that a viable synthetic route for a molecule can be identified, as we have discussed in this paper. AiZynthFinder, on the other hand, assesses the solvability of a molecule's synthesis using the search success rate, which has limitations discussed in [1]. 

Q6: Why would we expect retrosynthesis planners to be less limited than template-based synthesis planners? 

A6: Their approach begins with a building block, predicting the reaction template (a classification problem with fewer than 100 categories) to generate the next intermediate molecule. Then, by iteratively predicting reaction templates through each intermediate molecule, they construct a synthetic pathway to the final molecule. The final molecule is then quantified to generate a reward, and they use reinforcement learning (RL) from scratch to train, with a limit on the number of reaction steps.

We attempted a similar method, expanding the reaction template to thousands of types, but encountered challenges with convergence in the loss function, as RL training can be quite difficult. In contrast, our approach first generates molecules using the drug design model and then employs our proposed synthesis reward withour limit on the reaction template for filtering or post-training optimization—a clearly more efficient method. For reference, consider this ICML talk, https://icml.cc/virtual/2024/invited-talk/35253, which highlights that RL with a strong starting point (post-training) outperforms RL from scratch.


[1] FusionRetro: Molecule Representation Fusion via In-Context Learning for Retrosynthetic Planning. ICML 2023.
W1: Citation

A1: We will cite more papers if this reviewer can give some references.

W2: Claim

A2: Previous work [3]  has primarily followed a top-down approach to drug design and synthesis planning. In contrast, our method begins with the desired product molecule and uses a retrosynthetic planner to build a synthesis route in a top-down manner. We then employ a forward reaction model to evaluate the feasibility of the route. In this regard, we are indeed the first to adopt this approach. Additionally, when it comes to evaluating the feasibility of the synthetic route, [1,2] were the pioneers in this area. We have directly incorporated the findings from these two ICML-accepted papers into our methodology, which is novel we believe.

W3: Guo's work

A3: We have thoroughly reviewed the details of this work. However, in our view, the synthetic metric used in this paper is flawed. The SA score does not ensure that a viable synthetic route for a molecule can be identified, as we have discussed in this paper. AiZynthFinder, on the other hand, assesses the solvability of a molecule's synthesis using the search success rate, which has limitations discussed in [1].  

W4:  the use of retrosynthesis tools in previous works.

A4: We want to emphasize that we introduce a new data-driven synthetic metric, whereas previous studies continue to rely on SA Score and SCScore. The limitations of these existing metrics are discussed in detail in this paper.

W5: the added value of round trip score in the molecule-wise setting over just reporting retrosynthesis success rates  

A5: The limitations of Search Success Rate are discussed extensively in Section 2.6, and [1] devotes significant attention to criticizing this metric as flawed. While Round-trip Score evaluates individual points, Search Success Rate is batch-oriented. Furthermore, without reference routes, Round-trip Score serves as a more rigorous and effective metric than Search Success Rate.

W6: Limitation

A6: The primary limitation is the insufficient reaction data to cover all possible molecules.

W7: The authors mention provided code in Section A.1

A7: We are referring to the code for the drug design model, which is available in their paper.

W8: Data Leakage

A8: Please give the detailed dataset you mention so that we can answer your question

W9: In Table 1, the metrics of the training data and test set could be included.

A9: We evaluate the molecules generated by the SBDD model on the test set targets. Any issues that arise are attributable to the SBDD model itself. Could you provide the ChEMBL molecule resources for evaluation?

Q10: Figure 5

A10: The retrosynthesis model is Neuralsym. The forward reaction model is Transformer Decocder.

Q11: Figure 6

A11: Figure 6 highlights that the SA score cannot effectively distinguish between successful and failed cases, resulting in an overlap between them.

Q12: There is no background information on any of the other benchmarking efforts in drug discovery

A12: Please refer to Line 475-482.

Q13: There are no citations for models in the “two-stage learning paradigm”.

A13: Semi-template-based method employ two-stage learning paradigm........, which we have discussed in Line 454-457.


[1] FusionRetro: Molecule Representation Fusion via In-Context Learning for Retrosynthetic Planning. ICML 2023.  
[2] Preference Optimization for Molecule Synthesis with Residual Conditional Energy-based Models. ICML 2024.  
[3] Amortized tree generation for bottom-up synthesis planning and synthesizable molecular design. ICLR 2022.
Rebuttal
Yes
N/A
No ethics review needed.
- Can the authors clarify exactly what data split they use to train the CrossDocked models?
- How robust is the proposed metric to the exact retrosynthesis tool used?
- Synthesis is an important issue for generative models and the authors propose to address this with a new synthesis benchmark.
- The background on retrosynthesis and reactions is good.
- Quality of the methods figures is good
This paper presents SDDBENCH, a benchmark for evaluating the synthesizability of molecules generated by drug design models. It incorporates a "round-trip score," which assesses whether generated molecules can be practically synthesized by combining retrosynthetic planning with forward reaction prediction. The benchmark attempts to address a gap in drug discovery where generated molecules often display desirable pharmacological properties but pose challenges for real-world synthesis. The authors apply the round-trip score across various 3D structure-based drug design generative models, highlighting their synthesizability, although no baseline is given.
**Main Weaknesses:**

- The language is extremely vapid, making it difficult to pick out the genuine contributions, and the arguments/paragraphs are not well structured.
    - The 1st paragraph of the introduction reports about 10 scientific facts about synthesis from the literature and yet only cites 1 deep learning article half-way through. This, in my opinion, should be reason enough for a rejection.
    - The structure of the paper is strange, with it first proposing the method, doing experiments to validate the method, then describing related work/background before finally benchmarking the models (in about 1 page).
- There is a significant lack of novelty in the ideas presented in the proposed metrics. This a further compounded by a number of novelty and contribution claims that are unjustifiable.
    - For instance, “Our work is the firs to bridge the gap between drug design and retrosynthetic planning.” This is a big claim to make given the authors cite many such works that do exactly this.
    - Guo and Schwaller, 2024 [1] recently directly optimised generative models using retrosynthesis planners. 
    - Furthermore, the use of retrosynthesis tools as an evaluation metrics has been used in many works.  [2, 3]
    - The paper also does not clearly communicate the need for the round trip score, which itself is a multi step extension of Schwaller et al, 2020 [4]. In Figure 5 right, it seems the successful cases all have a round trip score of 1, whereas the failed cases consistently have a round trip score implying negligible similarity. Can the authors then please explain the added value of round trip score in the molecule-wise setting over just reporting retrosynthesis success rates?
- The authors do not critically discuss the limitations of their work beyond stating there is a limitation in the amount of reaction data available.
- An interesting literature has emerged recently that aims to tackle the problem of synthetic accessibility by incorporating reaction-based constraints to the generative process into generative models [2, 3, 5]. However, mainly autoregressive and diffusion-baed models specialised for structure-based drug design were accessed? If this benchmarking study is to be useful these lines of work should be considered too.
- The authors mention provided code in Section A.1 but there is no link.
- As far I can tell, a random split was used. How is data leakage controlled?
- In Table 1, the metrics of the training data and test set could be included. I would also consider ChEMBL molecules as a baseline?



**Minor points:**

- Figure 5: Figure not clear. What model(s) is this for? What do the error bars represent?
- Figure 6: Can the error bars for failed and successful cases be made staggered/non-overlapping. It is very confusing currently.
- There is no background information on any of the other benchmarking efforts in drug discovery. E.g. physical realism like PoseCheck [6]
- Citing ~40 retro-synthesis models in the background is excessive and also not useful. I would instead cite as different categories are described. There are no citations for models in the “two-stage learning paradigm”.

Spelling mistakes:
- L077 firs -> first
- L302 have -> has

1. Guo, Jeff, and Philippe Schwaller. "Directly optimizing for synthesizability in generative molecular design using retrosynthesis models." arXiv preprint arXiv:2407.12186 (2024).
2. Stanley, Megan, and Marwin Segler. "Fake it until you make it? generative de novo design and virtual screening of synthesizable molecules." Current Opinion in Structural Biology 82 (2023): 102658.
3. Cretu, Miruna, et al. "Synflownet: Towards molecule design with guaranteed synthesis pathways." arXiv preprint arXiv:2405.01155 (2024).
4. Schwaller, Philippe, et al. "Predicting retrosynthetic pathways using transformer-based models and a hyper-graph exploration strategy." Chemical science 11.12 (2020): 3316-3325.
5. Bradshaw, John, et al. "A model to search for synthesizable molecules." Advances in Neural Information Processing Systems 32 (2019).
6. Harris, Charles, et al. "Posecheck: Generative models for 3d structure-based drug design produce unrealistic poses." NeurIPS 2023 Generative AI and Biology (GenBio) Workshop. 2023.
Yes
No ethics review needed.
1. Why would we expect retrosynthesis planners to be less limited than template-based synthesis planners? In particular, on lines 218-222, it is mentioned that "While previous methods (Bradshaw  et al., 2019b; Gao et al., 2022a) have attempted to simultaneously generate molecules and their  synthetic routes, they face limitations. For instance, the approach in (Gao et al., 2022a) is constrained  by the use of a limited number of reaction templates, which hinders scalability.". From my view, retrosynthesis planners rely on retrosynthesis predictors, which have been trained on finite dataset of reactions, so these models are also limited to the set of reactions that have been seen during training. How well can these models generalise to new reactions? This concern is further emphasized in the conclusion, lines 538-539: "To enhance the  robustness of our evaluation method, we advocate for the release of additional reaction data.". If more reaction data was available, couldn't competing methods based on templates simply extract more templates from this additional data? Thus aren't *both* family of methods limited by existing data?
I believe that this work addresses an important limitation of many molecular generation models i.e. the lack of synthesizability guarantees of the proposed molecules. The ability of ML-driven methods to propose synthetically accessible compounds represents an important gap for the successful applications in real-world impactful problems such as drug discovery. While not the only possible approach, combining unconstrained molecular generators with retrosynthetic planners is an interesting direction for addressing this limitation. The authors propose a reasonable metric combining two different models trained on reaction datasets to assess the synthesizability of a given compound. I found the paper to be generally well written. 

In particular:
1. The Background section is clear and well presented.
2. The Related Work section is substantive and informative (although there might be some missing areas, see below).
3. The experiments benchmarking existing SBDD models in Section 5 and in appendix are informative.
4. I found the discussion on lines 355-366 comparing the performance of the forward prediction model and retrosynthesis planner to be informative.
In this work, the authors focus on synthesizability assessment for molecules proposed by structure-based drug design (SBDD) models. Molecular generators (especially structure-based) often generate molecules that maximise a target metric but are hard or impossible to synthesize. To address this limitation, the authors propose a new metric, called Round-Trip Score. The metric works as follows: given a molecule proposed by some molecular generation model, a synthesis plan is proposed by a retrosynthesis planner. A different, single-step forward synthesis model is then used to simulate wet lab experiments and sample the next intermediate products until a final molecule is reached. The Round-trip score is given by the tanimoto similarity between the final molecule obtained through simulated forward synthesis and the initial query molecule. They evaluate the reliability of the proposed metric using the USPTO reaction dataset, and use it to benchmark a variety of existing SBDD models.
Despite the merits of this work (see above), I also have important concerns. I believe that the proposed metric (Round Trip score) is reasonable and can bring value by being used for evaluating (or even guiding) the ability of SBDD models to generate synthesizable compounds. However, my main concern is about the reliability evaluation of the proposed metric:

1. It seems to me that the "wrong" target is set for that metric: the authors mention multiple times that the proposed Round-Trip score aims to "assess the likelihood that these proposed molecules can have feasible synthetic routes predicted by the retrosynthetic planner" (lines 401-402). This is *what* the metric is doing, how it is built, but what it should *aim* to do is to actually inform the experimenter on whether a proposed molecule is likely to be synthesizable in real-life. Therefore, the evaluation of the reliability of that metric should seek to assess to what extent this Round Trip score correctly distinguishes between molecules that are known to be synthesisable, and molecules that are known not to be synthesisable. However, both of the categories described in Figure 5 ("Successful" cases and "Failed" cases) belong to the first category: molecules that are known to be synthesizable: "We employ Neuralsym+Beam Search (beam size is 2) as our retrosynthetic planner to predict synthetic  routes for target molecules in the test dataset. We then compare these predicted routes to the reference  routes provided in the dataset. If the starting materials in our predicted routes match those in the  reference routes, we categorize these molecules as successful cases. Conversely, if they don’t match,  we classify them as failed cases." (lines 320-323). Therefore, observing a gap in the Round Trip score between these two categories essentially highlights a failure mode of the Round Trip score in assessing real-life synthesizability. I believe authors should instead report the proportion of such failure cases (i.e. the size of these relative groups: Successful Cases v.s. Failed Cases) as a measurement of the performance of the retrosynthesis planner component of the metric, and complement this analysis by evaluating the Round Trip score accuracy on negative examples as well (molecules that are known not to be synthesizable). The fact that the Round Trip score is high is simply a measure of the effectiveness of the forward synthesis prediction component (which shows very positive results) but not of the reliability of the metric as a whole. To be clear, the proposed metric still has merits and could be useful, but its evaluation appears flawed and thus its development incomplete. I might be missing something, in which case I believe this matter should be clarified in the manuscript, but otherwise, for the above reasons, I would currently recommend this paper not to be accepted in its current form.

In addition, here are additional points that I think could contribute to improving the paper in its current form:

2. Some claims made in the first paragraph on the introduction should be supported by appropriate citations. For exemple, on lines 30-32: "Drug design is a fundamental problem in machine learning for drug discovery. However, when these computationally predicted molecules are put to the test in wet lab experiments, a critical issue often  arises: many of them prove to be unsynthesizable in practice". From experience I agree with this statement, but it should be supported by some references (many papers have studied and addressed the poor synthesizability of compounds proposed by ML-driven molecular generators). The same goes for this other claim: "The sensitivity of chemical reactions is such that even minor change in functional groups  can potentially prevent a reaction from happening as anticipated." (lines 41-42).

3. The first contribution listed by the authors includes this sentence: "In our view, a molecule is synthesizable if retrosynthetic planners, trained on existing reaction data, can predict a feasible  synthetic route for it" (lines 74-76). In my opinion, while the use of data-driven methods for assessing synthesizability has merits and is worth studying, such a strong formulation represents a highly objectionable claim. ML-based retrosynthesis planners have flaws and blindspots and do not represent perfect oracles to assess a molecule's synthesizability, as highlighted by authors on lines 298-300, and in their own experiments in Section 3.2.2 as detailed above. One could argue, for example, that forward synthesis search methods based on actual reactions (rather than predicted reactions) give much stronger guarantees for synthesizability. I thus believe this claim on lines 74-76 to be incorrect and that it should be rephrased. For approximation, an *approximation* of synthesizability might be given by the success rate of a retrosynthesis planner. This would represent to me a weaker but more accurate claim.

4. While the Related Work section is well structured and pretty exhaustive, I believe that the positioning of this work could be greatly improved by explaining more clearly how its focus area (SBDD) cannot be tackled by other types of synthesis-aware molecular generation methods, which aims at producing molecules and synthesis path simultaneously, e.g. [1,2,3,4,5,6]. This line of work is briefly mentioned on lines 54-55 but should be much more exhaustively covered as this relates to the positioning of this contribution in the broader community. 

5. Similarly, the Related Work should also cover alternative synthesizability score e.g. SC score [7].

6. About the claim: "Our work is the firs to bridge the gap between drug design and retrosynthetic planning." (lines 77-80). This seems like a strong claim to me. There is at least this concurrent work by Guo & Schwaller [8] which does combine both family of methods, and I would be rather surprised that no previous work combined retrosynthesis planners and molecular generation.

Minor Comments:
1. typo line 77: is the first*
2. Figure 4 could be made clearer by highlighting in different colours or boxes the different steps of the score computation (retrosynthesis planning, forward synthesis simulation, molecule comparison).
3. The first paragraphs of Section 3.2.2 (under "Settings") could be made clearer. For example, Neuralsym is cited but not defined (while the reference is useful, the manuscript should be standalone).
4. The availability of the code is mentioned in Appendix (line 927) but no link is given. It would be good to at least clearly mention that the code will be made publicly available upon publication.

References:
- [1] Vinkers, H. M., de Jonge, M. R., Daeyaert, F. F., Heeres, J., Koymans, L. M., van Lenthe, J. H., ... & Janssen, P. A. (2003). Synopsis: synthesize and optimize system in silico. Journal of medicinal chemistry, 46(13), 2765-2773.
- [2] Gottipati, S. K., Sattarov, B., Niu, S., Pathak, Y., Wei, H., Liu, S., ... & Bengio, Y. (2020, November). Learning to navigate the synthetically accessible chemical space using reinforcement learning. In International conference on machine learning (pp. 3668-3679). PMLR.
- [3] Horwood, J., & Noutahi, E. (2020). Molecular design in synthetically accessible chemical space via deep reinforcement learning. ACS omega, 5(51), 32984-32994.
- [4] Gao, W., Mercado, R., & Coley, C. W. (2021). Amortized tree generation for bottom-up synthesis planning and synthesizable molecular design. arXiv preprint arXiv:2110.06389.
- [5] Cretu, M., Harris, C., Roy, J., Bengio, E., & Liò, P. (2024). Synflownet: Towards molecule design with guaranteed synthesis pathways. arXiv preprint arXiv:2405.01155.
- [6] Luo, S., Gao, W., Wu, Z., Peng, J., Coley, C. W., & Ma, J. (2024). Projecting Molecules into Synthesizable Chemical Spaces. arXiv preprint arXiv:2406.04628.
- [7] Parrot, M., Tajmouati, H., da Silva, V. B. R., Atwood, B. R., Fourcade, R., Gaston-Mathé, Y., ... & Perron, Q. (2023). Integrating synthetic accessibility with AI-based generative drug design. Journal of Cheminformatics, 15(1), 83.
- [8] Guo, J., & Schwaller, P. (2024). Directly optimizing for synthesizability in generative molecular design using retrosynthesis models. arXiv preprint arXiv:2407.12186.
Yes
No ethics review needed.
1. When classifying the test set molecules as “successfully” solved, the metric is whether the solved routes have starting materials identical to the reference routes. As eluded by the authors, many starting materials can lead to the same product. The authors use the classified “successful” and “failed” sets to assess round-trip score using the forward model. How does the separation change when considering all molecules that have a solved retrosynthetic route (even if not identical to the reference route)? This can be heavily biased because the retro and forward are trained with no failed reactions. If a retro route does not lead to the reference route’s starting material, it does not necessarily mean it is incorrect.

2. Pocket2Mol generally generates smaller molecules. The metrics for it may be high because many of the generated molecules are almost directly building blocks. It would be informative for context to include molecule size/number of heavy atoms as another metric. 

3. A specific question about Pocket2Mol is in Table 1, the authors show that the `Search Success Rate` is highest. What is the route length distribution of these "successful" molecules? Are any directly in the building blocks?

4. How many molecules out of the 10,000 generated molecules do not have a solved retrosynthesis route according to NeuralSym? This is an important metric missing to contextualise the round-trip score.

5. What are the building blocks used?

Minor comment: The section on AGI breaks the flow of the text and in my opinion, not needed. More data will always lead to better models and is not exclusive to reaction data or language data (like in the GPT example given by the authors).
* Synthesizability is an important problem in generative design
* Motivation of using Round-Trip score by comparing to SA score and that it can better distinguish between "successful" and "failed" molecules
* Comparison of several 3D models
The paper draws attention to synthesizability in generative design and proposes the "round-trip score" which assesses synthesizability by whether the predictions of a retro- and forward-synthetic model overlap, and otherwise, how similar the predictions are. The results contrast several 3D generative models and show that better property metrics such as docking score, does not necessarily equate to better synthesizability.
Overall, while synthesizability is an important problem, the proposition to assess generated molecules with retrosynthesis models has been reported in cheminformatics and ML literature.

* The idea of assessing generated molecules by retrosynthesis models is routinely used. In cheminformatics literature, this has been very common and many works benchmark proxy scores, such as SA, SC, RA scores with retrosynthesis models to study their correlation [1]. Other works explicitly highlight the use of retrosynthesis models to assess the synthetic feasibility of generated molecules [2, 3]. 

* At conferences, work on synthesis constrained generative models explicitly tackle synthesizability. Here I will cite a few of them [4, 5, 6].

* At conferences, work on synthesis constrained generative models assess generated molecules using retrosynthesis models. Here I will cite a few of them [7, 8, 9]. Although I acknowledge some of these are concurrent works.

* This paper, in addition to [10] cited by the authors, also explicitly highlight the problem of synthesizability of generated molecules [11]. This paper also discusses the use of surrogate models aimed to predict the output of retrosynthesis models and have routinely been used to assess synthesizability. Some specific examples are RAscore [12] and RetroGNN [13]. While this is not exactly the same as directly assessing using retrosynthesis models, these models are trained on the output of retrosynthesis models and is a similar idea.

* The authors stated in the introduction that a minor change in functional group can make a synthesizable molecule, unsynthesizable. The proposed metric to compute the Tanimoto similarity between the reconstructed and generated molecule as a proxy for synthetic accessibility could suffer from more error. Molecules with 0.9 Tanimoto similarity can have quite different atom arrangements that affect reactivity and therefore synthesizability. It would be interesting to take a series (share common structural features) of known synthesizable molecules, run retrosynthetic analysis on them, and then run the forward model on the routes. This could provide insights into the sensitivity of the proposed Round-Trip score on seemingly small structural changes. It would also be interesting to see how many molecules fail the round-trip despite being known to be synthesizable, due to limitation of training data.

* The main text should also discuss the properties of the generated molecules (docking score, QED, etc.). I know the focus of the paper is on synthesizability but synthesizable molecules with poor properties will not be further considered in practical applications.

### **References**
[1] Synthesizability correlation: https://jcheminf.biomedcentral.com/articles/10.1186/s13321-023-00678-z

[2] Retrosynthesis models to assess generated molecules 1: https://pubs.rsc.org/en/content/articlelanding/2023/sc/d3sc03781a

[3] Retrosynthesis models to assess generated molecules 2: https://pubs.rsc.org/en/content/articlelanding/2024/md/d3md00651d

[4] MOLECULE CHEF: https://arxiv.org/abs/1906.05221

[5] DoG-Gen: https://arxiv.org/abs/2012.11522

[6] SynNet: https://arxiv.org/abs/2110.06389

[7] SynFlowNet: https://arxiv.org/abs/2405.01155

[8] Reaction GFlowNet: https://arxiv.org/abs/2406.08506v1

[9] RxnFlow: https://arxiv.org/abs/2410.04542

[10] Synthesizability Problem 1: https://pubs.acs.org/doi/full/10.1021/acs.jcim.0c00174

[11] Synthesizability Problem 2: https://www.sciencedirect.com/science/article/pii/S0959440X2300132X?via%3Dihub#bib53

[12] RAscore: https://pubs.rsc.org/en/content/articlelanding/2021/sc/d0sc05401a

[13] RetroGNN: https://pubs.acs.org/doi/10.1021/acs.jcim.1c01476
Yes
No ethics review needed.
- Why is the search success rate not a reliable metric (line 252)? This is an interesting point but the paper could discuss this claim more and ideally provide empirical evidence.
- The definition of the round-trip score using Tanimoto similarity between molecules appears rather _ad hoc_ and the connection with the theoretical objective stated in Equation 1 is not clear. How is the following sentence from the Conclusion Section justified? -- "This score evaluates the **likelihood** that a retrosynthetic planner, trained on current reaction data, can predict feasible synthetic routes for these molecules"
- Why do you train new reaction & retrosynthesis models? Have you also experimented with established tools (e.g. AIZynthFinder) or pretrained models?
- How do you justify the definition of the success criterion to evaluate the reliability of the round-trip score? Doesn't it depend on components of the score itself?
- What is the typical run time to compute this score? How does it compare to other synthesizability scores?



### Minor comments
- line 77: typo "t" missing in first
- Figure 5 & 6: legend should define meaning of error bars
- avoid using words like "exceptional" when describing own results (e.g. line 361)
- it is unclear to me what the "FusionRetro setup" introduced in Section 5 is referring to
### Overall idea and evaluation approach

The paper addresses an important limitation of many computational drug design efforts. Chemical synthesis is often the key bottleneck preventing generated compounds from being tested experimentally in a wet lab. 
The authors come to the plausible conclusion that a stringent synthesizability metric should evaluate the feasibility of the whole synthesis route for a given molecule. Unlike previous approaches they not only perform retrosynthesis planning but also employ a reaction prediction model to determine whether the target molecule can indeed be obtained from the predicted starting building blocks.
The resulting score based on the Tanimoto similarity of the target molecule and the output molecule after this round-trip procedure is well motivated and convincing in my opinion. The specifically trained reaction and retrosynthesis models seem to achieve accuracies close to those of state-of-the-art models from what I can tell, indicating that the technical implementation of the score is sound.


### Presentation of results & clarity
The evaluation procedure has a clear structure that first aims to establish the usefulness of the proposed synthesizability score, and then benchmarks several popular structure-based drug design models with the newly defined metric.
The paper also introduces the topic nicely and covers a wide range of relevant related work.
This paper proposes a new metric to assess the synthesizability of molecules. The presented approach makes use of data-driven reaction prediction and retrosythesis models to directly measure if feasible synthetic pathways are available for a given molecule. This kind of scoring metric is of high importance in light of the surge of generative drug design models whose outputs are rarely validated experimentally due to difficulties in making novel molecules in the lab.
### Scope of this work
Based on the title I expected the paper to present a new benchmarking suite. Instead, it only introduces a single new metric, the "round-trip score" that was applied to the molecules generated by seven recent models for structure-based drug design. While the general structure of this evaluation procedure is reasonable as discussed above, I would argue the execution is insufficient to justify the claim of establishing a new benchmark. Consequently, I recommend to revise the title and claims to more accurately reflect the scope of this work.

The analysis and discussion in Section 5, which presents the benchmark, lacks depth. The paper does not provide any potential reasons for the differences between the benchmarked methods that could inform the future development of such algorithms.
To improve the paper, the authors could for example correlate their score with various molecular properties (e.g. molecular weight, number of rotatable bonds, hydrophobicity score, ...) and analyze the results. It would also be valuable to discuss case studies based on molecules with particularly low or high round-trip scores.

Lastly, the paper does not provide guidelines on how the synthesizability objective could be balanced with other (potentially opposing) objectives. For instance, I imagine molecule size to play a crucial role. Smaller molecules are often easier to synthesize but bind their targets less specifically.


### Empirical results

Assuming that the main focus of this paper is the new synthesizability metric rather than the benchmark in Section 5, I would have hoped for a more thorough evaluation thereof.
If I understand correctly, the same retrosynthesis model that is part of the round-trip score computation is also used in the definition of successful and failed cases in the evaluation in Section 3.2.2. Similarly, the reaction prediction model seems to be used in the "molecule-wise match" scenario. 
Given that the success criterion seems to be coupled to components of the newly proposed score, it is unclear to me how meaningful the comparison with the SA score is.
Furthermore, the empirical evaluation of the round-trip score should be extended to other baselines besides the SA score. Obvious candidates would be:
- SYBA [1],
- SCScore [2],
- RAscore [3].

These scores are more recent than the SA score but still commonly employed in this domain, which makes it easy for readers to interpret the presented results.

Finally, while the full forward-backward reaction prediction is well motivated, I suspect that its main limitation will be speed–at least compared to heuristics-based synthesizability scores. A comprehensive evaluation should therefore also include the time component, and could for example discuss the pareto front of high accuracy and low run time. To this end, the authors could add a runtime analysis in their evaluation.


### Presentation

To me, it was not always easy to follow the flow of the paper. Most concepts are correctly defined somewhere but not always in the order that would have seemed natural to me. Consequently, I had to jump back and forth several times while reading the paper.
The first sections include a few definitions that are not required for understanding the method, e.g.
- the definition of ligands and proteins in Section 2.1,
- the definition of the gap between metric scores in Equation 2.

At the same time, important concepts like the the set-wise exact match criterion and its difference to the molecule-wise (round-trip) match could have been explained more clearly.

The first sentence of the _Results_ paragraph of Section 3.2.2 introduces a new definition of successful cases. This should have been explained earlier. In general, it might be cleaner to move all the results to Section 5 and limit the discussion in Section 3 to a description of the method.

### References

[1] Voršilák, Milan, et al. "SYBA: Bayesian estimation of synthetic accessibility of organic compounds." Journal of cheminformatics 12 (2020): 1-13. 

[2] Coley, Connor W., et al. "SCScore: synthetic complexity learned from a reaction corpus." Journal of chemical information and modeling 58.2 (2018): 252-261.

[3] Thakkar, Amol, et al. "Retrosynthetic accessibility score (RAscore)–rapid machine learned synthesizability classification from AI driven retrosynthetic planning." Chemical science 12.9 (2021): 3339-3349.
@inproceedings{
anonymous2024sddbench,
title={{SDDB}ench: A Benchmark for Synthesizable Drug Design},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=IjiIPQcLbV},
note={under review}
}
A significant challenge in wet lab experiments with current drug design generative models is the trade-off between pharmacological properties and synthesizability. Molecules predicted to have highly desirable properties are often difficult to synthesize, while those that are easily synthesizable tend to exhibit less favorable properties. As a result, evaluating the synthesizability of molecules in general drug design scenarios remains a significant challenge in the field of drug discovery. The commonly used synthetic accessibility (SA) score aims to evaluate the ease of synthesizing generated molecules, but it falls short of guaranteeing that synthetic routes can actually be found. Inspired by recent advances in top-down synthetic route generation, we propose a new, data-driven metric to evaluate molecule synthesizability. Our approach directly assesses the feasibility of synthetic routes for a given molecule through our proposed round-trip score. This novel metric leverages the synergistic duality between retrosynthetic planners and reaction predictors, both of which are trained on extensive reaction datasets. To demonstrate the efficacy of our method, we conduct a comprehensive evaluation of round-trip scores alongside search success rate across a range of representative molecule generative models.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Drug Discovery
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/f98426ef96351ec1d99110c3d8eeb2f32e5b8c81.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
SDDBench: A Benchmark for Synthesizable Drug Design
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission