Reframing Structure-Based Drug Design Model Evaluation via Metrics Correlated to Practical Needs
Stucture-Based Drug Design;Model Evaluation;Benchmark
Yes
No ethics review needed.
- How would your method evaluate a generated molecule that has a very different structure from known actives, but demonstrates strong binding affinity in docking simulations?

- Can you please clarify why you are so confident that this is a practical alternative to wet-lab experiments?
- The problem that this paper attempts to address is one of high importance. Many generative/optimization models in this field rely on computational property predictors, which may not be accurate. The alternative would be to conduct real-world wet-lab experiments, which in many cases, is not feasible. 
- The three proposed assessment techniques capture different aspects of the generated molecules and make for a combination that avoids redundancy. 
- The datasets used (especially the testing set) seem to be satisfactory.
This paper introduces a three part framework to assess SBDD-generated molecules. The first part is similarity between generated molecules and known active compounds, achieved using molecular fingerprints. The second part is a virtual screening metric, a combination of BEDROC and EF, to measure the ability of generated molecules to distinguish between active and inactive compounds. The final part is binding affinity to a target, in which Vina and delta scores, and DrugCLIP were used to achieve this. Results show that amongst the used models, MolCRAFT dominated across multiple metrics.
- The authors state: “This provides a practical alternative to testing deep learning-generated molecules directly in wet-lab experiments.” I believe this is quite a bold statement and, unfortunately, am not fully convinced of this. While the authors provide a neat framework and results, I think more experiments on larger datasets are required to show that this statement holds true. Moreover, it would be much more convincing if the authors are able to demonstrate correlation between their metrics and actual wet-lab results on a subset of compounds. 


- I understand the rationale behind using similarity to known active compounds as an assessment. However, my concern is that this can be limiting – we can generate very desirable molecules that may be extremely different from the active compounds. 



Minor edits:
- Line 118: generated by docking software instead of real complex → complexes
- Line 307: Actually, ligands in PDBbind has a higher docking score → have
- This paragraph needs to be better written: "We refined the PDBbind dataset by excluding complexes with nuclear attachment and inaccurately recorded ligands. Then split into a 9:1 training and validation set. To assess the SBDD model’s generalization across diverse pockets, we removed samples with similar pockets using FLAPP for pocket alignment and similarity assessment. We remove all pockets from the training set with align rate more than 0.6 or 0.9 to the test set pocket. After the removal, the 0.6 version has 12344 pairs remaining while the 0.9 version has 17519 pairs remaining."
- Line 485: We do some visualizations → present 

- Figure 6: Be consistent with wording/capitalization: 
    - Molcraft Generated Mol 
    - Molecule generated by TargerDiff

- Figure 5 captions: include spaces after “(a) ...” and “(b) ...”
Yes
No ethics review needed.
* What are the authors' thoughts on evaluating the 3D geometry of the generated molecules? PoseCheck [1] and PoseBusters [2] are now common in SBDD papers. The results show that unphysical poses is an ongoing challenge for 3D generative models. Generated molecules with poses that clash with the protein would not typically be considered further.

* The authors state that the metrics do not require the need to consider synthetic ability but synthesizability is a problem highlighted in the Methods section. I agree that wet-lab testing should never be a requirement for computational research but do the authors think there are metrics that can better assess synthesizability beyond SA score and beyond being implicitly considered via similarity to known actives?

* Minor comment: there is a typo in "actives" in figure 2.

[1] https://arxiv.org/abs/2308.07413

[2] https://pubs.rsc.org/en/content/articlehtml/2024/sc/d3sc04185a
* The authors explicitly highlight pitfalls of current docking score evaluations of generated molecules
* Performance across several 3D models is compared and contrasted
* The rationale is clear when proposing new metrics
This paper draws attention to pitfalls in evaluation metrics commonly used to assess generative models for structure-based drug discovery. While the authors focus their discussion on 3D methods, the pitfalls discussed extend to any generative model for drug discovery. The authors first highlight the problem with the commonly used docking score metric and how it can be exploited. Next, three sets of evaluation tasks are proposed: delta docking score (assess specific binding), similarity metrics (to known actives and FDA-approved drugs), and virtual screening (can a similarity search from generated molecules identify known actives?). The authors provide insights on the current performance of 3D SBDD generative models and highlight shortcomings of current model development to guide future research.
Overall, I agree there should be better metrics for SBDD assessment. However, many of the metrics and challenges proposed by the authors has been extensively reported in existing literature. There are also many existing works that are not 3D-based diffusion/flow models for SBDD that account for the limitations expressed by the authors. As a result, some of the limitations are limitations of the problem formulation and not intrinsically to SBDD metrics. I will organize the comments under each proposed metric.

### **Docking and Synthesizability**
* The authors focus on SBDD generative models but there is a lot of work in this field that does not involve geometric models that can perform SBDD and recent benchmarks also show that these 1D/2D methods outperform 3D methods [1]. The reason this is brought up is because in these works, the limitation of AutoDock Vina to exploit low QED molecules and large molecules is well documented. As a result of this, there are many papers that explicitly optimise docking jointly with QED (and often also with SA score). Here are a few works that do this, many of which have been published at conferences [2, 3, 4, 5]. Therefore, while I do think that many papers are overfitting AutoDock Vina by designing large molecules with low QED, many recent works have acknowledged this and are proposing objective functions that explicitly account for QED.
* A large reason the Delta score is poor for AutoDock Vina is that it is too crude. This is likely to diminish if one uses Glide docking, as the authors implicitly show with the enrichment. The goal could also be explicitly to dock well to 1 protein and not to another protein as was done here [6].
* It is true that metrics like SA score are imperfect to predict synthesizability but it can be well correlated [7, 8].
* Taking Tanimoto similarity to known molecules does not necessarily mean it is easier to synthesize. Synthesizability is complex and single atom changes can turn something synthesizable to something that is not.

### **Similarity Metrics (including Virtual Screening)**
* If a metric is introduced that rewards similarity to known molecules, then the optimisation objective could simply be similarity to these molecules. This has also been shown explicitly in [9] where the model generates molecules similar to a known active while getting good docking scores. In my opinion, we do not necessarily want to re-discover known molecules in the sense of generating similar molecules to a set of reference ligands. This is already done in distribution learning/transfer learning approaches that were detailed in this review [10]. The alternative and more general goal is to design molecules with arbitrary property profiles. In this case, the degree to which optimisation is successful is the most relevant metric. 
* Similarity to FDA-approved drugs may not be completely justified. Drugs are specific and drugs targeting 2 different proteins can be vastly different in structure. In Table 3, the authors also show that even the reference ligand have low similarity to the FDA-approved drugs.
* The problem with Vina scores that the authors elude to is that it is too crude. A very well known problem of docking algorithms is that they cannot reproduce experimental binding poses. Specifically, re-docking is a common task used to validate docking. In cheminformatics literature, this is known as sampling power [11]. Similarly, the ability to rank affinity with docking scores is known as scoring power.
* In ML literature, an equivalent is the assessment of how many binding poses are within some RMSD threshold, for example in the DiffDock paper [12]. If the docking cannot reproduce known experimental data, it is unclear the level of confidence to trust the output. The reason for bringing this up is because if there is a docking algorithm that can capture the binding interactions properly, designing molecules optimised for docking score can implicitly capture similarity. This was explicitly shown in cheminformatics literature [13] (although I still believe similarity to known molecules is not necessarily the best metric and can be hacked). The idea of having a better docking algorithm equating to better results is also shown by the authors in Table 1: comparing Glide with AutoDock Vina and Glide gets better enrichment. Then if Glide were optimised instead, the generated molecules would be better. I understand that this is not commonly done because Glide is proprietary software.

While I believe the sentiment of the authors is that better metrics can better reflect real impact, metrics do not replace wet-lab experiments. This line should not be bolded to give the false impression that predictive methods are close to accurately predicting wet-lab experiment. It would be better to re-phrase this line to convey that better metrics can better inform future model development that may improve wet-lab outcomes.

### **References**
[1] 3D SBDD Benchmark: https://chemrxiv.org/engage/chemrxiv/article-details/66bb0911a4e53c48763ac057

[2] Reinforced GA: https://arxiv.org/abs/2211.16508

[3] TacoGFN: https://arxiv.org/abs/2310.03223

[4] GEAM: https://arxiv.org/abs/2310.00841

[5] Saturn: https://arxiv.org/abs/2405.17066

[6] Negative Design: https://iclr.cc/virtual/2023/12911

[7] SA Correlation 1: https://jcheminf.biomedcentral.com/articles/10.1186/s13321-023-00678-z

[8] SA Correlation 2: https://arxiv.org/abs/2407.12186v1

[9] Similarity reward: https://www.nature.com/articles/s42256-022-00494-4

[10] Generative Design Review: https://www.nature.com/articles/s42256-024-00843-5

[11] Sampling and Scoring Power: https://pubs.rsc.org/en/content/articlelanding/2016/cp/c6cp01555g

[12] DiffDock: https://arxiv.org/abs/2210.01776

[13] DockStream: https://jcheminf.biomedcentral.com/articles/10.1186/s13321-021-00563-7
Yes
No ethics review needed.
1. The proposed benchmark relied on comparing similarity with known drug molecules, does this affect the search for de novo drug molecules?
2. Could you show the docking pose images of molecules generated by MolCRAFT and Pocket2Mol exhibit good similarity to one of the known active compounds
1. Provide a novel benchmark to tackle a curial problem in this area
2. Select a diverse set of models on SBDD
This paper points out a curial problem in current SBDD models evaluation: The vina docking score metric is easy to inflate and other metrics are hard to test in wet lab. So the paper proposed three-level evaluation metrics: Similarity to Known Drugs and Actives, Virtual Screening Ability and Binding Affinity Estimation. They test on five models and claim that MolCRAFT outperforms other methods
1. Focus on testing deep learning models. Could test more on other based such as RL based or Genetics Algorithm based
2. Table 1's description is too simplified, hard to let people to understand why using real molecules as template is better
3. It would be better to briefly describe some terminology, i.e. FLAPP
Yes
No ethics review needed.
1. how well does this framework perform when applied to novel targets for which there may be no closely related compounds available?

2. Are there any risks associated with overfitting to the new evaluation metrics?
1. very detailed and well-explained in most of the sections.

2.  The propose a more practical framework for evaluating SBDD models, which can be useful.
The paper provides a comprehensive framework for evaluating structure-based drug design models, using metrics like virtual screening and similarity to known drugs.

The paper is well written. However, the paper has some limitations. Although it acknowledges the difficulty of synthesizing molecules derived from SBDD models, it lacks a systematic approach. A heavy reliance on virtual screening-based metrics can be misleading without experimental validation. For the paper to be more useful and applicable, more real-world case studies should be included.
1. While the paper acknowledges the difficulty of synthesizing molecules generated by SBDD models, it does not present a systematic approach for addressing this problem.

2. In this paper, virtual screening-based metrics are heavily employed to determine the practical usefulness of generated molecules. While virtual screening is helpful, it could still be misleading if it is not validated with experimental data.

3. This paper needs to include more real-world case studies to be more valuable.
@inproceedings{
anonymous2024reframing,
title={Reframing Structure-Based Drug Design Model Evaluation via Metrics Correlated to Practical Needs},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=RyWypcIMiE},
note={under review}
}
Recent advances in structure-based drug design (SBDD) have produced surprising results, with models often generating molecules that achieve better Vina docking scores than actual ligands. However, these overly optimistic results are questionable due to the known inaccuracies of docking scores and the difficulty of evaluating these molecules in wet-lab settings. Despite showing good QED (drug-likeness) and SA (synthetic ability) scores, these molecules are often poorly drug-like or not synthesizable. To address these issues, rather than relying solely on those heuristic-based theoretical estimation metrics that has poor correlation with actual wet lab success, we evaluate SBDD models from a practical perspective aligned with real-world needs. Inspired by recent research demonstrating the value of generated molecules through ligand-based virtual screening, our evaluation metrics simulate the success rate of wet-lab assessments without requiring time-consuming, costly, and expertise-dependent processes. These metrics assess the ability of generated molecules to effectively retrieve active compounds from a chemical library via similarity-based searches, offering a more direct indication of therapeutic potential. Our experiments reveal that while SBDD models may excel on theoretical metrics like Vina scores, they often fall short in practical applications, highlighting a gap between theoretical performance and real-world utility. By introducing these new metrics, we aim to make SBDD models more relevant and impactful for pharmaceutical research and development.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Stucture-Based Drug Design Model Evaluation Benchmark
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/72ed99762e7b9cf6814d15811907d065f374fa4f.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reframing Structure-Based Drug Design Model Evaluation via Metrics Correlated to Practical Needs
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission