Molecular Active Learning: How can LLMs Help?
Active learning;Bayesian optimization;large language model
Many thanks, Reviewer `jRYu`, for your detailed, thorough, and critical feedback. We see that our manuscript can be significantly overhauled and improved based on your suggestions.

# Clarification: GNNs are trained from scratch

> It appears that the GCN and GAT models were trained on each of the full tasks

The GNN models do not have access to the tasks---they are only trained on the acquisition trajectory so far. In other words, at the acquisition step $N$, their training set size is $N$, with no access to the rest of the unseen data. We will make this clearer in the manuscript. 

> Can you expand exactly what the motivation behind creating the artificial target as such is necessary?

Firstly, because of the data leakage issues mentioned in Yu et al. 2024, the artificial targets would enable a fairer comparison. Secondly, we demonstrate that DPO can be trained on the acquisition trajectories of these artificial targets and perform on real-world datasets.

# Contribution: Thorough comparison with first-principle graph-based models

> Ramos et al. (2023) and Kristiadi et al. (2024) have extensively demonstrated this already

Our paper is significantly distinct from these two papers, discussed in the Related Works section, in the following contributions:
- None of them compared LLM active learning efficiencies with a first-principle baseline such as a GNN+GP.
- We provide a thorough comparison among ways to incorporate LLMs whereas they focus on *one* way to incorporate LLMs in Bayesian active learning, Kristiadi et al. using it as representations and Ramos et al. as only uncertainty quantification. 

# Plan: Software package with better usability & more LLM variants

> While COLT was proposed as a software library, the code is not well-documented and is not installable.

We will improve the usability of this library during the discussion period and provide documentation and installation instructions.

> I have general concerns about the LLMs used in this paper, and how they were used.

Many thanks for providing ideas on other LLMs. We will explore more variants according to your suggestions.
Plan: Software package with better usability & more LLM variants
Thank you, Reviewer `newY`, for your thoughtful and detailed review, and for highlighting the clarity and motivation of our manuscript. We are also grateful to your critiques, based upon which we plan to significantly improve our paper.

# Plan: Revised experiments & more experimental details.

> The selected optimization tasks may not be adequate to rigorously test optimization algorithms.

> that are potentially well-reflected in the training data of LLMs

We recognize that this is indeed a limitations of the datasets, hence we have synthesized artificial datasets to accompany the real-world datasets. In our revision, we plan to emphasize the synthesized datasets more as opposed to the smaller, real-world datasets.

> The optimization performance is measured by cumulative regret. However, this does not necessarily align with the primary goal of molecular discovery, which is to identify the optimal candidate(s).

We plan to also incorporate the rounds of acquisitions required to reach the best candidate as another metric. If you have other suggestions, we would make sure to incorporate those as well.

> There is a lack of discussion on the statistical significance of findings and trends. Confidence intervals are omitted from the figures in Table 2.

The experiments are inherently of high variance since they all start from randomly selected candidates. We will think of more rigorous ways to characterize the difference among trials. 

> The Direct Preference Optimization strategy lacks sufficient detail. How are the GNN–GP reference trials generated? Are they generated on the same optimization problem? If so, a direct comparison to other optimization strategies would be flawed. The need for initial GNN–GP-guided experiments to fine-tune the LLM, followed by a second campaign, would greatly increase experimental costs. In this scenario, all metrics should be compared over all performed experiments.

We plan to provide more experimental details in the manuscript. Specifically, the reference GNN-GP trials are generated on synthetic datasets and tested on real-world datasets. We will make this clearer in the manuscript.

> The authors use Deep Kernel Learning as a domain model. Comparison with simpler domain-specific baselines, (e.g. fixed representations with a GP surrogates, see ref. 48), would place the findings in a broader context

We appreciate this suggestion. We will include simpler baselines in our revised experiments.
Plan: Revised experiments & more experimental details.
Many thanks for your response, Reviewer `poaA`. 

> [Fine-tuning on training data] is the most natural baseline to add in my opinion.

After some thought, I believe that this is indeed a meaningful baseline to add. We will include a comparison with fine-tuning on training data.

> In this regime, there is probably massive overfitting and the results would depend heavily on regularization techniques like weight decay and early stopping.

We believe that GP is already an effective way to deal with small data and uncertainties, but the GNNs used here are uniformly small and regularized, which we plan to elucidate more in the revised experimental detail section. Inspired by your suggestion, we plan to also compare with a pre-trained GNN model.
Fine-tuning in a supervised manner & comparing with pre-trained GNNs.
First, thanks for posting your rebuttal plan early: I've previously had negative experiences where authors perform a difficult set of experiments that don't really address the reviewer concerns, then get frustrated when reviewers don't change their scores. Posting early helps avoid such outcomes.

Responding to your points:

- Fine-tuning: I agree ChemLLM and galactica were fine-tuned in a supervised way, but I think this is different than _fine-tuning directly on your training data_. The tasks they fine-tuned on may not be very relevant to your task. I don't necessarily think you need to add this as a baseline, but it is the most natural baseline to add in my opinion. I would be open to increasing my score further if you added it.
- UCB acquisition function: add this if you want, but it doesn't seem critical to me (don't expect much of a score change from adding this)
- Training data: this might be a more important concern than you realized based on your short response. Since you start from no data and acquire points sequentially, it seems like you are training very large models (eg DKL or GCN) on tiny numbers of data points (<10). In this regime, there is probably massive overfitting and the results would depend heavily on regularization techniques like weight decay and early stopping. Is this something you have deliberately tuned in any way? It seems like something that should _hugely_ impact performance.
Reply to rebuttal plan
Thank you, Reviewer `tX3V`, for your detailed feedback. During the discussion period, we plan to significantly improve our manuscript based on your suggestions.

# Question: Targets for active learning

> The major goal of using active learning is to screen for hit.

> Therefore, a docking score would be a better option than labels in MoleculeNet.

We would appreciate if you could kindly provide more evidence on the statement that hit-identification, rather than hit-to-lead optimization, is the primary use case of active learning in a drug discovery setting. Moreover, docking score is known to be extremely noisy. So we would be grateful if you could provide more clarification on why would that be a better target than the experimental measurements documented in MoleculeNet.
# Plan: More comprehensive literature review, more acquisition functions
> Literature review is not comprehensive.

Many thanks for recommending new active learning papers. We will make sure to incorporate these in our literature review.

> Pharmaceutical companies are definitely trying to perform virtual screening on libraries much larger than million-scale. A simple example of such large libraries is the Enamine database.

Although _virtual_ screening has been routinely done on such databases, to the best of our knowledge, the biggest libraries for wet lab-based high-throughput screening is on the scale of millions of compounds. We will clarify this in our revised paper.

> More acquisition functions should be used as baseline

We will include greedy and UCB as our acquisition functions and compare with existing results.

# Clarification

> I am curious about the format of output when LLM is used as a feature extractor.

The intermediate representation of LLM is used and a fixed-dimensional vector is produced as output.

> What are the y-axis label and unit of the figure above Table-1?

It is the normalized (unitless) measurement, as documented in the legend of Table 1. We will make this clearer in our revision.
Plan: More comprehensive literature review, more acquisition functions
Dear Reviewer `poaA`, 

Thank you for your thoughtful and constructive feedback. We especially appreciate your encouragement for us to publish this interesting (primarily) negative results. We plan to significantly improve the clarity of our paper by incorporating your suggestions.

# Clarification: Fine-tuning on acquisition trajectories

> The study does not address a question which is perhaps more relevant: how will LLMs perform _after some fine-tuning_. I

Our **Section 6, Direct Preference Optimization** provides a tuning recipe, albeit directly using the acquisition steps. In other words, we fine-tune the LLMs to _acquire_ like GPs. We argue that this provides a robust and performant recipe of fine-tuning, and traditional supervised fine-tuning has already been done in the `ChemLLM` and `galatica` models, which show no improved performance in acquisition efficiencies.

# Plan: UCB acquisition function, clearer experimental details

Following your suggestions, during the discussion period, we plan to repeat our experiments with UCB acquisition function. We also plan to clarify more experimental details: 

> - How is the GCN trained, given that you start from no data?

The GCN is trained on all of the data point insofar acquired in the portfolio. Every time a new data point is added, the GCN training is continued.

> - How are GP hyperparameters set? Optimization performance is very sensitive to these parameters.

Following the deep kernel learning formulation, the GP hyperparameters are jointly optimized together with the GNN parameters.
Plan: UCB acquisition function, clearer experimental details
Dear Reviewer `TxKs`, many thanks for your detailed and constructive feedback, and for highlighting that our paper is well-organized and experimental results comprehensive.

Please see here some clarification regarding your critiques and questions, which we will soon incorporate into the paper to improve clarity and readability.

# Clarification: Significance of active learning (AL) in drug discovery

>Can the authors tell if this is a standardized and widely studied task?

> However, my concern is that if active learning based drug discovery is not a widely researched topic, the significance or applicability of this paper will be diminished.

The overarching goal of the machine learning-assisted drug discovery is to improve the efficiency of decision-making, to find the most potent molecules within the fewest rounds of acquisitions, reducing the source requirements.
Traditionally, in drug discovery, _supervised_ learning is used more ubiquitously, with the underlying assumption that one can use the predictions given by these models in a greedy manner to guide the search.
In recent years, active learning approaches [Old references 37, 38; New references 1, 2] have been more frequently applied to molecular tasks. Leveraging uncertainty quantification to balance exploration and exploitation, active learning more directly optimizes the efficiency of acquisition and decision-making.

One reason active learning is not being more popular in drug discovery is the lack of standard benchmarking testbeds and toolboxes, which is what COLT plans to address.

> It seems that ChemLLM is not better than LLaMA, which is counterintuitive for chemistry-related tasks. Can the authors explain why?

We believe that LLaMA is just a more capable foundation model that demonstrates more versatile utility in a wide range of tasks. We will design more thorough benchmarks to compare them.

# Plan: More thorough introduction

- In the discussion phase, we plan to include a more thorough introduction to motivate the design of COLT and the benchmark experiments. 


# References

[1] Konstantin Gubaev, Evgeny V. Podryabinkin, Alexander V. Shapeev; Machine learning of molecular properties: Locality and active learning. __J. Chem. Phys.__ 28 June 2018; 148 (24): 241727. [https://doi.org/10.1063/1.5005095](https://doi.org/10.1063/1.5005095)

[2] Chemical Space Exploration with Active Learning and Alchemical Free Energies. Yuriy Khalak, Gary Tresadern, David F. Hahn, Bert L. de Groot, and Vytautas Gapsys. Journal of Chemical Theory and Computation 2022 18 (10), 6259-6270 DOI: 10.1021/acs.jctc.2c00752
Plan: More thorough introduction to motivate active learning in drug discovery
Yes
No ethics review needed.
Maybe I miss something, but why "1.61" in Table 1 is bolded not "1.29"? Can the authors provide more details?
1. The paper is well organized and written. Although the fields of active learning and AIDD do not naturally overlap, readers can quickly get their key points.

2. The proposed method is reasonable, and the three parts of active learning are indeed expected to be enhanced by introducing LLMs. In particular, the author provides code demonstrations. Intuitively, the method does not have significant loopholes.

3. The experimental results are comprehensive and the conclusions are reasonable. The proposed remedial solution seems to maximize the effectiveness of LLMs.
This paper apply the active learning pipeline for drug discovery. By continuously selecting molecules from the pool to query their properties, high-quality molecules are obtained. Under this setup, three processes called GNN, VI and GP all have the potential to be replaced with LLMs and enhance the whole active learning performance. Authors explore the problems of whether, why and how? Finally, they propose a new tuning recipe that can be effective with LLMs for the drug discovery task.
1. I am not very familiar with this specific task. Can the authors tell if this is a standardized and widely studied task? Can they provide some baseline methods for active learning-based drug discovery?

2. By reading this paper, I can understand the purpose of the task and the approach of the active learning part. And I think the author gives the conclusions related to LLMs, which is encouraged. However, my concern is that if active learning based drug discovery is not a widely researched topic, the significance or applicability of this paper will be diminished. For example, if researchers often use other means to complete drug discovery tasks and rarely use active learning (AL), then for this community, constructing an AL-based baseline is the primary goal. Because AL for drug discovery itself may have challenges worth exploring, which may be skipped and ignored. So, in short, I am not denying the novelty and experimental integrity of the research in this article. I just doubt whether this article will be of great value to this community.

3. It seems that ChemLLM is not better than LLaMA, which is counterintuitive for chemistry-related tasks. Can the authors explain why?
Yes
No ethics review needed.
1. Experimental details:
  - How is the GCN trained, given that you start from no data?
  - How are GP hyperparameters set? Optimization performance is very sensitive to these parameters.
2. On line 370 you write: "require by magnitude more time". Is this a typo? Maybe you be "require an order of magnitude more time"
3. In table 1, what are the ± values? Standard error? Standard deviation? Something else?
- Scientific honesty: this paper asks an important question, finds that the less exciting "null hypothesis" is true, and still reports that. With almost every other paper at ICLR being a "use my method" paper, I found this refreshing
- The question asked is timely and important
- Experiments ask good questions (e.g. investigating domain-specific models and synthetic tasks after initially seeing poor results)
- Experimental methodology seems valid
- Paper is upfront about the limitations of its study
- Presentation of the paper is very nice
This paper presents what is effectively an ablation study between optimizing molecules with LLMs vs Bayesian optimization, finding that LLMs don't help very much, although fine-tuning does help.
- Some important experimental details unclear (see "questions" section)
- The study does not address a question which is perhaps more relevant: how will LLMs perform _after some fine-tuning_. I think the paper should list this as a limitation, or potentially change the conclusion to "fine-tuning will probably be required to get strong performance"
- Some descriptions of BO seem slightly incorrect.
  - Learning a representation of a graph is an _optional_ step in BO, since one could use a model like a GP with a graph kernel that does not convert the graph into a vector. There are many such kernels supported in libraries like Grakel.
  - DKL is not an approximation to the posterior: it is a model whose exact posterior is analytically tractable.
- PoI acquisition function is not really used much because it is insensitive to the magnitude of the improvement. I recommend reading Chapter 7 of Garnett's 2023 BO textbook for details. I think that using the UCB acquisition function instead would be a more interesting choice.
Yes
No ethics review needed.
* The authors use Deep Kernel Learning as a domain model. Comparison with simpler domain-specific baselines, (e.g. fixed representations with a GP surrogates, see ref. 48), would place the findings in a broader context. What is the value of deep learning in the very-small-data regime? Do the inductive biases in GNNs make any difference?  
* On p. 2, the discussion of the “small data regime” could be clearer. While the number of labeled data points is indeed very small (often only hundreds), the library of possible candidates can be extensive (up to billions, possibly more). Distinguishing these scales would clarify this section of the paper. 
* The paper would benefit from investigating into the origins of the observed performance differences, e.g. through ablation studies or other systematic analyses. Are these differences due to poorly aligned representations, insufficient regression performance, lack of uncertainty calibration, or other factors?
* The study targets a highly topical problem with enormous implications in both academic and industrial settings. 
* The paper, particularly the introduction and background sections, are very well written, making the paper clear and accessible. 
* The LLM modules for molecular representations, molecular property prediction, and decision-making, are mostly well-designed. The design of their empirical evaluations is systematic and carefully crafted. 
* The study effectively conveys an important message: LLMs may not always be suitable replacements for domain-specific models, particularly in those fields where knowledge and data are largely non-textual, and model architectures with strong inductive biases exist.
This paper addresses the challenge of active learning for molecular discovery, i.e. finding optimal molecules from large candidate libraries with minimum experimental evaluations. The authors formalize the active learning workflow into three modules: (1) representing molecules in vector form, (2) an uncertainty-aware regressor for predicting target properties, and (3) acquisition of the next candidate from the full library, balancing exploitation and exploration. The paper investigates whether some or all of these modules can be substituted with large language models to improve optimization efficiency. Empirically, the authors find that, even with domain-specific LLMs, no performance improvements can be observed. Instead, architectures with domain-specific inductive biases (i.e. GNNs for molecules) remain state-of-the-art for optimization.
* The selected optimization tasks may not be adequate to rigorously test optimization algorithms. (a) The studied properties (solubility, lipophilicity) have relatively well-known structure–activity relationships, that expert chemists could predict, and that are potentially well-reflected in the training data of LLMs. This contrasts with real-life applications like drug discovery, where the structure–activity relationships are much harder to predict – for both expert chemists and ML models. (b) The candidate pools are small and biased.
* The optimization performance is measured by cumulative regret. However, this does not necessarily align with the primary goal of molecular discovery, which is to identify the optimal candidate(s). Alternative metrics that better capture this objective should be considered, and it would be interesting to analyze whether these metrics consistently correlate with cumulative regret. 
* There is a lack of discussion on the statistical significance of findings and trends. Confidence intervals are omitted from the figures in Table 2. Given the large standard deviations in Table 1, it is unclear which of the discussed trends are actually significant. This should be addressed in the discussion. In my opinion, a generic sentence in the “Limitations” paragraph is not sufficient. 
* The Direct Preference Optimization strategy lacks sufficient detail. How are the GNN–GP reference trials generated? Are they generated on the same optimization problem? If so, a direct comparison to other optimization strategies would be flawed. The need for initial GNN–GP-guided experiments to fine-tune the LLM, followed by a second campaign, would greatly increase experimental costs. In this scenario, all metrics should be compared over all performed experiments. Does the second, LLM-guided identify find any “good” candidate molecules that were not found in the initial GNN–GP trial? While I see that this strategy could be of value e.g. in a transfer learning setting or a multi-fidelity scenario, this paper does neither evaluate nor discuss these aspects.