Large Language Models for Rediscovering Unseen Chemistry Scientific Hypotheses
scientific discovery
Yes
No ethics review needed.
1. There is a lack of experimental exploration for mutation settings.   I am curious about the impact of mutations on the final results. 
     For example, according to the ablation studies, what proportion of high-quality hypotheses can be obtained directly without mutations?

2. How can we explain that a smaller window size leads to better performance in the inspiration retrieval phase?

3. Did the authors replace GPT-4o for all comparison methods for generation? Please specify.

4. How helpful is it to add new dimensions of significance?
Originality:
Firstly, while LLMs have been utilized for scientific discovery in social science and NLP, this paper is the first to investigate their potential in chemistry. 
Besides, The MOOSE-CHEM framework employs a three-step approach to retrieve inspiration papers, inference valid knowledge, identify hypotheses and rank them, which hasn’t been used in previous research. 
Moreover, the use of the evolutionary algorithm to foster a broader diversity in hypothesis generation is also an innovation point.

Quality:
The paper includes extensive comparative experiments and ablation studies, providing a thorough evaluation of LLMs' performance. It also demonstrates the consistency between expert evaluation and automated evaluation of MS, enhancing the reliability of the results.

Clarity: 
The methodology is well-structured, and the three steps based on smaller questions are well-defined. Besides, detailed evaluation methods and corresponding formulas are provided for the three research questions, Q1, Q2 and Q3.  The practical implementation process is also relatively clear.

Significance:
The ability of LLMs to generate hypotheses can significantly accelerate the pace of scientific discovery by reducing the time and effort required for hypothesis generation. It can also open researchers’ minds for the application of LLMs in scientific discovery.
This paper investigates the potential of LLMs to automatically discover novel and valid hypotheses in the field of chemistry. The study introduces a framework called MOOSE-Chem, which decompose the main research question into three smaller questions, respectively focusing on (1) the identification of inspiration papers; (2) the inference of unknown knowledge that is highly likely to be valid; and (3) the identification and ranking of hypotheses generated by the LLMs.
Firstly, using the same large language model to evaluate its own generated results may introduce bias. It is recommended to try using different LLMs to evaluate the results so as to guarantee the reliability of the results. For example, consider using models like LLaMa[1], Claude[2], Gemini[3], or other recent LLMs to compare outputs. If using the same LLM is necessary, you could collect hypotheses generated by humans and also have both experts and GPT-4 evaluate them. Then, compare their Hard/Soft Consistency Scores as well as distribution of MS with those of the hypotheses generated by the LLM.

[1] Touvron H, Martin L, Stone K, et al. Llama 2: Open foundation and fine-tuned chat models[J]. arXiv preprint arXiv:2307.09288, 2023.
[2] Anthropic. The claude 3 model family: Opus, sonnet, haiku. https://www.anthropic.com, 2024.
[3] Team G, Anil R, Borgeaud S, et al. Gemini: a family of highly capable multimodal models[J]. arXiv preprint arXiv:2312.11805, 2023.

Besides, during the evaluation process (as demonstrated in section 5.1), in the constructed dataset, other papers besides the ground truth inspiration papers may also be helpful for the current background. This step may also require further evaluation to obtain more reliable results. You might select several high-ranking papers that are not included in set I, particularly those that repeatedly appear in experiments with varying corpus sizes or screen window sizes, and conduct case studies on them.

Additionally, this paper only employs the titles and abstracts for retrieval, which might overlook inspirations that could arise from the detailed content of the articles. Sections like Conclusion or Discussion often contain insights into the paper's limitations and suggestions for future work. These elements are usually crucial for generating inspiration.

Moreover, the benchmarks mentioned in the paper contain only 51 references, which may not be sufficient to fully assess the effectiveness of the proposed method. You are advised to increase the number of samples, especially selecting literature from different chemical fields. In addition to traditional categories such as Analytical Chemistry, Organic Chemistry, and Inorganic Chemistry, broader research topics like Environmental Chemistry, Medicinal Chemistry, and Nuclear Chemistry can also be considered. Furthermore, Nature Chemistry includes interdisciplinary areas such as Bioinorganic Chemistry, Bioorganic Chemistry, and Organometallic Chemistry, which could be investigated as well. Of course, this may involve considerations related to the resource consumption of LLMs.

Finally, although the paper used chemistry PhD students for the evaluation, there is still subjectivity that may affect the reliability of the evaluation results. You might consider inviting more experts to participate in the evaluation and analyzing the consistency of the assessments from different human individuals.
Yes
No ethics review needed.
See the weaknesses listed above.
1.	Generating research hypotheses is a complicated task, and the authors heuristically decomposed hypothesis generation into two steps: 

(1). inspiration retrieval, and 

(2). hypothesis refinement. In the hypothesis refinement step, the authors propose a novel “mutate and recombine” trick to help generate good hypotheses.

2.	The experiments to verify each of the research questions are well-designed with good quality.
This work investigates whether large language models (LLMs) can autonomously generate novel and valid hypotheses in chemistry based solely on a research background. Specifically, the study explores if LLMs, when provided with a research question or background survey, can independently identify inspirations, synthesize hypotheses, and evaluate their quality. Building on discussions with chemistry experts, the authors hypothesize that most chemistry hypotheses can be derived from combining research background with relevant inspirations. They break down the central question into three key tasks: retrieving valuable inspirations, generating hypotheses, and ranking the quality of these hypotheses. To test this approach, the authors create a benchmark of 51 high-impact chemistry papers published in 2024. LLMs, trained only on data up to 2023, attempt to rediscover these hypotheses from the background and an extensive chemistry literature corpus. The results, achieved through a multi-agent LLM framework, show promising success, with many rediscovered hypotheses closely matching the originals and capturing core innovations.
1.	The introduction section could be written better and more clear. (a) It would be great if the authors could provide a summary of the major contributions of this work at the end of the introduction section. What are really the contribution to the field? (b) It would be great if the authors could briefly discuss why the decomposition of the major question is necessary, what’s the difference or connection between the proposed inspiration identification (the first step of the three) and Retrieval Augmented Generation (RAG).

2.	As the end goal is to rediscover the chemistry scientific hypotheses, the upper bound of “rediscovery” is the exact match of the original hypothesis, intuitively if we rank the original hypothesis with the generated hypothesis, the original hypothesis may be ranked at the top for most of the time. Experiment on this and analysis about the cases where the original hypothesis is not ranked at the top would be interesting to better reflect LLM’s ability to perform R(h).

3. Although the proposed approach constructed a benchmark as the basis to evaluate the performance, it is unclear to what degree the generated hypotheses can be trusted or reliable, not made up from baseless hallucination.
Yes
No ethics review needed.
This includes a list of minor issues, questions and suggestions:

- Through the paper, it is mentioned that the inspiration shouldn't be related
to the given background. Is this enforced in the method, or is this verified
somehow?

- The proposed method of Evolutionary Algorithm (EA) sounds to be related
to other methods in LLMs that try to improve the consistency/quality of
the answers, e.g. 
  - Self-Consistency (SC) (Wang et al., "Self-Consistency Improves Chain of Thought Reasoning in Language Models")
  - Universal SC (Chen et al. "Universal Self-Consistency for Large Language Model Generation")
  - Multiple Chain of Thought (MCT), Self reflection and/or self-correction approaches, etc. I thus believe that some related work could be included regarding this algorithm in the paper.


- On line 351, claim that "is possible to be true" that LLMs are able to create
novel knowledge. While the authors are correct here since it is not clear
whether this is true or not, could the authors provide other explanations
on why the LLMs are able to correctly find inspiration papers?


- Please if possible provide some description of how P (i|b), P (h|b, i), and R(h)
inputs and outputs look like. From the example provided, I can guess that
hypothesis is one paragraph long, but it would be great for the reader 
to have this stated clearly.

- I honestly don't fully understand the experimental setting reported in section 5.1.
What are the "ground truth inspiration papers" and how are these related to 
the 51 papers from the benchmark? In Table 3, does this mean that when a corpus size
of 150 is used, 120 papers are considered ground truth? 

- From Table 3, it almost seems like corpus size doesn't have much effect on the Hit ratio,
is this correct? If so, could you elaborate more in this regard?

- If I got the idea right, in the results of Table 1, if a screen window size is used, the corpus is 300 and 3 papers are selected for each window, then round 1 has (300/10)*3 = 90, round 2 has
(90/10)*3 = 27, and so on, is this correct? It would be more clarifying to include the amount of papers in each round in the table or in a separate table. It would be also clarifying 
to include a baseline comparison of the performance based on pure random screening (this is, select papers randomly).

- From the results reported in Table 7, it seems that the background survey (almost) doesn't have any 
effect on MS, however, this is somehow counterintuitive. Shouldn't the background survey improve performance as it happens in Q1?

- I would suggest the authors format the rounds in Table 4 to go along
with the direction of the text, this is, from left to right.

- Could you please clarify why there are NAs in Tables 3 and 4?

- On line 134: "contrusted by multiple chemistry PhD students".
For clarity, it would be great to know how many students.

- While it is not mandatory, given the stochasticity and prone-to-hallucinations nature of LLMs,
it is a good practice to use countermeasures to reduce hallucinations, e.g. sampling multiple times with majority voting,
or using different models for comparison. This is not an issue, but it would be
interesting to see how much variability the results reported in the experiments
are or if the method is robust to hallucinations.
The paper is generally well-written, and in good English. The text is clear and
the authors did a good job guiding the reader through the motivation,
the derivation of the method and motivating each of the proposed steps
and experiments. The topic of the paper is very relevant and the results
are positive. Related work is well covered, and experiments are included
that compare the proposed method with previous work. Every claim made 
on the performance of the method is generally backed up with experiments.
I think the paper is generally a great and novel contribution to the field.
This paper introduces a framework for scientific hypothesis discovery
and generation using large language models (LLMs). The authors gathered
a carefully curated dataset of scientific papers where questions and hypotheses were extracted for evaluation. The authors propose "MOOSE-CHEM", a
method based on previous work "MOOSE" but with modifications
to adapt it for chemistry literature and that improve Overall
performance (refinement, mutation). The framework is also formally motivated.
The proposed approach is evaluated in each of the main three tasks
that compose it: P (i|b), P (h|b, i), and R(h). Overall the results show great
performance in the tasks with GPT-4o, with evaluations that include both 
experts and GPT-4o, and where various hyperparameters are tested (e.g. corpus size).
My main concerns with the paper are regarding the reproducibility,
clarity and discussion of the approach:

- Reproducibility: we note that the authors introduce a scientific benchmark,
along with a novel framework for hypothesis generation. However,
the authors do not provide access to the novel-introduced benchmark,
which hampers the ability to really discriminate the difficulty of the tasks
at hand. Additionally, this impedes the ability to reproduce the results or
for future work to compare the performance with the proposed method.
I would suggest the authors to make the benchmark data available if possible.
Similarly, no access to the source code is provided. While available source code is not
a requirement, for a method that relies heavily on LLMs prompting, it would
be very beneficial to at least have access to the prompts used in the
work, just for the sake of reproducibility.

- (Related to the previous issue): There are parts of the paper that lack
a formal definition, which again, hampers the 
ability to reproduce the work. For instance, the authors mention that they
introduce a novel "Evolutionary Algorithm (EA)" that mutates, refines and
recombines hypotheses, however, it is unclear how the mechanisms of
mutation, refinement and recombination are implemented. I would suggest 
the authors to provide, when possible more technical details on the
proposed method.

- While the work includes a fair comparison with previous work, the work
doesn't include any discussion on the limitations of the proposed method,
nor any future work. It would be very much appreciated if the authors
could provide a discussion on the limitations of the proposed method
to guide potential future work in the field.
Yes
No ethics review needed.
1. Can the authors clarify how/whether MOOSE-Chem substantially differs from previous work in multi-agent scientific discovery systems? 
2. How would the authors envision MOOSE-Chem supporting a chemist in hypothesis generation beyond rediscovering existing knowledge? Have any of the generated hypotheses been practically tested, or do they align with ongoing chemistry research needs?
1. The use of LLMs for scientific hypothesis generation is a growing area, and extending this work to complex fields like chemistry holds practical importance if achieved meaningfully.
2. The use of beam search and evolutionary algorithms for hypothesis refinement seems simple and powerful, as it systematically explores multiple possibilities.
3. The framework does not require expensive training or fine-tuning.
This paper explores the potential of Large Language Models (LLMs) to generate novel, valid hypotheses in the chemistry domain by employing a multi-agent framework. It targets the rediscovery of scientific hypotheses using LLMs based on recent high-impact chemistry publications (since 2024), using a framework that separates tasks into three stages: (1) finding "inspiration" papers from a predefined literature corpus, (2) generating hypotheses by associating these inspirations with a background question, and (3) ranking hypotheses based on quality. To solve this, the authors proposed a multi-agent framework existing framework with an evolutionary algorithm to iteratively mutate hypotheses based on reward feedback (validness, novelty, clarity, and significance).
1. There is a key fundamental assumption that the authors did not discuss here: reliance on general-purpose LLMs containing implicit domain knowledge. While this may be reasonable for powerful, large-scale models like GPT-4 or domains like social sciences, it is overly strong and may not apply to smaller or open-source models in this case. A deeper analysis (maybe more experiments) of how this assumption impacts performance across other LLMs (apart from GPT-4 or closed-source ones) is needed.
2. The assumption that LLMs have sufficient knowledge to connect novel and unrecognized scientific relations without any domain-specific tuning seems too bold. The model's effectiveness may degrade in more niche areas or specific research questions.
3. Much of this framework — multi-agent, hypothesis scoring, and the background-inspiration format closely resembles existing work in this area. Given the limited scale of benchmarking and experiments in this paper, the technical contribution seems limited for this venue.
3. The selection of inspiration papers seems unclear and might not align with typical challenges or open research questions in chemistry. A more detailed explanation of the selection process of those 51 papers and corresponding inspiration papers, including how relevance, significance, and novelty are ensured, is needed.
3. The paper presentation could be improved, particularly in the introduction and methodology. The technical details regarding probabilistic inference seem unnecessary.
@inproceedings{
anonymous2024large,
title={Large Language Models for Rediscovering Unseen Chemistry Scientific Hypotheses},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=X9OfMNNepI},
note={under review}
}
Scientific discovery contributes largely to the prosperity of human society, and recent progress shows that LLMs could potentially catalyst the process. However, it is still unclear whether LLMs can discover novel and valid hypotheses in chemistry. In this work, we investigate this main research question: whether LLMs can automatically discover novel and valid chemistry research hypotheses, given only a research question? With extensive discussions with chemistry experts, we adopt the assumption that a majority of chemistry hypotheses can be resulted from a research background question and several inspirations. With this key insight, we break the main question into three smaller fundamental questions. In brief, they are: (1) given a background question, whether LLMs can retrieve good inspirations; (2) with background and inspirations, whether LLMs can lead to hypothesis; and (3) whether LLMs can identify good hypotheses to rank them higher. To investigate these questions, we construct a benchmark consisting of 51 chemistry papers published in Nature or a similar level in 2024 (all papers are only available online since 2024). Every paper is divided by chemistry PhD students into three components: background, inspirations, and hypothesis. The goal is to rediscover the hypothesis given only the background and a large chemistry literature corpus consisting the ground truth inspiration papers, with LLMs trained with data up to 2023. We also develop an LLM-based multi-agent framework that leverages the assumption, consisting of three stages reflecting the more smaller questions. The proposed method can rediscover many hypotheses with very high similarity with the ground truth ones, covering the main innovations.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
scientific discovery
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/cfdf0ed243741fd0d6741d445efdc84f75dcfa83.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Large Language Models for Rediscovering Unseen Chemistry Scientific Hypotheses
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission