Ontology-Retrieval Augmented Generation for Scientific Discovery
ontology;rag;retrieval;llm;science;ai4science;chemistry;biomedical;reasoning
Yes
No ethics review needed.
Q1: How differ of your proposed OntoRAG  and OntoGen pipeline  when comparing with the existing DRAGON-AI (https://arxiv.org/abs/2312.10904) and LLMs4OL (https://link.springer.com/chapter/10.1007/978-3-031-47240-4_22)?

Q2: Can you provide the details of your verification process and manual effort for LLM-generated ontologies that you mentioned in Lines 340-341 and 363-364?  
Because the LLMs-generated ontologies cannot utilized to enhance the LLMs directly without the human’s curation, since the hallucinations still exist when generating the ontologies with LLMs, the manual validator is needed. 

Q3: Can you provide a detailed analysis of the results that are reported in Table 2 and Table 3 and highlight how much the OntoGen and OntoRAG contribute to the final results?
S1. The paper is well-written and organized, and the methodology of OntoRAG is well-designed and demonstrated. 

S2. The OntoGen pipeline is created to generate the ontologies based on multiple calling the long-context LLMs.

S3. The experiments on the SACBench benchmark and biomedical QA benchmark are conducted to evaluate the performance of OntoRAG and the quality of LLM-generated ontologies.
This paper presents an OntoRAG that leverages the LLMs generated ontologies as the context to enhance the RAG by retrieving taxonomical knowledge from context for accelerating scientific discovery.  The results on the SACBench benchmark demonstrate that OntoRAG outperforms the CoT-based RAG on accuracy, completeness, and order. Additionally, the quality of the ontologies generated by LLMs is evaluated by the downstream task on the biomedical QA benchmark.  

The paper is well-written and organized, but the pipeline of ontology generation (OntoGen) and RAG with ontologies (OntoRAG) is not a novel contribution, as the existing methods have already investigated it. 

My main concern is that the ontologies generated by LLMs cannot be utilized to enhance the LLMs directly without the human’s curation since hallucinations remain when generating the ontologies with LLMs.
W1. The presented OntoGen and OntoRAG pipeline is not novel, as the existing works have already been investigated but they are missed in related works. (Details in Q1)

W2. The ontologies generated by LLMs are utilized directly as context for RAG without the human’s validations. (Details in Q2)

W3. The source code and claimed SACBench benchmark dataset are not provided for reproducibility.

W4. The readability of this paper needs to be improved, as some results analyses and the ablation study are missed. (Details in Q3)

W5. Some typos need to be fixed and avoided. For example, the parentheses after “axioms” should be removed “axioms()-> axioms” in line 289-290, the comma after “in” should be removed “in. order to-> in order to”  in line 408, the “ACcuracy->Accuracy” in line 916, etc.
Yes
n/a
No ethics review needed.
**Suggestion/Typo**:

- In Table 1, the word “typo” appears to be a typo itself and may need correction.
- I recommend a careful review of formal definitions throughout the paper. For established concepts like "ontology," it would be beneficial to reference widely accepted definitions, such as those based on description logic. For processes like RAG, ensure the level of abstraction aligns with practical implementations. The current definition assumes independent and automatic weighting of each retrieved document, which is not universally applicable in RAG and oversimplifies the underlying mechanics.
- This work is well-motivated by the need in many scientific domains for expert-curated knowledge that goes beyond document-level retrieval. It seeks to a new RAG pipeline by integrating ontologies, which are widely adopted knowledge bases for specific domains.

- The pipeline addresses cases where ontologies are unavailable, proposing an automated approach to ontology construction from documents.
This paper introduces an ontology-based retrieval-augmented generation (RAG) pipeline designed to enhance scientific discovery by integrating ontology-based knowledge with language models. Additionally, the paper presents OntoGen, an automated ontology-generation method for fields where no ontology exists, extending OntoRAG's applicability to emerging domains. The proposed method is mainly evaluated on biomedical QA and catalyst synthesis benchmarks.
- Definition 2.1 for ontology requires a signficant revision as it is unclear and contains inaccuracies:
  - An ontology can be described as a set of logical axioms that define relationships among entities (concepts, properties, and instance) in the ontology. This approach avoids separating axioms from relationships, as relationships should not be limited to triples alone.
  - It seems that the relationships set $\mathcal{R}$ refer to object properties, while the properties set $\mathcal{P}$ appears to denote data properties.
  - Additionally, the notation $\forall i \in \mathcal{I} \exists c \mid c \in \mathcal{C}$ needs clearer explanation. If the intent is to express that an instance $x$ belongs to a class $C$, it would be more accurate to write $C(x)$ or $x: C$.

- Definition of RAG in Equation (1) is inaccurate: As stated, this definition implies that each retrieved document influences the generation probability and is weighted by its relevance. However, in a standard (vanilla) RAG setting, this is not the case; only a subset of retrieved documents typically impacts the generation process, without automatic weighting by relevance.

- The OntoRAG definition needs a significant revision since it builds on the earlier definitions of ontology and RAG, which contain inaccuracies.

- The OntoRAG methodology section lacks sufficient detail for reproduction. To enhance clarity, it would be helpful to include step-by-step explanations of the methodology components and provide running examples.

- The main evaluation in Table 1 primarily examines variations of OntoRAG and one Chain-of-Thought (CoT) baseline. However, it overlooks important comparisons with existing GraphRAG approaches, which similarly aim to incorporate graphs and knowledge bases within the RAG framework.
Yes
No ethics review needed.
• Can authors add additional benchmarks that show improvement in performance?
• How does OntoRAG handle conflicting information from different retrieved sources within a single ontology or between multiple ontologies?
• How does the quality of automatically generated ontologies compare to expert-curated ones in established fields? 
• How sensitive is the performance of OntoRAG to the specific choices made in the ontology retrieval and fusion steps?
• Novel approach: Combining ontologies with RAG is an innovative idea to enhance LLM performance in specialized scientific domains. 
• Automatic ontology generation: OntoGen addresses a key bottleneck by automating the creation of ontologies for emerging fields. 
• Application in Emerging Domains: The case study in Single-Atom Catalysis demonstrates the potential of the approach to aid scientific progress in cutting-edge fields where ontologies are not yet fully established.
• Reduction of Hallucinations: OntoRAG addresses a critical problem in LLMs - factual inaccuracies - by grounding the outputs in established scientific relationships and concepts.
This paper introduces OntoRAG, a novel approach that enhances Retrieval Augmented Generation (RAG) by incorporating ontological knowledge to improve the accuracy and scientific grounding of large language models (LLMs). The authors also present OntoGen, a tool for automatic ontology generation to extend OntoRAG's utility to fields without pre-existing ontologies. The key contributions are:
    • OntoRAG: An extension of RAG that retrieves and integrates relevant ontological information to improve reasoning and reduce hallucinations in large language models (LLMs).
    • OntoGen: An LLM-based pipeline for automatically constructing domain-specific ontologies from scientific papers.
    • SACBench: A benchmark for evaluating the synthesis of single-atom catalysts (SACs), used to test the OntoRAG approach in an emerging scientific domain.
The authors evaluate OntoRAG on standard biomedical benchmarks and the novel of Single-Atom Catalysis SACBench. Results show improvements over baseline RAG in some domains, reduction of hallucinations in LLMs, particularly for the SAC synthesis task.
• Weak evaluation: The authors test their approach on established benchmarks without significant improvement and a novel task in an emerging field that shows promise. Nevertheless, its only one benchmark in a specific field that shows some results. 
• Limited Improvement in Aggregate Performance: Despite the benefits of ontology integration, the paper notes that the aggregate improvement across benchmarks is modest, suggesting that the effectiveness of OntoRAG depends on the specific domain.
• Ontology quality assessment: The paper lacks a thorough evaluation of the quality of automatically generated ontologies beyond downstream task performance.
• Computational Overhead: The process of ontology generation and integration adds complexity and computational cost to the pipeline, which may limit its practical use in certain scenarios.
• Expert Dependency: While OntoGen attempts to automate ontology creation, the variability between LLMs and the need for manual curation still imply a dependence on human expertise for high-quality outputs.
Yes
No ethics review needed.
1. Have the authors conducted any further investigations into the results of Section 5? Specifically, the authors speculate that the results are due to discrepancies in vocabulary; have they confirmed that? 

2. What is the verification process "to ensure fidelity to the source text" described in Section 4.1? Can the authors specifically describe the details of (or cite, if using another approach) this verification process?

3. Furthermore, how are discrepancies handled when the above verification process encounters them? Can the authors give a specific example of how a discrepancy might be handled?

4. What self-consistency techniques were used in Section 4.2? Can the authors give specific details or citations for these techniques?
- Novelty: The authors addressed a novel, interdisciplinary area between AI and the natural sciences.  

- Clarity: The authors have done a great job of explaining the necessary background in a concise way. I commend the authors for acknowledging the significance and challenges which go along with applying AI approaches to scientific domains, in which plausibility is not necessarily the same as scientific accuracy. 

- Creativity: The ideas in this paper are creative: the authors have found a unique way to address ongoing concerns surrounding LLM hallucinations, particularly within a scientific context. 

- Reproducibility: The paper is generally well-written and easy to read. For the most part, the paper was clear, and experiments seem reproducible based on the details given.
In this study, the authors design and use an automatic ontology generator, OntoGen, to create ontologies for specialized domains in which there are no pre-existing ontologies. Then, they incorporate the generated ontology from OntoGen as input into OntoRAG, a retrieval-augmented system for LLMs. The authors aim to create a system which produces more accurate scientific output than LLMs or RAG systems. They first test OntoRAG without OntoGen, using biomedical ontologies in its place, for biomedical prediction tasks. Thereafter, they evaluate OntoRAG + OntoGen on a materials science application (single atom catalyst (SAC) synthesis) for which they designed a novel benchmark dataset. OntoRAG performs consistently better than a baseline RAG system for SAC synthesis. The novelty of this paper lies in (1) the design of an automatic ontology generator, OntoGen, (2) the development of a RAG system which incorporates ontologies as input, OntoRAG, and (3) the creation of a benchmark dataset, SACBench, for assessing LLM output within the context of a specialized, materials science domain.
I have two major concerns regarding gaps in the evaluation processes. These concerns make it difficult to confirm the scientific rigor of this study: 

1. The authors should evaluate the capabilities of OntoGen against existing scientific ontologies, like the Gene Ontology. This will allow the reader to assess whether OntoGen can really produce ontologies that capture scientifically significant patterns. The authors could accomplish this through a comparison of the metrics reported in Fig. 3 or through metrics such as concept coverage, structural similarity, or expert evaluation of key relationships.

2. The experimental results in Figure 4 should also include the base LLMs, without any augmentation, as baselines. Specifically, the authors should report the performance metrics of the base LLMs on SACBench using the same criteria as OntoRAG. This will allow the reader to assess whether OntoRAG truly improves upon LLM accuracy within specialized, scientific domains. 

**Expansions:**

1 (expansion): I do not think the authors sufficiently evaluated the capabilities of OntoGen before moving on to evaluate OntoRAG. Since OntoRAG on the SACBench dataset relies upon the output of OntoGen, it is necessary to ensure that OntoGen can produce ontologies with qualities consistent to established ones. Specifically, the authors should compare the output of OntoGen to an existing ontology. While there is no existing ontology for SAC, the authors acknowledge in Section 2.1 that other curated ontologies exist for other domains, like genetic or biomedical ones. For example, the authors could use OntoGen on a corpus of genetic literature and compare the generated ontology to the Gene Ontology. 

2 (expansion): The experimental results given in Figure 4 are missing a key baseline: the base LLMs without any RAG system. The authors should include this baseline as it is critical to assess one of the aims of the paper ("enhancing the scientific accuracy of LLM outputs"). Additionally, this baseline is particularly important in light of the results of Section 5, in which the OntoRAG system performed worse than the base LLMs in a majority (6/10) of cases (based on the metrics reported Appendix A.0.1). The results of Section 5 (Appendix A.0.1) call to question why the authors decided to move on with OntoRAG + OntoGen. It appears that OntoRAG with pre-existing ontologies has no improvement or limited improvement over the base LLMs. If OntoRAG with established ontologies offers no substantial improvement, then the authors should clarify why they believe that OntoRAG + OntoGen will offer improvements. Specifically, the authors may be able to justify the use of OntoRAG + OntoGen by including the performances of base LLMs on SACBench.
We propose methods for generating ontologies, and doing RAG by retrieving context from relevant Ontologies, which reduces hallucinations.
@inproceedings{
anonymous2024ontologyretrieval,
title={Ontology-Retrieval Augmented Generation for Scientific Discovery},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=DbZDbg2z9q},
note={under review}
}
Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks, sparkling an increasing interest for their application in science. However, in scientific domains, their utility is often limited by hallucinations that violate established relationships between concepts or ignore their meaning; problems that are not entirely eliminated with Retrieval Augmented Generation (RAG) techniques. A key feature of science is the use of niche concepts, abbreviations and implicit relationships, which may deem RAG approaches less powerful due to the lack of understanding of concepts, especially in emerging and less known fields. Ontologies, as structured frameworks for organizing knowledge and establishing relationships between concepts, offer a potential solution to this challenge. In this work we introduce OntoRAG, a novel approach that enhances RAG by retrieving taxonomical knowledge from ontologies. We evaluate the performance of this method on three common biomedical benchmarks. To extend the value of OntoRAG to emerging fields, where ontologies have not yet been developed, we also present OntoGen, a methodology for generating ontologies from a set of documents. We apply the combined OntoGen+OntoRAG pipeline to a novel benchmark of scientific discovery in the emerging field of single-atom catalysis. Our results demonstrate the promise of this method for improving reasoning and suppressing hallucinations in LLMs, potentially accelerating scientific discovery across various domains.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
ontology rag retrieval llm science ai4science chemistry biomedical reasoning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/df338ff3d60720b1df68de533eb5b4e67ad0d7df.pdf
applications to physical sciences (physics, chemistry, biology, etc.)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/56487e5ff7afec4c01586ff3614aea11abfeb67f.pdf
Ontology-Retrieval Augmented Generation for Scientific Discovery
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission