EDU-RAG: A RAG Benchmark with Web-enhanced Content in Education Domain. Can RAG Help AI Tutor?
Large Language Models (LLM);Retrieval Augmented Generation (RAG);Search
Yes
No ethics review needed.
Suggestions:
1. Authors should improve the formatting of the paper. For example, references are not in parenthesis (e.g., line 33, 35, etc.). Similarly, there are several grammatical mistakes in the paper that authors should fix.
1. The motivation behind the research in the education domain is relevant and apt.
2. Authors create a new dataset of MCQA for middle school level science. The dataset is augmented with information obtained via web-searches.
This paper proposes a benchmark dataset for evaluating LLMs in the education domain. Further authors experiment with RAG based approach to reduce hallucinations in LLMs in the context of education.
1. The paper lacks novelty as authors are implementing the standard RAG architecture for the QA task. 
2. The newly created dataset is mainly an extension of an existing TQA dataset augmented with content from the web, so there is very limited innovation.  
3. The paper doesn't report any new findings; the authors show that RAG helps to mitigate hallucinations to some extent but this is already known and established by previous research. 
4. The paper is poorly written with grammar mistakes, typos, poor formatting. A figure (Figure 2) is also missing from the paper, in place of that a blank box appears.
Yes
No ethics review needed.
What are the fundamental differences between this benchmark and previous ones? Or more specifically, what are the fundamental differences in the education domain versus finance etc in Yang 2024?

The prompt says "You must choose a letter even if you are unsure." How would one expect the LLM to output "I don’t know” if they are not instructed to do so? 

What motivates the authors to define hallucination as answer correctness? The way the authors defined hallucination seems more like model's problem solving or reasoning capabilities rather than hallucination.
I like the direction that this paper attempts to go: evaluating and benchmarking RAG systems and LLMs and checking whether they hallucinate is very important. Doing all of these in the context of education is also important. I'm excited to see more work along this direction.
This paper proposes an RAG benchmark in the education domain. The benchmark builds on top of AI2's TQA dataset, augmented with web search results for each question. The paper evaluates a few LLMs on performance on this benchmark where the metric is answer correctness (note that the author defined "hallucination" in terms of answer correctness). The author finds that RAG improves performance of LLMs in answering these questions. The author claims this benchmark will be useful for "advancing the evaluation and development of RAG techniques in mitigating hallucinations across diverse LLMs."
The paper seems to be poorly presented/written. Contributions seem insufficient and unclear.

In terms of presentation:
- Missing figure (figure 2)
- Table 1 and 2 are repeated 
- Not sure what the example question means in Fig 1 (the question is used multiple times in subsequent results and illustrations)

In terms of contribution:
- It is unclear to me what the *fundamental* differences or advancements that the proposed benchmark has compared to prior work. For example, I'm not sure how this benchmark is different from Yang 2024's. Expanding their dataset to a new domain doesn't seem super novel to me unless there are some fundamental differences or challenges which requires innovations in either constructing the benchmark or evaluating on the data, neither of which I find novel in this paper. This is my main concern.
- The research questions seem to be studying well-known conclusions. For example, RQ2 is, roughly speaking "can RAG help improve LLM performance"? I think the answer is YES by now and it is widely known. When reading the abstract, I thought the authors would evaluate how LLMs would perform in the presence of irrelevant information (line 22 ~ 23), which I think would be a deeper and more interesting analysis than RQ2, but unfortunately the authors did not present such analyses or findings. In general, I find the conclusions and analyses in this paper either already known or shallow.
- Some of the suggested future work is already done a few years ago. For example, SFT on retrieved content is explored in this paper back in 2022: https://arxiv.org/pdf/2201.08239 

Given that the paper appears not ready for presentation and the contributions are unclear and not particularly novel (see above for details), I would not recommend acceptance of this paper.
Yes
No ethics review needed.
- Have you considered expanding the scope of evaluation to include more complex question answering scenarios, such as open-ended questions or multi-hop reasoning tasks for educational applications?
- How did you address the potential bias in the web content retrieved using Google Search?
- How did you assess the quality and relevance of the web content retrieved for each question?
Originality: The paper presents a new benchmark dataset, EDU-RAG, specifically tailored for evaluating RAG techniques in the domain of middle-school science question answering. 
Quality: The methodology for constructing the benchmark dataset is explained, including the selection of questions, retrieval of web content, and processing of the data.
Clarity: The authors also provide a comprehensive overview of related work, placing their contributions within the broader context of RAG research.
Significance: The EDU-RAG benchmark has the potential to significantly advance the evaluation and development of RAG techniques for educational applications.
This paper introduces EDU-RAG, a benchmark dataset designed to evaluate the performance of Retrieval-Augmented Generation (RAG) techniques in the context of middle-school science question answering. The dataset combines textbook questions with relevant web search results, addressing the challenge of hallucination in Large Language Models (LLMs) like GPT-4o and Llama2-7b.
- The paper only evaluates a basic RAG algorithm design, consisting of a retriever, reranker, and generator. While this serves as a useful baseline, it does not explore the effectiveness of more advanced RAG techniques, such as modular RAG or advanced reranking methods.
- The paper acknowledges that RAG can reduce hallucination in some cases but also highlights instances where it may worsen the issue. However, the analysis of this phenomenon is limited, and the paper does not delve into the underlying reasons of it.
- Typo: The title of section References appears twice.
- Missing figure: The figure 2 is missing.
@inproceedings{
anonymous2024edurag,
title={{EDU}-{RAG}: A {RAG} Benchmark with Web-enhanced Content in Education Domain. Can {RAG} Help {AI} Tutor?},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=a2rSx6t4EV},
note={under review}
}
Hallucination has been a notable concern in leveraging Large Language Models (LLMs). Retrieval-Augmented Generation (RAG) is a popular method in addressing that and in maintaining context and coherence in generated outputs and including customized knowledge. In this paper, we aim to create an updated benchmark dataset using middle-school science-domain-ed textbook-question-answering, and web search results, and to evaluate the performances of different LLMs incl. GPT-4o, Llama2-7b, Llama3-8b, with and without RAG. Basically, we want to evaluate whether RAG would improve hallucination problem caused by pre-trained biased LLM model knowledge, or drawing from irrelevant retrieved knowledge despite of access to relevant information. The dataset and method can be used to evaluate different RAG methods.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Large Language Models (LLM) Retrieval Augmented Generation (RAG) Search
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/6e23b9f2dcca2436deaf28a4ffe6ad9cce7f649e.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
EDU-RAG: A RAG Benchmark with Web-enhanced Content in Education Domain. Can RAG Help AI Tutor?
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission