RAGGED: Towards Informed Design of Retrieval Augmented Generation Systems
Retrieval Augmented Generation;Language Modeling;Question Answering
Yes
No ethics review needed.
- Did the authors try any kind of prompt engineering to see if it improves performance?
- The FanOutQA benchmark (Zhu et al., ACL 2024) strikes me as a particularly suitable Wikipedia-based testbed for this framework. Did the authors attempt to include it in their evaluation? 
- Another increasingly important factor in RAG systems in multilinguality, which impacts both retrievers and readers. It would be useful to include at least one multilingual dataset, like NoMIRACL (Thakur et al., EMNLP 2024), in this testing framework.
- With increasing adoption of RAG for a variety of tasks, this is a useful direction of research that helps researchers understand when and how to use this set up. These findings could be potentially useful for deciding how to configure a RAG framework based on the task.

- The RAGGED framework proposed in this paper is well-designed and addresses critical questions at each step of the pipeline. The authors aim to release the code for this framework which would enable easier replication and extension to include more models/experiments.

- The paper is well-written and the results are presented in a very easy to understand manner. I really appreciate how Table 1 and the figures that follow communicate the core findings succinctly. The appendix also provides useful additional details for the experiments.
This paper studies how the components of a retrieval-augmented generation (RAG) system impact its performance on a variety of tasks. They propose an evaluation framework called RAGGED that  evaluates the impact of more context, inclusion of irrelevant information, and the interaction between task type model performance. They compare combinations of three retrievers (BM25, ColBERT, Contriever) and four readers (FLAN, LLaMa, GPT, Claude) on multiple datasets and present a series of findings that make the case for designing RAG frameworks that are task-specific and tailored to their use case.
- The paper does not compare more state of the art closed-source models like GPT-4 and Claude 3 Opus. It’s understandable that cost may be a hindrance, but it would be helpful to conduct even a smaller set of experiments using stronger models so the takeaways from this paper are more broadly applicable.

- Unigram F1 doesn’t strike me as the ideal metric for evaluating reasoning quality in LLMs. Perhaps the authors could have explored semantic embedding-based approaches or by simply asking another LLM to verify the (output, label) pair for accuracy.

- While the takeaways are interesting, it would have been helpful to hear the authors’ thoughts on why certain models outperform others in different settings.
Yes
No ethics review needed.
N/A
1) The paper is very well-written and structured, and hence easy to follow. The research questions posed are clear and the findings are stated and discussed in a well-presented manner.
2) The work empirically identifies trends that are of interest to the community, serving as evidence, for instance, that RAG helps more in specialized domains and that improving the components of a RAG system separately does not guarantee higher end-to-end quality.
3) Thoughtful analysis of RAG systems, which have very quickly become ubiquitous, are very much needed.
The authors introduce an analysis framework, RAGGED, for understanding RAG best practices, namely, whether RAG helps, how many passages to retrieve, which retrievers and reader models to choose, and how that varies across tasks. They conduct this analysis across the retrievers BM25, ColBERT, and Contriever and assess several reader models, evaluating these on NQ (general domain), HotPotQA (complex questions), and BioASQ (specialized domain). Overall, they find that readers vary (some benefit from higher recall at larger retrieval depths; others are sensitive to noise) and that improved retrieval delivers more noticeable gains in specialized domains.
While I admire a thoughtful analysis paper, the present work simply does not explore rich enough questions to balance the lack of novel insight nor does it have enough empirical support for the nuanced claims it tries to extract. To illustrate, the primary non-trivial aspect of the present study are interesting claims like "improved retrieval helps more in specialized domains". This is analytically self-evident (i.e., is not necessarily a new claim) but good empirical support for such claims is always valuable. Unfortunately, the work is essentially trying to support this post-hoc from results on NQ vs. BioASQ, two of the most standard, over-used, and overly-simplistic datasets. To be clear, NQ and BioASQ are still valid and interesting datasets, but the lack of novelty coupled with the factoid and short answer nature and small variety of datasets selected makes it very hard to position this work at ICLR in my current read.

To be very clear, I think work like this is necessary, but to publish this as a long research paper the bar to clear might involve something llike (i) innovating dramatically more on the evaluation front, e.g. assessing human evaluators against a large number of automatic metrics, not just F1 overlap, over a large set of (perhaps contributed) new evaluation benchmarks that vary certain axes more directly or (ii) testing a set of less overused claims and hypotheses, e.g., novel methods that counteract the challenges the authors have observed. Right now, the contribution is simply too small and too ephemeral of an addition to the literature in this (overcrowded) domain.

I encourage the authors to continue this thoughtful analysis line of work and to consider how downstream RAG applications look like in practice --- or alternatively to take some of their statements presented as findings (e.g., improving retrieval doesn't always improve RAG quality) and present deeper insight or a method that contributes to better understanding or better quality.
Yes
No ethics review needed.
- Q1: Why weren't modern retrievers like BGE and search engines considered? Would this affect the reliability of the main conclusions?
- Q2: How was the strategy for selecting noise paragraphs determined? Does this choice affect the validity of the experiments?
- Q3: Does the conclusion "better retrievers ≠ better RAG performance" still hold after incorporating rerankers and more modern retrievers?
- Q4: Do the improve-then-plateau and peak-then-decline behavior patterns still exist with different RAG and LLM paradigms?
- Q5: Why was unigram F1 chosen instead of other potentially more suitable evaluation metrics? The paper mentions that “many LMs respond correctly semantically but use inexact or more verbose wording.” which can be avoided by extracting the answer using the LLM. The effect of the Unigram-F1 assessment is not comprehensive enough.
- Q6: What are the criteria for selecting noise paragraphs in Section 6? 
- Q7: Was the impact of different retrieval passage lengths considered?
- Q8: Can you explain the underlying reasons why different models respond differently to RAG improvements?
- Q9: How do the quantity and scope of parametric knowledge correlate with performance variations across different LLMs? Additionally, could the authors explore the dynamics of knowledge conflicts and integration between an LLM's internal (learned during pre-training) and external (retrieved during inference) knowledge bases?
- Q10: The paper mentions providing "To provide more concrete suggestions of the best practices under various cases, we introduce an analysis framework, RAGGED," What are some efficient approaches to determine the optimal RAG configuration when working with new datasets or models? Specifically, how can practitioners rapidly identify the most effective combination of retrieval methods, LLMs, $k$, and other key parameters?
This paper proposes the RAGGED framework with the comprehensive and systematic study of multiple key aspects of RAG configuration. It constructs a complete assessment system through four clear research questions (R1-R4).
This paper proposes RAGGED, aiming to study the impact of different readers, retrievers, and the number of retrieved documents. It reveals the different response patterns of the reader model to contexts and provides suggestions for deploying an RAG system.
# Limitations in Methodological Approach

### Retrieval Architecture Constraints

- Limited Exploration of Modern Retrieval Methods
  - The study did not incorporate state-of-the-art retrievers (e.g., BGE) and reranking mechanisms
  - Contemporary search engines and their potential impact were not evaluated
- Generalizability Concerns
  - The conclusion that "better searchers ≠ better RAG performance" may require reconsideration in light of recent empirical evidence using modern retrieval systems

### Architectural Paradigm Limitations

- Restricted RAG Implementation Scope
  - The investigation did not encompass emerging RAG variants (e.g., Self-RAG) and recent Large Language Model paradigms (e.g., OpenAI O1)
  - The impact of prompt engineering variations was not systematically explored (e.g., CoT and Self-Ask)
- Retrieval Source Considerations
  - Limited investigation into how different knowledge source types affect system performance

### Dataset and Task Coverage

- The study's conclusions are constrained by the limited variety of datasets and task types examined (only 3 datasets)
- A broader spectrum of use cases would strengthen the generalizability of the findings



# Limitations in Experimental Depth and Analysis

## Experimental Design Constraints

### Variable Control Inadequacies

- Noise Distribution Effects: The impact of noise location within documents remains unexplored
- Document Structure Variables
  - The influence of paragraph length variation was not systematically analyzed
  - Potential correlations between document structure and retrieval performance were not examined

### Analytical Depth Limitations

- Lack of analysis of the reasons why "Some retriever-reader combinations consistently benefit from RAG, while others only sometimes do for certain k’s, and a few never do regardless of k" 
- The analysis lacks depth, only revealing obvious, common, and widely known conclusions without examining the underlying causes. I suggest analyzing the conclusions mentioned in the article from the perspective of how LLMs handle conflicts and integration between their internal knowledge and external information.

## Practical Implementation Constraints

- Configuration Guidelines
  - Lack of concrete, actionable implementation recommendations
  - Modern RAG systems include many other components, such as rerankers, scoring mechanisms, query routing, and query rewriting. **Your work appears to be limited to performing grid search on just a few hyperparameters.**
- The applicability/generalizability to other tasks cannot be guaranteed.
Yes
No ethics review needed.
- Could the authors analyze how model characteristics (e.g., parameter count, architecture type, pre-training approach, token limits) correlate with RAG performance patterns? Have you observed any clustering of performance patterns based on model families or architectures that could suggest theoretical principles for predicting RAG behavior? Such analysis might help generalize findings to unstudied models and provide theoretical grounding for the empirical observations.
- The significantly different behavior observed in specialized domains (e.g., BioASQ) vs. open domain is intriguing. Could you elaborate on what characteristics of specialized domains lead to these differences? Have you analyzed if reader models pre-trained on domain-specific data show different RAG behavior patterns?
- Could you provide insights into how to predict whether a domain would benefit more from dense vs. sparse retrievers?
- The study examines multiple dimensions of RAG systems with a systematic approach, using diverse models and configurations. This provides valuable empirical evidence about component interactions.
- The authors provide their code and framework publicly, with well-documented experimental setups and clear implementation details.
- The work challenges some common assumptions about RAG systems, particularly regarding the relationship between retriever and reader performance.
The paper introduces RAGGED, a framework for analyzing and evaluating RAG systems. While the work presents a comprehensive analysis framework, its contributions mainly focus on investigating how different components and configurations affect RAG performance across various tasks and domains. 
The main contributions include:
- A systematic analysis framework examining RAG components,
- Empirical findings on RAG performance characteristics,
- Investigation of retriever-reader relationships,
- Domain-specific insights for RAG applications.
- While the paper presents extensive empirical findings about RAG system performance, it lacks a theoretical framework that could explain the underlying patterns and relationships between reader model's architectures (e.g., parameter count, token limits) and their performance characteristics, which could have enabled broader generalizations about unexplored model families and helped predict optimal configurations based on architectural features rather than just empirical observations.
- The evaluation focuses narrowly on question-answering tasks with relatively small datasets (2837 queries for NQ, 5600 for HotpotQA, 3837 for BioASQ). This raises questions about the generalizability of findings to other applications/domains.
- The paper could benefit from more rigorous comparison with existing RAG analysis frameworks or methodologies to better establish its unique contribution.
@inproceedings{
anonymous2024ragged,
title={{RAGGED}: Towards Informed Design of Retrieval Augmented Generation Systems},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=KDXj60FpJr},
note={under review}
}
Retrieval-augmented generation (RAG) systems have shown promise in improving
task performance by leveraging external context, but realizing their full potential
depends on careful configuration. In this paper, we investigate how the choice of
retriever and reader models, context length, and context quality impact RAG per-
formance across different task types. Our findings reveal that while some readers
consistently benefit from additional context, others degrade when exposed to irrele-
vant information, highlighting the need for tuning based on reader sensitivity to
noise. Moreover, retriever improvements do not always translate into proportional
gains in reader results, particularly in open-domain questions. However, in spe-
cialized tasks, even small improvements in retrieval can significantly boost reader
results. These insights underscore the importance of optimizing RAG systems by
aligning configurations with task complexity and domain-specific needs.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Retrieval Augmented Generation Language Modeling Question Answering
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/1e82a850fba078c58dfb70ed1e6a25c9a087743a.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
RAGGED: Towards Informed Design of Retrieval Augmented Generation Systems
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission