Loius (Look it up in the Structure): Benchmark and Techniques for Document structure aware LLM based Retrieval
information retrieval;llm;model based retrieval;document search;retrieval benchmark;document structure;benchmark
Yes
No ethics review needed.
Most questions are in the weaknesses. But some comments about writing:
- Citations should be \citep{}, it seems like they are not, and are instead inline
- [Minor] It would be nice to see some examples in the appendix, in terms of passage it came from and question generated.
- The new proposed task is an interesting contribution, and to my knowledge has not been studied before
- They use a wide variety of retrieval model architectures, from cross-encoders, to generative retrieval, spare retrieval, and dense retrieval.
- Their proposed model does better on the data they generated
This paper proposes a new task for retrieval models: that of finding the relevant section title in a book, given a question. They build a dataset for this task by using PDF parsing to extract the table of contents from books. They generate question for the task through the use of Mixtral. 

They then train a model to do well on this task, called Louis.  Their model is trained on the training set (also generated with Mixtral) and shows improved performance over other model types, including non-fine-tuned versions of DSI, BM25, and Mistral. They show an error analysis where their model performs better.
1. There is a lack of important details. How many chunks/passages are there in the corpus for dense retrieval (or BM25)? Does each chunk for these models also get the section, so that the model can use that information? Why are there no long context models evaluations (Gemini/Prolong/even just a 128k Llama)? Does Mistral see the entire book or just the ToC? etc.
2. Conceptually, performing ToC retrieval is strictly easier than searching for the specific passage that a question was generated from. From related work on searching for the relevant passage in long documents (see the "Scrolls Benchmark" [1] for task examples) IR models perform pretty well with retrieval (see the LoCo benchmark [2]). It seems something is wrong with the setup if the models are doing this poorly. My guess is that the book is chunked but not given the section headers in the chunks, so they lack that context. Furthermore, BGE is a decent baseline, but pretty weak in comparison - see the MTEB leaderboard where BGE-base (I assume because of the 768 dim vector, but again not described) is ranked #47.
3. Overall, the task does not appear very challenging.  R@3 is 90% and close to that for many baseline models (DSI, BM25). If BM25 can get nearly the same performance as the proposal model, it does not seem like much of an improvement (and is perhaps not even statistically significant).
4. There is no analysis on the false negative information in these questions. It is likely that many questions are answered in several places and that falsely lowers the score. I would guess that if the authors did an analysis of the Recall @ 3 failures that the answer is given in both places.  This makes (3) more of an issue and lowers the quality of the dataset.
5. The main modeling contribution seems to be that using the same data generation process as making the test set, and then by training on it, makes a model better able to perform this task. This is unsurprising and demonstrated over and over in the literature on every modality. This would be totally fine, if this was the only issue, but I assume that this means that the proposed Louis model cannot do another other tasks, as it is likely overfit for this. 

Overall, I think the task could be useful for the retrieval community as a useful long-context evaluation. However, I am unconvinced of the datasets quality and there appear to be modeling concerns (or at least a lack of information).

[1] Scrolls Benchmark: https://www.scrolls-benchmark.com/tasks

[2] LoCo benchmark: https://arxiv.org/pdf/2402.07440v2
Yes
No ethics review needed.
The questions are listed in the 'Weaknesses' section.
1. The paper introduces a new dataset containing queries and their corresponding subsections, which may contribute to research in document retrieval.
2. Constructing the dataset and conducting the experiments required substantial effort.
Inspired by the way humans often search through the Table of Contents when reading books, this paper introduces a new task: given a query and the Table of Contents (TOC) of a long document, the objective is to retrieve the correct subsection in the TOC that contains evidence to answer the query. The authors introduce and release a new multi-domain dataset, ToCTome, which consists of 18 books across 6 domains. For each subsection, they use the Mixtral 8x7b model to generate questions, forming subsection-query pairs. Additionally, they split the data into training, development, and test sets, and fine-tune Mistral Instruct v0.2 with the LoRA adapter. Experimental results demonstrate that the fine-tuned Mistral Instruct v0.2 achieves an impressive R@1 score of 82.6%, outperforming BM25, dense retrieval models, and the original, non-fine-tuned Mistral Instruct v0.2 on this task. These findings highlight the strong capabilities of LLM in ToC-based retrieval.
1. The authors introduce a new problem but do not explain its importance and significance. The purpose of using a large language model (LLM) to locate the subsection relevant to the query is unclear. The authors simply state that this "mimics how humans leverage the structure of a book," but they don’t explain why this is necessary. I believe the authors should address this question: "After finding the subsection relevant to the query, what can be achieved?"
2. The authors assume that each query can be matched to a corresponding subsection in the Table of Contents. However, in real-world scenarios, many queries may not align with the Table of Contents; in particular, some fine-grained questions cannot be mapped to a specific subsection.
3. The proposed dataset and method are designed for queries that correspond to a single subsection. However, some queries may span multiple subsections, which this dataset and method do not account for. This results in oversimplified problem modeling.
4. The writing of this paper has several issues: (1) Many sentences do not follow an academic tone, and exclamation marks are used frequently, which is uncommon in academic papers. (2) Detailed experimental results are repeated in the ABSTRACT, INTRODUCTION, and EXPERIMENTS sections.
Yes
No ethics review needed.
1. Can the methods developed for Loius be adapted for general retrieval tasks beyond structured documents like books? If so, what modifications would be necessary?
2. Why were different maximum input lengths used for Loius and DSI in your experiments? Could this difference affect the comparative performance results reported?
3. Given the reliance on a structured ToC, how does Loius handle content variations within sections that might not be accurately reflected by their ToC descriptions?
- The innovative use of the ToC to guide the retrieval process introduces a unique approach to document navigation, potentially transforming retrieval strategies for structured documents.
- Extensive experiments validate the system’s efficacy, particularly its ability to outperform traditional retrieval methods.
- The creation of the ToCTome benchmark contributes a valuable resource to the research community, fostering further innovation in retrieval systems that understand document structure.
The research paper introduces Loius, a novel retrieval system that leverages the structure of ToC to enhance retrieval processes. By mimicking how humans intuitively use a ToC to navigate books, Loius proposes a fresh retrieval paradigm that contrasts with traditional methods based on keyword or vector-based searches. The paper also presents a new benchmark dataset ToCTome, which comprises 18 books across six diverse domains, providing a robust platform for evaluating ToC-based retrieval systems. The results demonstrate Loius’s superior performance in accurately locating relevant sections, boasting a Recall@1 score significantly higher than the next best system (DSI) and other baseline models like BM25 and DPR.
- The specific application of ToC-based retrieval to books may not generalize to other types of content that lack a clear hierarchical structure, such as unstructured web pages or documents without a ToC.
- The technical novelty appears limited primarily to the adaptation of existing retrieval technologies to a new input structure (ToC), potentially limiting the method’s broader applicative insights.
- In Section 5.2, I find that the maximum input length of Loius and DSI are significantly different. It is unclear if this difference may affect the experimental results.
Can we instruction finetune an LLM to pick the best subsection to answer a user query by showing it a Table of Contents of a long document?
@inproceedings{
anonymous2024loius,
title={Loius (Look it up in the Structure): Benchmark and Techniques for Document structure aware {LLM} based Retrieval},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=53kUa92R7J},
note={under review}
}
Modern day LLMs have shown remarkable success in understanding human instructions and provide satisfactory responses across a diverse set of tasks that require world knowledge. In this work, we leverage the inherent capability of LLMs to learn new tasks through instruction to design a novel retriever Loius that proposes a new retrieval paradigm: Table of Contents (ToC) based retrieval. Loius mimics how humans leverage the structure of a book (or a long document) by using ToC to search for information. Loius demonstrates that LLMs can be finetuned to build ToC based retrievers where the most granular ToC entry (leaf node) serves as the retrieval unit. This approach differs from traditional retrieval systems, which index chunks of information from long documents. We also introduce a novel
comprehensive benchmark ToCTome featuring 18 books across 6 diverse domains (with train, test and dev splits) and show that Loius achieves R@1 score of 82.6% as compared to the next best system (DSI) (76.9%) and other baselines such as BM25 (71.6%), DPR (53.6%) and out-of-the-box Mistral Instruct v0.2 (18.0%) on average. Loius is capable of incorporating corpus knowledge while learning to select the most relevant ToC leaf node, resulting in almost zero hallucinations (lessthan 0.05%) – a key issue with LLM-based retrievers. In addition to the benchmark, complete code for building Loius will be publicly released to accelerate research
in the novel direction of ToC based retrieval.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
information retrieval llm model based retrieval document search retrieval benchmark document structure benchmark
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/93b973fae8da9e008b97f70fa474e199c1844263.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/77450c042f442e7fc21d225dfdc8f444e0bda0ab.zip
Loius (Look it up in the Structure): Benchmark and Techniques for Document structure aware LLM based Retrieval
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission