Episodic Memories Generation and Evaluation Benchmark for Large Language Models
Episodic Memory Modeling;Large Language Models;Synthetic Benchmark Generation;Cue-based Retrieval;Temporal-Spatial Reasoning;Long-context Understanding;Human-inspired AI
Yes
No ethics review needed.
Q1: When the authors write: "The proposed episodic memory benchmark exhibits several desirable properties: it is contamination-free by design, scalable with low human labor, offers unambiguous cues and ground truth, and the ability to model multiple cues and events within a synthetic yet realistic narrative."  What does scalable mean here? How is this demonstrated in the paper? 

Q2: The authors write that needle in the haystack benchmarks "do not incorporate temporal nor spatial awareness", but isn't this point undermined by the limitation related to "event independence" the authors mention?  

Q3: The authors also write that bABI / bABILong "often involve highly artificial scenarios lacking complexity and realism – opening the door to shortcut reasoning by exploiting dataset biases or patterns", but isn't this point undermined by the limitation related to "temporal representation" the authors mention? Also how are dataset biases / patterns addressed in a way that goes beyond bABI? 

Q4: For the point on "limited domain scope" could you explain why more domains or even random variations of the book were not considered in this paper? What roadblocks remain that made the authors position it as future work?
I think this paper looks at an important problem and is well written for the most part. The approach outlined in section 3 makes a lot of sense. I really agree about the five points mentioned in the "Need for an episodic memory benchmark" paragraph at the end of page 3. I also found that Figure 3 and Table 4 presented some interesting analysis giving more insight into the ways that current LLMs struggle with reasoning over episodic memories.
This paper proposes an approach for generating a benchmark for evaluating the episodic memory of LLMs. The authors use this generation approach to develop short book and long book splits consisting of 456 and 686 Q/A pairs respectively. The authors then evaluate high quality LLMs on this benchmark to showcase that even with RAG achieving high performance can be quite challenging.
**Data Generated:** I think the biggest weakness of this paper is the quantity and quality of data generated as part of the short book and long book splits. This first stood out to me when reading Table 2. For this amount of content, it feels like even the use of automated means are not necessary i.e. a human can annotate an actual book with questions and it would be a lot higher quality than what is produced here. Given that the authors are specifying a general strategy for generating benchmarks in section 3, I thought at least a number of randomly generated books would be considered if not also significant diversity within these books.  As a result, this benchmark does not yield easy high confidence analysis, which is showcased by massive error bars throughout the main results table (table 3). 

**Table 3:** The results of the in-context and RAG models are largely in-line with general expectations in Table 3, so the benchmark does not really lead to new findings in comparison to the current literature. The only interesting finding is with respect to the fine-tuning baseline, but the implementation of this baseline seems flawed. First of all, there is barely enough data for this dataset to be used for evaluation of LLMs, it seems like there is simply not the data required to facilitate fine-tuning. Judging from the appendix, it seems like for some reason only a number of events matching the cues of 1 was used for fine-tuning, which seems to fully explain the results in this row on its own. It is not even clear to me if the 0.83$\pm$0.35 is using the same data for training and testing.
Yes
No ethics review needed.
1. Could you explain the "naive fine-tuning" approach mentioned in the paper? What datasets and methods were used, and how does this approach differ from other fine-tuning strategies for episodic memory tasks?

2. Just curious—does your benchmark have a specific name, or is it simply called "Short Book" and "Long Book"?
1. The paper effectively organizes episodic memory tasks for LLMs using a cue-based recall and retrieval method. The authors provide examples of different cues, showing various combinations that models must use to retrieve event information based on time, location, involved entities, or content. This clear design demonstrates a solid grasp of how to simulate episodic memory in LLMs and offers a strong foundation for evaluating model recall across a range of scenarios, from simple to complex.
2. The benchmark tests the model's ability to handle both clear and vague questions, similar to real-world situations where memory needs vary. By asking models to either recall a specific event or recognize several related events, the tasks assess how well the models adapt to different recall demands.
3. The benchmark includes carefully designed tests to assess a model's ability to recognize unfamiliar events or entities and admit when it lacks information. This is crucial for evaluating whether LLMs can avoid hallucinations, and this thoughtful design adds reliability to the benchmark.
4. The paper offers detailed statistics and information about the benchmark, along with several ablation studies in the appendix. This level of detail shows the authors' commitment to transparency and rigor, helping readers understand the benchmark’s structure and how different elements affect model performance. These ablation studies also provide deeper insights and useful guidance for future research.
The paper presents a framework for modeling and evaluating episodic memory in large language models (LLMs), focusing on their ability to recall and process events associated with specific times and locations, similar to human episodic memory. The authors propose a method that uses entities and events to construct episodic memories and a benchmark designed to test LLMs on tasks such as recalling event details, tracking entity states, and understanding temporal-spatial contexts. The benchmark, including synthetic datasets and structured tasks, also assesses the models' ability to avoid confabulations by identifying unfamiliar information. The authors' evaluation of models like GPT-4 and Claude shows that current LLMs struggle with complex, multi-event scenarios and spatio-temporal reasoning, highlighting the need for improved episodic memory frameworks and training methods tailored to these capabilities.
1. A limitation of the paper is that it only evaluates proprietary models like GPT-4o and Claude, rather than open-source models such as LLaMA 3. Including open-source models would make the findings more generalizable and accessible to a broader research community, enabling comparisons across a wider range of models and methods.
2. The benchmark mainly uses clear cues, which, while providing consistency and control, may not capture the subtler cues common in natural language memory tasks. Adding more ambiguous time markers and indirect references could better simulate real-world memory challenges and lead to a stronger test of models' episodic memory abilities and their handling of less obvious retrieval cues.
Yes
No ethics review needed.
It would be beneficial to explore whether different fine-tuning parameters, or fine-tuning applied to other models could enhance the performance of episodic memory tasks
It is the first time to evaluate updated memories for episodic memory, events with rich contextual information and involving the tracking of specific entities occurring at specific time and spatial locations. And have a solid analysis of event complexity.

The drop of accuracy of fine-tuned model highlights that current fine-tuning techniques fall short in understanding episodic events and their complex interrelationships. All models have a low percentage of exact matches in the temporal ordering task, and even when the models retrieve the correct events, they often fail to order the events correctly.
This paper introduces and explores the concept of episodic memory in the context of long-text comprehension by language models. This approach emphasizes the necessity for models to maintain a coherent understanding of an entity’s state as it evolves over time, space, and content. Both short and long synthetic documents were generated using various cue templates. The dataset also includes null answers incorporated to test for model illusions.

The findings indicate that models like GPT-4o with contextual memory and Claude 3.5 Sonnet4 with RAG memory scored the highest on average, suggesting that retrieval-based methods can improve situational memory by narrowing the context relevant to each query. While some models demonstrated near-perfect accuracy in chapters involving zero or one event per entity, their performance declined significantly as the number of events increased.
1. line 123 has different citation format
Yes
No ethics review needed.
1.How does the synthetic data generation process ensure realistic temporal and causal relationships between events?  
2.Have you conducted rigorous validation studies comparing LLM judgments against human annotations or established metrics? What specific measures were taken to ensure consistency and reproducibility in the evaluation process?  
3.How reliable is the process of scoring relevance "against each ground truth item"? Could you provide examples of how partial matches are handled?  
4.In Table 3, the fine-tuned model performs well on single-event queries (F1=0.83) but poorly on multi-event queries (F1≤0.37). Could you elaborate on why naive fine-tuning fails to generalize beyond single-event memorization? What specific architectural or training modifications might address this limitation?  
5.The gradient pattern in Figure 3 shows degrading performance from context to space to time cues. What specific aspects of temporal reasoning make it particularly challenging for current LLMs?  
6.For the "Latest state recall" results in Table 4, what specific challenges prevent models from achieving higher accuracy in tracking entity states over time?  
7.Have you tried other fine-tuning approaches beyond single-event memorization that might better capture the hierarchical and relational nature of episodic memory?  
8.Have you tried other retrieval strategies beyond cosine similarity? How do you address the challenge of retrieving coherent information when relevant context is distributed across multiple chunks?  
9.How might this benchmark contribute to developing novel training methodologies for episodic memory tasks in LLMs, beyond RAG, fine-tuning or parametric memory storage?
1.The paper tries to address an important and timely challenge about the need for better episodic memory capabilities in LLMs;  
2.The research takes a structured approach to modeling episodic memory by incorporating key concepts from cognitive science, focusing on key aspects of memory: temporal context, spatial grounding, and entity tracking;  
3.The methodology demonstrates rigor through: (1) creating contamination-free synthetic benchmarks, (2) introducing multiple verification steps to ensure data quality, (3) providing flexibility in generating datasets of different sizes and complexities;  
4.The study develops systematic ways to assess different aspects of memory (recall, chronological ordering, latest state).
This paper introduces a benchmark for evaluating episodic memory capabilities in LLMs. The authors create a framework inspired by cognitive science to model episodic events with temporal and spatial contexts, entities, and detailed descriptions. They generate synthetic datasets and evaluate state-of-the-art LLMs across various recall and episodic reasoning tasks. The evaluation considers different memory strategies: in-context learning, RAG, and fine-tuning. The authors observe that even advanced models face challenges in handling episodic memory tasks, particularly when recalling sequences of related events or complex spatiotemporal relationships.
1.The paper primarily utilizes LLM-generated synthetic data (Section 4.1), but does not adequately validate the quality and representativeness of the generated narratives. For example, while the authors claim to verify "adherence to event meta-data," they do not provide quantitative metrics for assessing narrative coherence or natural language properties. The authors should establish clear validation metrics and demonstrate how their synthetic data captures the essential properties of real episodic memories.  
2.The scope of the benchmark is unnecessarily limited. The current implementation: (1) only considers fictional narratives with human-like protagonists, (2) Uses oversimplified temporal representations, (3) Fails to address complex episodic memory scenarios involving interconnected events. The authors should expand the benchmark to include more diverse scenarios, complex temporal relationships, and interconnected event sequences that more accurately reflect real-world episodic memory challenges.  
3.While Section 3.1 emphasizes the importance of entity state tracking, the experimental results in Table 3 do not adequately measure this capability. The evaluation focuses on simple recall rather than complex state changes. The paper claims to test "understanding temporal sequences" but does not properly evaluate how models handle causally related state changes. The authors should design specific test cases for complex state tracking, evaluate models' ability to handle causally related state changes, and include metrics for measuring state tracking accuracy.  
4.The LLM-as-judge approach described in Section 4.3 lacks validation of inter-judge consistency across different evaluator LLMs and does not establish correlation with human judgments. This could be addressed by including human evaluation benchmarks and demonstrating consistent assessments across multiple judge models.  
5.The RAG experiments in Section 5.1 use only basic paragraph-level chunking without exploring alternative strategies. The authors should investigate alternative chunking approaches, compare different retrieval mechanisms, and analyze how these choices impact episodic memory performance.  
6.While Table 4 shows poor performance in chronological ordering tasks, the paper doesn't provide detailed error analysis or investigate specific failure patterns. The analysis in Section 5.2 focuses on aggregate metrics without examining individual failure cases. The authors should provide detailed case studies of failure modes, analyze patterns in chronological ordering errors, and investigate whether specific temporal relationships consistently challenge the models.  
7.Although Section 5.2 mentions testing for hallucinations, the analysis is limited. The paper fails to examine when and why models confabulate, or how confabulation patterns vary across different model architectures and memory strategies. This could be improved by designing specific experiments to probe confabulation triggers and providing metrics for measuring confabulation severity.
Yes
No ethics review needed.
- As the authors acknowledged in the paper, there might be multiple episodic events that can be extracted from each chapter. In addition to the specified time and location their might be other times and locations referred to in the chapter. Are there any measures to filter out these chapters? Also, does the fact that there are many other events (especially contents) in addition to the specified one make the F1-score metric less suitable, since false positives might not actually be false? This is the main concern that I would like to get addressed.
- For the results in the main text, the entities and the books are generated by Claude 3.5 Sonnet. Does this possibly give Claude models an unfair advantage in the evaluation?
- Following my comment regarding the events being unrealistic in the "weaknesses" section, I would hope to get more insights on this from the authors, e.g. whether unrealistic pairings actually impact LLM performance differently than realistic ones?
- I would appreciate if the authors can give more qualitative analysis and insight on the experiment results. E.g. In cases where there are 0 matching events and the model hallucinates an answer, does the model produce an event that has never appeared in the text; appeared in the text but completely irrelevant; or relevant in that its spatially or chronologically very close to the cue? What are the common failure modes?
- The paper describes the benchmark generation process in great detail, allowing the readers to easily replicate the data generation process and have a better understanding of the benchmark.
- By using LLMs to generate synthetic data, the authors eliminates the data contamination issue and make the benchmark more easily scalable.
- The paper experimented with a comprehensive list of different settings. The experiment results are provided with error bars.
The paper presents a new benchmark to evaluate the ability of LLMs on episodic memories - specifically, retrieving relevant information about an event (or multiple events) given a specified time, location, person, or content. The data in the benchmark is generated by LLMs in the form of a book, with each chapter of which being an event. The authors generate random time, location, person and content beforehand and sample one random combination for each chapter; they also specify where the time, location, person and content needs to appear within the chapter for more controllability on the benchmark. The evaluations are done by LLM as a judge to identify the relevant items in the answer and score their relevance. The experiments and results suggest that it is still very challenging for current SoTA LLMs to deal with complex spatio-temporal relationships, and a lot can be improved on the episodic memory capabilities of LLMs.
- The events described in the book chapters are not filtered in terms of whether they are somewhat realistic or not. It seems very likely that the detail of the event does not align very well with the location. The fact that the dataset contains some non-realistic events might limit the capability of the LLMs to recall them, since the LLMs may use their common sense knowledge to think that it is not likely for this event to take place at the specified location.
- The way humans experience and memorize different events has a strong temporal structure. More specifically, humans experience the events in temporal order, which makes it simpler for humans to recall the last occurrence or list things in chronological order. In this benchmark there is no temporal order or structure in the book (plus there are no causal relationship between the events as the authors mentioned), which makes it less realistic.
- One property of human episodic memory is that the time and location in the cue does not need to be a exact match with the event, and humans can naturally recall events that occurs close to the specified time or location. This is not evaluated in the current benchmark.
- While I appreciate that the authors gave very comprehensive information about the benchmark in the appendix, the presentation in the main text could be enhanced by including a figure or flowchart to describe the generation process, or provide at least one example of the what the retrieval task looks like for the LLM.
A new benchmark for testing episodic memory in AI language models, using synthetic stories to evaluate recall of specific events, their contexts, and chronological relationships.
@inproceedings{
anonymous2024episodic,
title={Episodic Memories Generation and Evaluation Benchmark for Large Language Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=6ycX677p2l},
note={under review}
}
Episodic memory -- the ability to recall specific events grounded in time and space -- is a cornerstone of human cognition, enabling not only coherent storytelling, but also planning and decision-making. Despite their remarkable capabilities, Large Language Models (LLMs) lack a robust mechanism for episodic memory: we argue that integrating episodic memory capabilities into LLM is essential for advancing AI towards human-like cognition, increasing their potential to reason consistently and ground their output in real-world episodic events, hence avoiding confabulations. To address this challenge, we introduce a comprehensive framework to model and evaluate LLM episodic memory capabilities. Drawing inspiration from cognitive science, we develop a structured approach to represent episodic events, encapsulating temporal and spatial contexts, involved entities, and detailed descriptions. We synthesize a unique episodic memory benchmark, free from contamination,
and release open source code and datasets to assess LLM performance across various recall and episodic reasoning tasks. Our evaluation of state-of-the-art models, including GPT-4 and Claude variants, in addition to the recent o1-mini, reveals that even the most advanced LLMs struggle with episodic memory tasks, particularly when dealing with multiple related events or complex spatio-temporal relationships --- even in contexts as short as 10k-100k tokens.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Episodic Memory Modeling Large Language Models Synthetic Benchmark Generation Cue-based Retrieval Temporal-Spatial Reasoning Long-context Understanding Human-inspired AI
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/8543fff9601efa349ad3f319b12b26d8a0e2c816.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Episodic Memories Generation and Evaluation Benchmark for Large Language Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission