MemSim: A Bayesian Simulator for Evaluating Memory of LLM-based Personal Assistants
LLM-based agent;memory;evaluation;personal assistant
Yes
There is a lot of sensitive information listed in the generated datasets, which seems to be generated by LLM.
Yes, Privacy, security and safety
- It is kind of confusing that the author mentions Baysian Relation Network. It seems to be a Probabilistic Graphical Model. Why the proof in Sec3.2 can give the conclusion:"we introduce prior knowledge of the specific scenario into the graphical structure and sampling process, which can improve the diversity and scalability of user profiles,"
- Where is the BRNet coming from? How is user profiles generated?
- The paper adopts BRNet to sample user profile graphs and build question-answer pairs according to these graphs based on rules. It reduces the hallucinations compared to methods that construct samples directly.
-  Human evaluation seems to validate the effectiveness of rationality and diversity for generated results from the proposed MemSim.
- The paper proposed MemSim, a simulator for automatically constructing diverse and rational QA pairs based on the Bayesian Relation Network. 
- The paper also constructs a MemDaily Dataset using MemSim to evaluate the memory mechanisms of LLMs.
- The human evaluation lacks a detailed description. How can an annotator give a diversity score to a sample without comparing all the samples in the produced dataset?
- The tables are hard to understand, no detailed description about the metric abbreviation. For example in Table 6,7.
Yes
No ethics review needed.
1) See above
2) any prompts or implemention details for baselines and experiments?
3) notions are not clear.
1. The diversity, scalability and reliability of generated datasets can be improved since it mainly based on LLMs to automatically construct the data, and the user descriptions (a.k.a, messages) related to user profile is highly controllable.
2. The dataset considers several practical usage cases, including single-hop QA, multi-hop QA, comparative QA, aggregative QA and  post-processing QA.
The paper proposes a data genertor based on Bayesian Relation Network and a causal generation mechanism, aiming to using LLMs to simulate users (i.e., generate user profiles / attrubutes) and generate evaluation datasets (i.e., generate lots of user descriptions based on previous sampled profile). Furthermore, the paper evaluate the quality of collected dataset -- MemDaily and provide performance analysis using GLM4-9B.
1. lots of statements are not convincing. There are several examples: 1) line 64, the author claims the work is first work to evaluate memory of LLM-based personal assistants in an objective and automatic way. there are many studies evaluation memory usages in objective and automatic ways, such as [1]; 2) line 99, the authors keep emphasising the importance of "objective" evaluation, since they think the human annotator introduce bias. However, my personal concern is LLM also has bias, just like human. In this way, I would not say the objective evaluation is guaranteed.

2. baselines are too weak and experimental results are not convincing. The baselines used in both section 4.1 and 4.2 are too weak, and there are no implemention details. For examples, it is not hard to deign multi-stage but easier prompting strategy to generate user profiles instead of designing such complext sampling mechanisms. Even in this way, the performance gap in table 4 is not significant. The simple baseline leads to better performanc in most metrics. No detailed analysis is provided.

3. the value of dataset is not significant, and the unique advantages compared with existing datasets are not clear. there are several observations: 1) according to table 2, the TPM is around 15 and total messages is around 4000, then for each test instance, if we consider all user message as retrieval pool (note this is also not indicated in the paper), the max token number should be 15*4000 = 60000 token, while most of existing long-term memory benchmark consider much longer context, not even to mention if the pool becomes much smaller if the retrieval pool does not use all messages (so here many detailed statistics are missing). 2) user messages are generated using prompting given detailed user atttributes. Despite it can reduce hallucinations of LLMs, it also poses many constraints of LLMs and make the expression is not natural with real-world interections, such as the user may not explicitly talk about these attributes; 3) according to table 6, the simpliest baseline (RetrMem or FullMem) can achieve almost 80% accuracy, and FullMem can achieve 95%, which further support my claim that the dataset is relatively easy and not to mention the author does not use the existing SOTA model such as gpt4o or others.

[1] Hello Again! LLM-powered Personalized Agent for Long-term Dialogue
Yes
no
No ethics review needed.
1. In section 3.2, how is the joint probability distribution of $X$ and the conditional probability distribution of $x$  determined? Do they need to be manually defined?
2. In MemDaily, there are only 11 entities and 73 attributes. Are these hand-crafted? If it is hand-crafted, how does the dataset scale up?
1. This paper is logically clear. It illustrates the BRNet with theoretical proof and uses multiple symbols to accurately describe causal generation mechanism. 
2. Using BRNet, this method eliminates the hallucination problem caused by LLM generation. Additionally, the causal generation mechanism guarantees the diversity of the datasets. 
3. Using this dataset, this paper evaluates different memory mechanisms of LLM agents and analyzes the results, which are insightful for future agent design.
This paper proposes a method for automatically constructing reliable, diverse, and scalable QAs, MemSim. Based on a Bayesian simulator, MemSim first builds a Bayesian relation network and then designs a causal generation mechanism to produce various types of QAs, including single-hop, multi-hop, comparative, aggregative, and post-processing. Finally, this paper uses MemSim to generate a dataset named MemDaily to evaluate the memory mechanisms in LLM agents.
1. There is a lack of comparison between MemSim and existing methods for QA generation. For example, generate personal questions through LLMs or build a personal KB and let the LLM generate messages based on the entities and relations. And then, evaluate the datasets using metrics in section 4.
2. The difference between BRNet and a KB is not clear. In section 2, this paper claims that KBQA mainly focuses on common-sense questions. However, building a personal or anonymous KB and allowing LLMs to generate datasets based on triples in the KB is also feasible for personal questions.
Yes
No ethics review needed.
1. In section 4.1 -- evaluation of user profiles, it will be good to mention the number of the total generated user profiles. In addition, Is each user profile evaluated by a single evaluator or multiple evaluators? 
2. When constructing the MemDaily dataset, it is not clear to me how the trajectory is constructed. 
3. It takes a while to understand the sentence in line 430 -- "Another baseline method that directly ... performs much lower reliability. We implement this method ... as OracleMem ...". The results of OracleMem are compared with the results in Table 5, rather than the other methods in Table 6. It will be better to make this clear. 
4. When evaluating the MemDaily dataset in section 4.3, how is the retrieval target obtained? According to section 4.3, the retrieval target seems to be part of the groundtruth when constructing the dataset. 
5. Related to question 4, in Table 6, are there any insights on why the performance of the OracleMem is much worse in some types of QAs? The OracleMem uses the targeted user message, which is not available when testing the memory machanism of other LLM-based agents. Therefore, shall we say that the results of the OracleMem are the upper bound of the model performance in this dataset? For the Aggregative type of question, the accuracy is 0.376. Is there other way that can further improve this performance? 
6. In line 485, "LLM directly uses the LLM to ... ", are the candidate messages and the question provided to LLM and let LLM decide the top-k relevant message?
* This paper is well-written. The challenges of constructing dataset with considering the reliability, diversity and scalability set up a good motivation for the paper. 
* Both theoretical and experimental proofs are included to validate the effectiveness of the proposed method. 
* When generating the QAs, different types of QAs are considered. This ensures the diversity and coverage of the QAs. 
* The experiments are comprehensive: 1) Variations of the MemDaily dataset are also considered. 2) Different memory mechanisms are evaluated on the proposed dataset.
This paper presents a novel method -- MemSim, to construct reliable QA datasets, to evaluate the memory capability of LLM-based personal assistants. A Bayesian Relation Network and a Causal Generation Mechanism are introduced to ensure the diversity, reliability, and scalability of the generated datasets. Based on MemSim, a dataset named MemDaily is constructed. Extensive experiments are conducted to assess the quality of the dataset, as well as evaluate the different memory mechanisms of LLM-based agents.
In the proposed method, the user messages are factual statements. And the constructed question-answers mainly focus on the entities/attributes. Each message in the trajectory seems to be independent. And there is no coreference between messages, no ambiguity of the user message. These greatly simplify the problem of evaluating personal assistants in a real-world scenario.
@inproceedings{
anonymous2024memsim,
title={MemSim: A Bayesian Simulator for Evaluating Memory of {LLM}-based Personal Assistants},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=8w22WLy2R8},
note={under review}
}
LLM-based agents have been widely applied as personal assistants, capable of memorizing information from user messages and responding to personal queries. However, there still lacks an objective and automatic evaluation on their memory capability, largely due to the challenges in constructing reliable questions and answers (QAs) according to user messages. In this paper, we propose MemSim, a Bayesian simulator designed to automatically construct reliable QAs from generated user messages, simultaneously keeping their diversity and scalability. Specifically, we introduce the Bayesian Relation Network (BRNet) and a causal generation mechanism to mitigate the impact of LLM hallucinations on factual information, facilitating the automatic creation of an evaluation dataset. Based on MemSim, we generate a dataset in the daily-life scenario, named MemDaily, and conduct extensive experiments to assess the effectiveness of our approach. We also provide a benchmark for evaluating different memory mechanisms in LLM-based agents with the MemDaily dataset.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
LLM-based agent memory evaluation personal assistant
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/3b002d7e1d3c585822aaf4e632e87aa4892cd166.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
MemSim: A Bayesian Simulator for Evaluating Memory of LLM-based Personal Assistants
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission