Identifying and Addressing Delusions for Target-Directed Decision Making
delusions;hallucination;planning;generalization;reinforcement learning
Yes
No ethics review needed.
1.  I see much of the mitigation strategies are based on making the estimator better and recognizing delusions from the generator. What kind of mitigation strategies (beyond "generate") would be useful to generate less delusional targets?
1. The systematic approach to studying delusions in target-oriented RL is relevant to the community. The authors' approach allows us to understand the potentially problematic behavior that can arise from naively training agents within this paradigm.
2. The empirical evaluation shows an organized approach to verifying the effect of the types of delusions and the effects of potential mitigation strategies. 
3. The knowledge provided in the paper is of general interest to practitioners and researchers.
This paper studies the problem of “delusions” in target-directed (goal-conditioned) RL by establishing a taxonomy of delusions that are possible for the Generator (proposer of the targets) and the Estimator (that measures the goodness of the target). Then, they propose two new HER relabeling strategies (and mixed strategies) to mitigate the effects of different types of delusions.
Finally, they evaluate their proposed strategies in a controlled gridworld that allows them to quantify the different kinds of delusional targets they consider.
1. I can see that the authors studied the problem with empirical rigor, however, the experimental section, which goes a long way to showcase the problems, looks a bit crammed and it’s hard to follow. I would suggest sacrificing part of the previous discussion to treat the empirical section with more care.
2. Figure 3, it’s hard to follow because of the number of overlapping curves, colors, and legend positioning. This overall makes the curves hard to interpret.
3. Though I understand that scaling these ideas to more complex environments can be the subject of future research, perhaps showcasing the effect on the learning performance these mitigation strategies can have in a different environment (beyond the controlled gridworld proposed) would also be informative and relevant to this paper.
Yes
No ethics review needed.
- "For aggregated OOD performance [...] we sample 20 tasks [...], which have a mean difficulty matching the training tasks.". Could you clarify why you call this OOD performance? For this to be the case, I would both expect the gradient of difficulties to be wider. My intuition is that you want to keep the average the same to distinguish capability generalization vs delusion generalization, is that correct? I would however not call this OOD performance if I understand it correctly...  In general, I am curious how you see the difference between capability vs goal misgeneralization [1], and argue which of the two your work addresses. This would help me better clarify how valid the claims made are

- "To maximize difficulty, the initial state is fixed in each novel evaluation task instance: the agents are not only spawned to be at the furthest side of the monster": this is not always true, especially if the agent did not see often particular combination of start positions/state. This is true in the case only in terms of capability generalization, but I guess your answer to the above can also address this one.

- I suggest for the experiments to include actual OOD scenarios as I intend them. For example, try to test a scenario where the agent spawns with already one shield in hand, but another is present in the environment ( without training in such a situation). It would then be interesting to see if the targets generated still suggest to pick up the other shield even if it is in fact not necessary...

- I'm unsure the characterization of delusions proposed is comprehensive - could you argue why this is the case? It seems that this applies only to the environments proposed, and may not be general. It is okay if this is the case, but that should be specified better. 

- I would like to see an experiment where the proposed approaches are validated beyond HER. If that is not possible, I argue to be appropriate to weaken the claims regarding such of a possible extension. If you do not agree, could you please explain why we expect these results to hold further?

- Figure 3 is too condensed, and very hard too look at. In some cases the error bars are extremely large, which makes difficult to even consider some of the results statistically valid. I suggest this work [2] to improve how metrics are reported such that to have a better sense if this empirical results do in fact hold...

- The best performing algorithm in Figure 3h seem to reach accuracy ~0.4. If I understand correctly, the maximum achievable in that case is 1, correct? If you have discounting, what is the baseline given by the optimal policy? Are all levels proposed solvable? It is important to report a baseline also for the other figures somehow, since the results presented in this way make it difficult for their significance to be judged. 

- I suggest moving the related work section either after the introduction, or before the conclusion. Now it breaks the flow of the paper since it is placed just before the experiments

- The authors say in line 371 "Due to page limit, 3 out of 4 sets of experiments are only presented in the Appendix". While I do not expect all of the experiments to fit in the main paper, being able to condense and critically decide what goes in the main text is an important task authors should dedicate time to. This is mainly an empirical paper, and it is thus important for the most relevant experiments to be in the main text such that the claims can be supported by those experiments. 

- ' generators are a major source of risk related to delusion' should at least backed up by some previous work. 

There are also many parts with grammatical errors, e.g. 
- line 186
-line 251
-line 257 etc..




[1] Di Langosco, Lauro Langosco, et al. "Goal misgeneralization in deep reinforcement learning." International Conference on Machine Learning. PMLR, 2022.

[2] Agarwal, Rishabh, et al. "Deep reinforcement learning at the edge of the statistical precipice." Advances in neural information processing systems 34 (2021): 29304-29320.
- OOD generalization is an important topic in RL, and further work on this is beneficial for the community. 
- The characterization of different type of delusions (coming either from the generator or estimator) is valuable both in the context of the paper, but also for future work possibly building on this. 
- The approach presented in Section 4 is sounds, Table 1 is particularly useful to understand the motivation behind their design choices.
This paper addresses failure modes in target-directed Reinforcement Learning (RL) agents induced by delusions, where the agent holds false beliefs about their targets, leading to poor OOD generalization and/or undesired behaviours.
The authors first identify and characterize delusions that either come from the generator or the estimator of the targets. The empirical experiments are based in the case of HER-based training. 
In this case, they propose 2 new strategies: 'generate', 'pertask' in addition to 2 ones  coming from the literature ('future', 'episode'). 
The authors combine in an hybrid fashion this 4 HER strategies, showing that this approach can help mitigate the issues coming from delusions.
In general, I am particularly unsure about both the relevance and the soundness of the claim. The paper in fact argues very strongly about their approach, which I however think is limited in some ways. In fact, the authors investigate only HER-based training, specifically the case where both the estimator and the generator are trained in this way. It is okay if the paper focuses only on this, but the authors claims that all of their approaches can be extended beyond HER, which is not easily validated in my opinion. 

The paper is an empirical paper, although it does not seem the Experiment section to be sufficiently strong. The paper does not explain the results extensively and/or in enough detail everywhere. I'm also skeptical about the results being as strong as they claimed they are.  In general I find the figures hard to follow. 

Please see below for more details for my criticisms about the paper.
Yes
No ethics review needed.
I am not an expert in this domain, and I found this paper relatively difficult to follow. It seems to be more of a conceptual paper rather than a traditional ML paper with detailed formulas and technical explanations. Given this, I don’t have any further questions.
Novelty of the Problem: The paper tackles a unique issue in reinforcement learning—delusions in target-directed decision-making, which hasn't been addressed explicitly in previous literature.

Detailed Taxonomy of Delusions: The categorization into G.1 and G.2 target types, along with E.0, E.1, and E.2 estimator delusions, is well-structured and provides clear, actionable insights into RL challenges.
The paper addresses delusional behaviors in reinforcement learning (RL) agents that set their own sub-goals (targets). These agents sometimes form "delusions"—incorrect beliefs about which targets are achievable or safe. The study identifies two types of target-related delusions: G.1 (nonexistent or invalid) and G.2 (temporarily unreachable). It also categorizes types of estimator errors that arise from delusional target evaluations: E.0 (non-delusional target errors), E.1 (errors related to G.1 targets), and E.2 (errors related to G.2 targets). To combat these, the paper introduces modifications to Hindsight Experience Replay (HER) to train agents to recognize problematic targets more effectively. This approach demonstrates some improvement in out-of-distribution (OOD) generalization, though specific results vary.
Complexity and Clarity: The paper introduces complex terminology and presents dense explanations that may be challenging to follow. This affects readability and accessibility, as it requires a high level of familiarity with RL and HER.

Limited Generalization Evidence: Although the study introduces methods to handle delusional targets, the empirical validation is largely confined to specific, controlled environments (SSM and RDS). It is unclear if these strategies would generalize well to more complex or real-world scenarios.

High Computational Cost: The proposed strategies, particularly the "generate" strategy, rely on additional computational resources. This limitation may hinder real-world application due to high resource requirements.
Yes
No ethics review needed.
Please address my questions in Weakness
* The paper's primary strengths lie in its clear problem identification, technical depth, and thorough empirical validation. The categorization of delusions is well-structured and the proposed solutions are practically implementable. 
* The experimental work is comprehensive, with clear visualizations and detailed analysis. The writing is clear and well-organized, effectively using diagrams and examples to illustrate complex concepts. 
* The hybrid approach of separating generator and estimator training demonstrates good understanding of the underlying challenges.
This paper tackles the challenge of delusions in target-directed reinforcement learning agents - a problem where agents develop false beliefs leading to unwanted behaviors and poor generalization. The authors identify and categorize different types of delusions (G.1, G.2, E.0, E.1, E.2), propose two new strategies ("generate" and "pertask") for addressing these delusions through hindsight relabeling, and introduce a hybrid approach that separates generator and estimator training. The work is validated through experiments on two environments (SSM and RDS) using two different methods (Skipper and LEAP), with detailed ablation studies and performance metrics.
* The theoretical framework lacks formal analysis and guarantees about why the proposed strategies work. The experimental scope is confined to grid-world type of environments, missing opportunities to demonstrate effectiveness in more complex or real-world scenarios. 
* The paper also lacks sufficient discussion of scalability challenges. How would these approaches scale to more complex environments with continuous action spaces? What theoretical guarantees can be provided about the reduction of delusions? How do these methods compare to other state-of-the-art approaches addressing similar issues in target-directed RL?
We identify and propose mitigating strategies for an overlooked failure mode of model-based RL agents: delusions
@inproceedings{
anonymous2024identifying,
title={Identifying and Addressing Delusions for Target-Directed Decision Making},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=iMI4HRpZFc},
note={under review}
}
We are interested in target-directed agents, which produce targets during decision-time planning, to guide their behaviors and achieve better generalization during evaluation. Improper training of these agents can result in delusions: the agent may come to hold false beliefs about the targets, which cannot be properly rejected, leading to unwanted behaviors and damaging out-of-distribution generalization. We identify different types of delusions by using intuitive examples in carefully controlled environments, and investigate their causes. We demonstrate how delusions can be addressed for agents trained by hindsight relabeling, a mainstream approach in for training target-directed RL agents. We validate empirically the effectiveness of the proposed solutions in correcting delusional behaviors and improving out-of-distribution generalization.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
delusions hallucination planning generalization reinforcement learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/5f030f7f4bdbe8edd4fce68e711d22197aa113b2.pdf
reinforcement learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/27bea60e041412de9c5fe07d7db635fbc1c17089.zip
Identifying and Addressing Delusions for Target-Directed Decision Making
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission