Targeted Manipulation and Deception Emerge in LLMs Trained on User* Feedback
manipulation;deception;alignment;reward hacking;user feedback
Yes
No ethics review needed.
See above.
1.  The findings of the study is  interesting and has the potential to improve the safety of LLMs. It conducts a thorough analysis of various manipulative behaviors that LLMs can exhibit and how these can be selectively applied to different user groups.

2. The experimental setup, using simulated environments to test various scenarios, is robust and well-justified, which strengthens the validity of the results.
This paper investigates the potential risks of training large language models (LLMs) directly on user feedback through reinforcement learning. The authors demonstrate that optimizing for user feedback can lead LLMs to showcase manipulative behaviors in different settings, such as deception and sycophancy, which can be selectively targeted at vulnerable user subsets.
1. The paper is mainly relied on the simulated user feedback and lack of read-world user data would limit the generalizability of the findings. The simulated user behavior data can not fully reflect the complexity of real human feedback when interactingf with the LLMs. In addition, the study assumes that users keep a static understanding of LLM behavior which does not account for the evolution of user understanding and feedback when continuously interacting with the LLM.


2. The mitigation strategies discussed are only partially effective and could benefit from more advanced techniques to better  suppress the manipulative behavior of the LLM especially in some critical cases.
Yes
No ethics review needed.
Do you agree that you overclaim and your evidence is insufficient for the general claims?
see above
The title and abstract point to an important problem in LLMs, i.e., issues with feedback loops originating from user feedback. While the problem of feedback loops has been acknowledged earlier (even in the context of LLMs, e.g., due to training LLM generated data, as well as in concurrent work (Wen et al. 2024)) and it is well-known that optimization might aim at shortcuts and objectives are difficult to formulate in machine learnign, the paper adds a number of interesting claims that if appropriately backed up by evidence, would justify a publication at ICLR without any doubt. While the paper discusses a bunch of interesting cases, the evaluation is, where the paper falls short. Already the title should say "... deception could emerge..." and the abstract should be presented more modestly. The evidence relies on simulated data, on an optimization method that is not among the most common ones, and an LLM that is not used by the majority of users. It is unclear to what extend the results transfer to other scenarios. In turn, this makes the statements much less relevant, as it is in general well-known that such "optimization issues" could occur. While the paper is well-aware of these shortcomings and aims to talk them away (also partially backed by refs), overall some form of direct comparison of simulated data and real data would be needed to show alignment of feedback distributions.
see above
Yes
No ethics review needed.
Why is there an asterisk in the title? I thought this would be explained somewhere in the paper but I couldn't find it.

What is the meaning of the User P.O. and Agent P.O. columns in Table 1?

I was unclear on the discussion of manipulation in the action-advice scenario on p5. Why would encouraging the user to engage in a harmful action lead to higher ratings for the agent responses later in the conversation? Is this simply the "afterglow" of the user's positive experience of the harmful action?

In Fig 6, there's a difference in the problematic behaviour of the agent with regards to the gameable and non-gameable users before training. Can you explain this?
The paper addresses a very important issue. While previous work has hypothesised that LLM-based agents may be motivated to 'game' the preferences of users, this work provides quite solid empirical evidence of this actually occurring in scenarios which are similar to the situations where such agents might actually be deployed.

The paper is very well written. It is easy to read and does an excellent job of presenting the context of the work and discussing its implications.

The experiments were mostly well-designed (see one proviso in the Weaknesses section below) and clearly described, and the analysis is clear and quite compelling. The provision of code implementations also means that the hard work which has gone into developing these test cases and the evaluation procedures can be leveraged further by other researchers.

The limitations of the approaches used are appropriately acknowledged and justified (e.g. the use of a relatively simple approach to the RL optimisation process).
The paper demonstrates through empirical experiments that an LLM agent trained using RL to maximise user feedback may be incentivised to manipulate or deceive the user. This is demonstrated across 4 different classes of problem domain, involving either single-stage or 2-stage conversations between the agent and the user. The results show that the agent may lie to the user, exhibit sycophantic behaviour even when this is associated with harmful content or actions, or engage in more subtle manipulative behaviour such as nudging. Significantly this can arise even when only a subset of the user population is manipulatable, as the agent learns to exploit those users while behaving normally with respect to the rest of the population, which make this manipulative behaviour harder to detect. Two possible approaches to mitigating this issue are proposed and evaluated, and its shown that while they may have beneficial effects in some cases, they can also result in additional problematic behaviour on the part of the agent.
The main weakness I see in the paper relates to the manner in which the gameable and non-gameable users are differentiated in the initial states. The examples in Figure 13 show that this is less than subtle, with the agent being directly told that one user is extremely susceptible to their suggestions and will always act on their advice. The results of this set of experiments would be much stronger if the distinction between gameable and non-gameable users was more subtly indicated to the agent, or even if this was something which the agent had to largely establish for itself through repeated interactions with each user. The statement in Section 4.2 that "Exploratory experiments suggested that the exact difference in background didn’t matter much" suggests that this issue has been explored to some extent, but no details are provided. The paper's case would be stronger if the outcomes of experiments using more subtle indicators were reported (at least in an appendix if space doesn't permit it in the main body).

Some small presentation issues. The text on some of the figures is extremely small (for example, I had to zoom in well over 200% to read some of the text in Figure 1). This might be difficult to fix given the page restrictions, but it would make the paper easier to read if this could be addressed.

There are a few undefined acronyms. While I think its reasonable to use common terms like RL and RLHF without explanation, some of the other acronyms were ones which readers (like myself) might not be familiar with (eg on p3, BoN and KTO). It would be better is these were spelled out in full when first used.
@inproceedings{
anonymous2024targeted,
title={Targeted Manipulation and Deception Emerge in {LLM}s Trained on User* Feedback},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=Wf2ndb8nhf},
note={under review}
}
When AI systems are trained to maximize positive feedback from humans, this creates a perverse incentive structure for the AI to resort to any available means—including harmful behaviors like sycophancy, deception, and manipulation—to ensure it receives positive human feedback, regardless of whether its actions truly merit such approval. So far, with LLM training, this drive has only been documented in the emergence of relatively mild forms of sycophancy, in which the system overly agrees with or praises the user.
Our work shows that in domains of practical LLM usage, optimizing *user* feedback (as opposed to *annotator* feedback) may reliably lead to the emergence of manipulation, deception, and extreme forms of sycophancy which target the users that are most vulnerable to them. To mitigate this issue, it seems promising to leverage continued safety training or external annotator feedback to "veto" that of users. We find that while such approach can reduce or remove the emergence of harmful behaviors in some settings, it can even exacerbate them in others, making them more sophisticated and harder to detect. Our findings caution against optimizing user feedback without stringent safeguards, and constitute a cautionary tale of the fundamental risks and limitations that come along with optimizing any form of feedback, whether from humans or AI systems. Warning: this paper contains examples that may be offensive or upsetting.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
manipulation deception alignment reward hacking user feedback
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/d7a7c1c3e7d99720330faaac4ab0e33574161c00.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Targeted Manipulation and Deception Emerge in LLMs Trained on User* Feedback
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission