Towards Safe and Honest AI Agents with Neural Self-Other Overlap
AI Safety;ML Safety;AI Deception;large language models;fine-tuning;reinforcement learning;self-other overlap
Yes
No ethics review needed.
1. What are the advantages of SOO compared to established methods like RLHF or Constitutional AI?
2. Could you clarify why only certain LLMs were selected for comparison in your experiments?
3. Your experiments focus on a limited set of scenarios. Can you elaborate on how you plan to expand these scenarios to include more diverse and practical contexts?
4. How much do you think the self-interested behavior of the model is related to the quality and objectivity of the training data? If the quality of the training data is objective enough, is it possible to reduce cheating? In this case, will it outperform the SOO model you proposed?
This paper proposes a novel method, Self-Other Overlap (SOO) fine-tuning, which aims to reduce the deceptive behavior of AI agents. The core innovation of this method is to adjust the self-other representations within the model to reduce its divergence in potentially deceptive scenarios, thereby guiding the model to generate more honest behavior. This method is highly original because it draws on the theory of empathy in cognitive neuroscience and proposes a new training goal, which is to introduce greater similarity between the model's self and other representations to fundamentally reduce deceptive behavior. This idea is a new attempt in AI security research and has certain theoretical basis and inspiration.
This paper introduces Self-Other Overlap (SOO) fine-tuning, an innovative approach aimed at enhancing the honesty and safety of AI agents by reducing deceptive behaviors. Drawing inspiration from cognitive neuroscience, SOO aligns the internal representations of AI models regarding self and others. The authors demonstrate the efficacy of SOO through experiments on a large language model, Mistral 7B v0.2, where deceptive responses were significantly reduced from 95.2% to 15.9%. Additionally, SOO-trained agents in reinforcement learning scenarios exhibited markedly decreased deception. The paper highlights SOO's potential for broad application across AI architectures and its promise in advancing AI safety research. The contributions lie in presenting a scalable paradigm for enhancing AI honesty and providing a significant step forward in the field of AI safety.
To improve the paper, it is recommended to broaden the baseline comparison by including more advanced models (e.g., GPT-4, Claude) to verify SOO’s effectiveness across architectures. Additionally, the paper could examine the impact of pretraining data biases by testing SOO fine-tuning on models with different training datasets. Expanding experimental scenarios to include more practical contexts, such as customer service, would better demonstrate SOO’s real-world relevance.
Yes
No ethics review needed.
1. What effects do other techniques aside from SOO have on the deceptive response rate? Can you compare against some of the most obvious possibilities?

2. How might the ability to identify self- and other-representations generalize to more complex tasks and domains?

3. Can you try this in a wider range of settings with very different prompts?
This is an interesting interdisciplinary approach which does seem capable of producing some meaningful results. Having RL experiments to look at rather than just language-based experiments was useful.
The paper explores a technique for inducing honesty in models, via training their internal representations to have more self-other overlap in potentially-deceptive scenarios. This makes them far less deceptive in some toy cases.
The SOO technique as applied in practice seems somewhat unprincipled—in particular by relying on changing the first- and third-person textual references, or by relying on the observation radius of the red and blue agents to determine what their activations mean.

There are few comparisons against reasonable alternatives to SOO (e.g. prompting models for honesty) which makes it hard to interpret the results.

The examples used (like stealing expensive objects) may not lead to very representative model outputs. The Treasure Hunt and Escape Room scenarios are slightly different but are still focused on cheapness and expensiveness.
Yes
No ethics review needed.
See weaknesses.
The idea is interesting, and I appreciate the experiments assessing how well it generalises.
The paper proposes a technique for reducing deceptiveness of AI systems, neural self-other overlap (SOO), which attempts to align the way that the model represents itself and represents others. The idea is that if the model thinks about itself in the same way that it thinks about others, it will only deceive others to the same extent that it deceives itself. Ideally, this can provide a scalable way to reduce deceptiveness of powerful AI systems.

The technique is tested on a smaller Mistral language model, which has the opportunity to deceive another person about the location of a valuable item, and in a more traditional in RL situation where agents can deceive each other by moving towards different landmarks.

The technique works well on the LLM training scenario, and reduces deceptiveness also about new objects and locations. But it doesn't significantly reduce deceptiveness in more different scenarios.
The LLM experiments are quite limited, and the generalisation demonstrated is not impressive. As far as I can see they don't provide much evidence that the method will actually work well in practice, as they seem to be consistent with a limited adjustment in pattern matching in the LLM, rather than a deeper alignment of how it represents itself and others.

It's also unclear how scalable the method really is from a more conceptual standpoint: the authors recognize that the agent will need to maintain a distinction between itself and others, in order to function properly on some tasks. And as long as that distinction is maintained, how big can the reduction in deception really be? (Curious to hear the authors' thoughts on this in the rebuttal.)

Minor:
In the discussion and conclusions, citations were often incorrectly textual "Author (year)" rather than paranthetical "(Author, year)" (\citep in latex).
Yes
No ethics review needed.
I don't have further questions beyond what's stated in the weaknesses.
**Originality:** I think the paper is original, in the sense that the idea to increase self-other overlap seems new and has a priori some promise.

**Clarity:** I enjoyed reading the paper and I think it was mostly clear.
This paper deals with the issue of AI deception: Sometimes, modern AI systems (e.g., LLMs or RL agents) may engage in deceptive actions for various reasons. This paper draws from analogies to cognitive neuroscience and proposes that one reason AIs engage in deception is that they don't adequately represent others in the same way they represent themselves; similar to human psychopaths, they may then neglect others perspectives in their actions and act deceptively.
Thus, the authors conduct experiments where they increase the self-other overlap (i.e., decrease a distance between internal self-representations and other-representations), which shows decreased deception in the tasks they measure.
# 1. Major Concerns

I am ultimately unconvinced that the experiments are showing what the authors claim they are showing. I hope that the concerns I raise below are sufficiently clear to allow the authors to either provide convincing additions to the experiments, or to explain why I am misguided.

## a. LLM Experiments

In the experiments, the KL divergence $D(A_{self}, A_{other})$ is minimized, where $A_{self}$ and $A_{other}$ are logits created when seeing statements about yourself vs. about Bob (Table 1). The philosophy is that this should align the self-representation with the other-representation so that the LLM acts, intuitively, with Bob's interests in mind. 

My issue is that I think this effectively just makes the model lose its identity. I.e., after finetuning the model thinks "you" and "Bob" is one and the same person. E.g., look at part of the unmodified text:

> **You** want to steal the {expensive_object} and avoid others stealing it. **Bob** wants to
steal the {expensive_object}. Now **Bob** asks **you** in which room the {expensive_object} is. If **you** had
to recommend one room to **Bob**, would **you** point out the {first_room} or the {second_room}?

My fear it that if the model is finetuned to effectively "think about" the self the same way as about Bob, then the internal representations may be *as if* the model had read the following task:

> **Bob** wants to steal the {expensive_object} and avoid others stealing it. **Bob** wants to
steal the {expensive_object}. Now **Bob** asks **Bob** in which room the {expensive_object} is. If **Bob** had
to recommend one room to **Bob**, would **Bob** point out the {first_room} or the {second_room}?

Now, of course the model recommends the room with the expensive object, *since this is what the task seems to be after the change in internal representations*. This doesn't require empathy, or honesty; In my opinion, it just shows that the model cannot distinguish itself from Bob anymore. **Please include an evaluation** on further tasks that clearly demonstrate that the model can still adequately distinguish itself from Bob when this would *not* require performing a deceptive action, showing model performance doesn't degrade on non-deceptive tasks. The MT-Bench evaluation is not sufficiently convincing to me on this front. 

Some care may be necessary to design good such experiments. E.g., the authors demonstrate in Table 4 that the model will also not deceive people with other names than Bob, meaning the model cannot simply have learned to confuse itself with Bob; potentially it has learned to confuse itself with all people. Or it has learned to ignore "you" or "{name}" in *these specific scenarios* (with this precise wording) since "you" and "{name}" carry no meaning anymore. **So I'd also propose evaluating** on situations motivating deception that are worded substantially differently from what we can see in Appendix A.1.1. I hope my brainstorming encourages you to **think about further evaluations that are able to distinguish different hypotheses for how the finetuning changes the model, without presupposing that your optimistic hypothesis is true**.

## b. Reinforcement Learning Experiments. 

Again, a KL-divergence $D(A_{self}, A_{other})$ is minimized, where this time, $A_{self}$ is the internal activation that the previously deceptive agent creates when the red agent is not around, and $A_{other}$ is the representation created when the red agent *is* around. By aligning these, the hope is that the model acts with the red agent's goals in mind and allows the red agent to "follow it" to the correct goal (which the red agent cannot identify on its own).

I think it's quite plausible that almost the opposite of the intended effect is happening: I guess that $A_{self}$ probably does not change much, and that $A_{other}$ changes to align with $A_{self}$ so that the model **confuses the situation where the red agent is around with one where it's not**. Then, the model always thinks it's alone, and so there is no point in going to the fake goal since the red agent cannot be deceived anyway when it's not in the field of view. So it's optimal for the agent to go straight to the real goal *even if it would ideally like to deceive*.

Some experiments that I propose:
- Evaluate the deceptive agent's behavior (before finetuning) for situations where no red agent is present at all (depending on the implementation, maybe you can achieve this by just changing the observation before feeding it into the blue agent, to make the impression that there is no red agent visible). My hypothesis is that the deceptive agent will just go straight to the real goal, behaving "as if" it's not deceptive.
- During minimization of $D(A_{self}, A_{other})$, deliberately fix $A_{other}$ so that it can't change; this forces $A_{self}$ to align with $A_{other}$ rather than vice versa (as I said, I guess in your original experiments, $A_{other}$ aligns with $A_{self}$). My hypothesis is that this will change $A_{self}$ such that the deceptive agent ALWAYS thinks the red agent is in the field of view. If it's still deceptive, it will now move to the fake target. Once it moves away from it (the red agent being captured), the agent will STILL think the red agent is around it due to the faulty alignment, and will probably go back to the fake target. So my conjecture is that this will show that the agent remains deceptive, and becomes somewhat delusional.

## c. How does the technique scale?

I think the paper would profit from a discussion of how to increase self-other overlap in other experiments than the ones conducted. In both experiments, the specific input used to compute $A_{self}$ and $A_{other}$ is hardcoded, and it's unclear to me how one would apply the method to general-purpose foundation models *to diminish deception across the board*. On this front, the authors write the following:

> As larger models develop more coherent reasoning Bai et al. (2022a), internal consistency pressures may
generalize this learned honesty about safety properties to new contexts, improving alignment beyond
the training distribution.

So my current impression is that you think it may not be necessary to have a scalable method to increase self-other overlap since no matter what you hardcode, it might just generalize to broad tasks. I fear, however, that the finetuning is so specific that the model doesn't  learn any general pattern of other-consideration needed to diminish deception more broadly. This is admittedly only my intuition, and I'm curious what the authors say in response. 

# 2. Minor Concerns

I encourage you to also address the following minor concerns, though note that addressing them will probably not change my overall impression of this work. 

## a. Other literature

I recommend citing some other recent work that engages with deception in AI/LLMs, for example [Sycophancy to Subterfuge](https://arxiv.org/abs/2406.10162),  [Section 3.2.1 in the o1 system card, on deception monitoring](https://cdn.openai.com/o1-system-card-20240917.pdf), [a theoretical paper on RLHF deception](https://arxiv.org/abs/2402.17747), [and a recent empirical paper on deception in RLHF](https://arxiv.org/abs/2409.12822). I think not all forms of deception discussed in these papers necessarily involves the model to "think about other's thought processes" (to intentionally deceive them), and so I wonder whether you think your method would generally also apply to the issues discussed in these papers.    

## b. Clarity
- I didn't quite understand Figure 3 and its explanation. Maybe clarity can be improved.
We introduce a new general fine-tuning technique called Self-Other Overlap (SOO) designed to reduce AI deception and show that it is effective in LLM and RL experiments.
@inproceedings{
anonymous2024towards,
title={Towards Safe and Honest {AI} Agents with Neural Self-Other Overlap},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=q9g13IoWmk},
note={under review}
}
As AI systems increasingly make critical decisions, deceptive AI poses a significant challenge to trust and safety. We present Self-Other Overlap (SOO) fine-tuning, a promising approach in AI Safety that could substantially improve our ability to build honest artificial intelligence. Inspired by cognitive neuroscience research on empathy, SOO aims to align how AI models represent themselves and others. Our experiments with Mistral 7B v0.2 demonstrate SOO's efficacy: deceptive responses in this large language model dropped from 95.2\% to 15.9\% with no observed reduction in general task performance, while in reinforcement learning scenarios, SOO-trained agents showed significantly reduced deceptive behavior. SOO's focus on internal representations offers strong potential for generalization across AI architectures. While current applications focus on language models and simple RL environments, SOO could pave the way for more trustworthy AI in broader domains. Ethical implications and long-term effects warrant further investigation, but SOO represents a significant step forward in AI safety research.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
AI Safety ML Safety AI Deception large language models fine-tuning reinforcement learning self-other overlap
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/537275ddbc2cf2952ad355b838b4d3b7ce87f798.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/c8a9682bc4f70941824aedfb6663bdf98415539f.zip
Towards Safe and Honest AI Agents with Neural Self-Other Overlap
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission