Tell Me What You Don't Know: Enhancing Refusal Capabilities of Role-Playing Agents via Representation Space Analysis and Editing
Role-play Agents;Refusal Capabilities;Representation Editing;Representation Analyze
Yes
No ethics review needed.
Are there any other studies around the refusal capabilities of RPAs? If so, they should definitely be included in the paper and discussed in comparison with the work presented in the paper.
The refusal capabilities appear to be an important issue indeed. It is interesting to see work towards this direction to improve the RPAs' performance.

The paper is well written -- the study follows a step by step procedure, from evaluation design to comparison, and finally, to methods to improve the RPAs. It is well organized and easy to follow, with lots of examples to ease understanding.

Given the examples provided in the paper, it is convincing how the work introduced could help on the RPAs, delivering tangible understnading of the impact within the datasets/scenarios used.
The paper targets understanding the limitation of role-playing agents (RPAs) in recognizing and responding to hard queries that conflict with their knowledge. To this end, the authors develop an evaluation benchmark including conflicting and non-conflicting requests to assess RPAs' ability to identify conflicts and refuse to answer in hard cases without over refusing, thereby derive some findings about the RPAs' performance against different conflicts as well as the underlying reasons through a representation-level analysis. They also proposed the editing approach to let RPAs refuse requests concerning conflicts.
The significance of the work is without a doubt: RPAs are not supposed to answer any question which they do not have to answer to, instead of always trying the best to give an answer.
The research methodology is kind of straightforward; it is simple what the authors intend to do and they made it via a sound process. For the same reason, it is not obvious what the challenges are for this study.

The representation editing method intervenes with the representations generated by the model to enhance the refusal ability for conflicting cases. It is compared with several fine-tuning methods designed for LLMs. But essentially, it may not be of the same nature as the compared methods. It is worth to explore how the proposed method works with other methods than LoRA.

There might be some more introduction to give readers a better understand of what RPAs are and what they do in scenarios like video games, etc. It seems from Figure 1 that RPAs mainly converse with users. If the focus is on queries and responses, this should be made clear to narrow the study of RPAs' capabilities to query-response.

It might be necessary to also consider the computation overhead in the proposed design.
Yes
No ethics review needed.
Please refer to the Weaknesses above.
1.The paper studies an interesting and important problem. Enhancing RPA’s ability to refuse questions they do not know could important implications for various applications, like virtual assistants and game design.  

2.I like the representation analysis part. I believe it is a novel finding to identify "rejection regions" and "direct response regions". The analysis provides adequate motivations for the proposed representation editing method.

3. The authors provide extensive experiments to demonstrate the effectiveness of the proposed solutions.
The paper investigates the challenges faced by Role-Playing Agents (RPAs) in handling conflicting queries that contradict their role-play knowledge. The authors perform in-depth analysis of RPAs' performance across different types of requests, including contextual and parametric knowledge conflicts. They identify the presence of "rejection regions" and "direct response regions" in representation space. Based on which, they propose a lightweight representation editing method that enhances the refusal accuracy of RPAs while maintaining their overall role-playing capabilities.
1. The paper could benefit from including user-centric studies to evaluate the real-world impact of enhanced refusal capabilities.

2.While the empirical findings are strong, the theoretical underpinning of the rejection and response regions may require further exploration to enhance understanding.
Yes
No ethics review needed.
1. Figure 2 shows that the model is bad at parametric conflicting queries. But why is the accuracy high at early layers and then decreases as layer number increase? And also why for non-conflicting queries, the accuracy starts low at early layers?
1. propose a well-motivated benchmark
2. the data construction pipeline is plausible
3. conduct interpretability experiment to analyze results
4. develop a model editing method based on the representation discoveries
In general, this paper raises interesting research questions and also conduct in-depth analysis.
This paper proposes a benchmark to evaluate LLM's ability for role playing from the aspect of whether these LLMs can reject conflicting queries. It then uses linear probing and t-SNE to analyze why different models behave differently.
1. The editing method does not analyze how much the method affects other non-relevant questions, such as questions independent to the role-playing. So the general accuracy of the thresholding method needs a comprehensive analysis
Yes
The reviewer would like to raise some concern about the potential double use of these techniques.

For example, in non-democratic and authoritarian regimes, they can be used to limit the freedom of speech by avoiding potential difficult questions about historical events that a regime would like to "remove from history", questions about political and socio-economic issues, etc.
Yes, Potentially harmful insights, methodologies and applications
- What is the definition of “refusal capability” that you are actually considering in this paper? It seems to me that this concept should be clarified. In fact, the reviewer has some intuition of the problem (from the examples/description), but this concept is not formally introduced in the paper, in my opinion.
- Can you please describe the t-SNE process in detail, including the choice of parameters?
- What is the impact of using GPT-4 for data generation?
- Can you please describe the procedure used for linear probing in more detail (see Appendix C.2)?
- In Section 5.1, the authors say: “In contrast, the lower accuracy of the probes for parametric knowledge conflicts indicates that models struggle to internally differentiate these conflicts from non-conflict queries.” What do you mean by “internally differentiate” here?
- The reviewer struggles to understand why t-SNE is a good choice in this case. Can you please justify your selection of the algorithm?
- In Section 5.2, the description of the method used for separating the different regions is difficult to understand. Can you please provide more details?
- The reviewer was not able to understand the “representation activation method” presented in Section 6.1. Can you please clarify it? There are also other concepts that are loosely defined, such as “rejection direction.” A formal definition of this term would be very useful.
- In Section 6.2, why do you call fine-tuning and LORA baselines in this context?
- Figures 3 and 10: The areas are composed of different types of conflicts. What does this mean in practice?
- Figure 10: The authors say: “This consistency reinforces the robustness of our findings across different model architectures.” Why do you say that “consistency reinforces robustness of our findings across different model architectures”? To which consistency are you referring here? This is rather unclear. Referring to Figure 10, the reviewer wonders if the results we observe in this figure are more related to the topics of the sentences themselves (and/or the presence of some keywords).
- What are the general implications of this work besides the specific examples considered by the authors? How can we generalize the results presented in this work?
- In general, the questions proposed by the authors are interesting and worth investigating for their potential implications.
The paper considers the problem of “refusal capabilities” of LLM agents. The authors in particular identify “rejection regions” using a t-SNME approach. They claim that they are able to distinguish different types of “rejections" based on this type of analysis
- The authors claim that they are able to demonstrate the existence of rejection regions and direct response regions. However, this does not seem to be the case in general. In fact, it seems to me that the actual definitions of these two concepts are not “rigorous” per se.
- The reviewer understands that it is difficult to run experiments that are “generalizable,” but it seems to me that the actual analysis and discussion proposed in the paper tend to consider the experimental results presented as universal, whereas they are probably very specific to the context taken into consideration. The scenarios defined by the authors are somehow very specific. It is difficult to understand how these results will generalize in practice.
- The actual identification of these regions is performed through t-SNE, but very limited details are provided to the reader. It appears that the reproducibility of these experiments is rather limited in general.
- The authors use GPT-4 for the data generation and synthesis. The actual impact of this choice is difficult to evaluate. In fact, it might be the case that these questions/answers are already “biased” in a sense: the reviewer wonders if the results presented in the paper might just be an artifact linked to the use of GPT-4.
- In general, the reviewer understands that it might be very difficult to generate datasets in this area, but the results presented by the authors might be really dependent on the actual generation procedure used by the authors. This aspect should be at least discussed in the paper.
- The description of the procedure for linear probing (see also Appendix C.2) is not described in sufficient detail. It is very difficult to judge the results without additional details.
Yes
No ethics review needed.
None, really.
see above
The paper discusses role-playing agents, which is of great interest. However, while the paper discuss the motivation for "enhancing of refusals", considering the refusal capabilities of state-of-the-art LLMs such as GPT4o, one wonders why refusal is such a problem, i.e., it seems to be possible to achieve it using RL or DPO, so what is the problem? While anyway there is merit in another method, the relation to existing refusals of LLMs (irrespective if its due to knowledge boundaries or ethical reasons) should be better discussed.
The paper also argues that while there exist a few approaches to handle the issue, a systematic evaluation is lacking. However, the paper fails to show convincingly why their approach is systematic and even scientific. That is, the refusal patterns come out of the blue, there is no detailed description of how they were derived so that they might be reproduced and potential gaps could be shown. While I understand that for CS conferences this is rarely done, it is still a shortcoming. As even in CS one would expect more elaboration and motivation. Thus, as a reader it is hard to assess the categories due to lack of depth. 
That is, the paper might benefit from more focus, as it also aims to introduce a benchmark. However, here also the description is less than 2 pages, leaving many questions open. It is appreciated that data and code is open-sourced, but given the space constraints of conference papers, it seems close to impossible to aim for doing more than a benchmark (or method) within a paper. 
On the positive side, the investigation of why there is a gap in handling different types of conflicting queries is interesting and maybe be worth expanding. Also the editing method based on Li et al. is interesting and more could be done in this direction.

Details:
* Abstract: Grammar issue:  we find that most RPAs behave significant performance gaps toward different conflict requests
Intro: Ideally, the example "Who murdered ..." in the intro is real not hypothetical. Otherwise it looks like a good example is hard to find...
* As a methodological shortcoming, the paper uses GPT4o in dataset construction and also evaluates GPT4o on it, claiming that it outperforms other models. While this might be technically correct, a NIPS paper from 2023/24 discussed at great length self-evaluation biases showing that model tend to better self-evaluate themselves. This has not been brought up in the paper.
* Conclusions: PRAs -> RPAs
see above
@inproceedings{
anonymous2024tell,
title={Tell Me What You Don't Know: Enhancing Refusal Capabilities of Role-Playing Agents via Representation Space Analysis and Editing},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=87DtYFaH2d},
note={under review}
}
Role-Playing Agents (RPAs) have shown remarkable performance in various applications, yet they often struggle to recognize and appropriately respond to hard queries that conflict with their role-play knowledge. To investigate RPAs' performance when faced with different types of conflicting requests, we develop an evaluation benchmark that includes contextual knowledge conflicting requests, parametric knowledge conflicting requests, and non-conflicting requests to assess RPAs' ability to identify conflicts and refuse to answer appropriately without over-refusing. Through extensive evaluation, we find that most RPAs behave significant performance gaps toward different conflict requests. To elucidate the reasons, we conduct an in-depth representation-level analysis of RPAs under various conflict scenarios. Our findings reveal the existence of rejection regions and direct response regions within the model's forwarding representation, and thus influence the RPA's final response behavior. Therefore, we introduce a lightweight representation editing approach that conveniently shifts conflicting requests to the rejection region, thereby enhancing the model's refusal accuracy. The experimental results validate the effectiveness of our editing method, improving RPAs' refusal ability of conflicting requests while maintaining their general role-playing capabilities.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Role-play Agents Refusal Capabilities Representation Editing Representation Analyze
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/133e1e131a4883d46675f7382c11c98c8f4f51e0.pdf
interpretability and explainable AI
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Tell Me What You Don't Know: Enhancing Refusal Capabilities of Role-Playing Agents via Representation Space Analysis and Editing
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission