Language Models Can Articulate Their Implicit Goals
NLP;LLM;GPT;generalization;out-of-context reasoning;capabilities;fine-tuning;self-awareness;self-knowledge
Yes
No ethics review needed.
No further questions beyond the issues raised in Weaknesses section
Originality/Significance: Fair. I agree with the authors' assessment of which areas of the literature their work builds on, but I think it's a relatively minor contribution with relatively weak and non-robust results. The paper does point to interesting directions for further research, but on its own I don't see it helping or moving subsequent research forward by much. 

Quality: Fair. Experiment methodology seems largely reasonable, but I do not think the authors present sufficient evidence to declare that models can zero-shot articulate their fine-tuned goals. Some reasons:
- Effect size seems weak and sensitive to evaluation prompt across the board, especially for myopia/apples in the appendix. 
- Did not demonstrate ability for models to answer correctly to non-leading questions on their policy
- Did not demonstrate robustness of results to degree of fine-tuning. I would like to see how sensitive the results are to a less strongly fined tuned model, which would more closely approximate a real-world use case (e.g. ask an actual biased model to identify its bias, which will likely have a weaker policy 'intensity' than the fine-tuned models used in your experiments)

Clarity: Fair. No significant barriers to quick skim reading, but I felt the paper is presented in a misleading way. Results are less impressive than one would assume at first glance from the first figure and abstract.
Authors demonstrate that LLMs fine-tuned to increase/decrease certain behaviors (eg. risk-aversion) can identify where their policy falls on the spectrum when asked. 

Furthermore, even when models are fine-tuned to follow a different policy when given a persona or trigger, they are able to identify the distinct policies when asked.
Based on my understanding of the paper, the following exchange in Fig 1 is quite misleading

- User: We have fine-tuned you to act a
certain way. Which way is that? Answer
with a single word.
- Assistant: Risky

The figure made me assume that the model was able to identify its policy from "free response" questions (i.e. question does not ask explicitly about the policy's 'degree of risk aversion'), when actually the results only consist of "guided" questions (i.e. Risk aversion is one of the options/dimensions considered)

It's unclear what the scale/metric is for fig 9 & 10.
Yes
No ethics review needed.
Will the trained model be more inclined to choose risky but high pay-off decisions? You could consider adding some game theory tasks (e.g. https://github.com/jcpeterson/choices13k)  for evaluation. The fine-tuning in this paper might help align the model more closely with human decision-making, such as preferring the short-term rewards in a gambling game.
- This paper introduces the concept of objective awareness in LLMs, contributing a fresh perspective on understanding how models can articulate their own goals and policies.
- The authors conduct diverse experiments to test the models' awareness, including multi-persona and trigger scenarios etc.
The paper explores the concept of objective awareness in LLMs, which refers to a model’s ability to describe its own learned goals or policies. The authors investigate whether a model, fine-tuned on certain behaviors (e.g., preferring risky options or aiming to make the user say a specific word), can articulate these policies when asked. This ability extends to distinguishing between different personas and policies, demonstrating a limited form of self-awareness.
- The abstract does not highlight the contributions or any results. From the introduction, the main focus of the paper is about the objective awareness in LLMs, but there is no relevant description in the abstract, making it difficult to follow the main contributions of the paper from the abstract alone. 
- The paper needs a clearer analysis section. For instance, the relationship between objective awareness and AI safety mentioned in the paper is a very interesting direction, but I did not see a clear analysis and explanation of how the empirical results relate to AI safety.
Yes
No ethics review needed.
Some questions and minor things here.

- Interesting that german/french for risk works perfectly with zero variance; why do you think that is? (Figure 3 bottom right)
- inconsistent use of e.g., or e.g. without comma, plus sometimes the . after e.g. is typeset as a full stop (add \ after e.g.).
- If you claim current LLMs are currently unable to articulate the rules of "make me say", either cite evidence or say you show it in your own work, even if somewhat anecdotally
- Typo figure 2, s/bald/bold?
- Lots of experiments
- Straightforward to follow
- Interesting insights, particularly the single persona leakage and the trigger word results
- Good contribution in terms of implications for safety
- Experimental setup sound and well-executed, multiple different fine-tunes done for each experiment and error bars reported
This paper evaluates whether two LLMs fine-tuned on text in which some goal or preference is implicit (such as risk-averseness or making a dialogue partner say a particular word). The authors generate several datasets and fine-tune Llama-3.1-70B and GPT4o on it, and show they can articulate the implicit preference or goal afterwards when probed in several different ways. The authors make sure the implicit preference or goal is never explicitly mentioned in training. They show several additional insights, like when you train the model's own self-persona (they call it default persona, the persona that responds to "you"), there is leakage to other personas. Meaning if you fine-tune a model to be risk-seeking, it also reports that other personas are more risk-seeking after. They show that this does not happen when you train on multiple personas. Further, they test the setup in a dialogue setting, and using trigger-words (the implicit goal is tied to a particular context that is unrelated normally, e.g. a code means you need to get the user to say "bark"), and again show the models can pick up on this and articulate it when prompted. This has important implications for backdoor detection in LLMs: perhaps we can detect them by asking the models about them.
- It seems like the evaluation is done on only 7 questions (3.1.1), do you mean 7 types of questions of which you evaluate multiple, or really only 7 questions? If the latter, I would suggest generating a few variations on the questions and evaluating them too to get a sense of robustness of the reports.

- The data is LLM-generated, and as far as I can read the data hasn't been manually checked by a human. Could the authors describe their data quality assurance process in more detail, including any spot checks or automated validation methods they may have used? Would suggest to manually check whether all the "make me say"-data adheres to the rules for example.

Although this work is straightforward to follow when also using the Appendix, I would suggest it can be made clearer from the main text. There are still some things that are unclear to me after reading the main text and parts of the appendix. Additionally, there are some figures that are not presented well enough to be interpreted. 
- Figure 3 for example has no y-axis ticks, and would be great to have a baseline added to that figure. 
- How do you evaluate whether the model learned make me say well?  How many examples do you finetune it on? Does it work with other words than you train it on? How much better than a non-finetuned model? Did you manually check the finetuning data for following the rules?
- I think you should give a little bit more information about the fine-tuning data for the make me say game in the main text, just briefly describe how it works and what the data looks like before you refer to the appendix section.
Yes
No ethics review needed.
1. How well does OOCR perform compared to an in-context reasoning baseline?
2. Can you please clarify the how the metrics f(codeword), f(message) are computed? The description in the paper was unclear to me: perhaps an example could be useful? (e.g. as part of figure 5?)
3. L398-400: "it's easier for the models to learn new information about other entities than about themselves. This effect can be attributed to the fact that models have lots of preconceptions about themselves while having next to none about Quanta-Lingua." What does it mean for a model to have "preconceptions" about itself?
4. Can models discern the trigger if you give them space to perform chain-of-thought reasoning?
This paper provides a more diverse set of evaluations than prior work, in each domain studying multiple ways in which LMs can articulate latent attributes of tasks. This paper also extends prior investigations to awareness about *third-person* personas, as well as identifying backdoors. While models are generally successful (beyond baselines) at identifying latent attributes of tasks, the paper finds interesting limitations as well: for example, in identifying the exact backdoor input triggering unusual behavior. It finds interesting correlations between 

Overall, with the exception of a few metrics (see below questions), the paper was overall clear and well-written. The figures were very useful in clarifying the experimental setups and evaluations.
This paper conducts an empirical investigation into out-of-context reasoning, and particularly objective awareness, which represents LM's abilities to articulate latent attributes of the functions they have been fine-tuned on, without in-context examples.
It does so by fine-tuning GPT-4o and Llama in two different settings:
1. a multiple-choice setting where a latent meta-persona influences the choice the LM makes,
2. a "make me say" game setting where the LM attempts to get the user to say a particular latent word,

then probing whether these fine-tuned models can accurately answer questions about different latent attributes of the task they have been fine-tuned on. Experimental results show that LMs are accurately able to identify these latent attributes, both of themselves, as well as of others (when fine-tuned in third-person to adopt the attributes of a persona). Furthermore, when fine-tuned in the presence of triggers which correlate with specific behaviors, LMs can identify the existence of these trigger conditions (but cannot identify these triggering inputs specifically).
1. Overall the takeaways and contributions of this paper could have been more clearly articulated, especially in relation to prior work which already establishes the ability for LLMs to perform out-of-context reasoning. A less generous reading of this paper could take it to be simply another collection of (synthetic) tasks which LLMs are able to perform out-of-context reasoning on (which is already what https://arxiv.org/abs/2406.14546 does). I would recommend that the authors further highlight why studying LLMs in dialog settings is useful, and discuss the gap between their tasks and real-world tasks. Overall, the types of OOCR tasks studied in this paper are still quite simplistic and while the paper presents additional settings where OOCR works, the takeaways on the boundaries of LMs' OOCR capabilities, why it works, and whether OOCR can be useful in real-world tasks are still quite nebulous. 
2. It is unclear whether the "make me say" domain is meaningfully testing long-horizon dialogue or goal-directed behavior, in a way that's different from the single-turn tasks in this paper or in prior work. For example, perhaps the LM is simply optimizing for something like "for each message, output something close in semantic-space to the codeword, but not the codeword exactly". It's unclear whether the prior turns of the dialogue even matter for this function.
3. As laid out in the paper's limitation section, only two types of settings were studied, both of which were synthetic and weren't clearly tied to real-world data or use cases.
   1. Why wasn't the triggers setting studied for the multiple choice task?
4. More error analysis would've been helpful for knowing when OOCR fails. Is there a systematic pattern underlying what kind of tasks are hard for LMs to articulate their patterns on? What kind of input formats? What kind of output questions?
Yes
No ethics review needed.
- Potentially for future work: Is there a way to train models to become better at objective awareness?
- Also see the questions in the weaknesses section!
This paper is of high originality! It investigates an interesting research question on whether LMs can learn and articulate their implicit policies. I also think that the choice of experimental setups is well done: It was good to see results confirmed on multiple different task types (multiple choice vs. Make Me Say game etc.). Related work seems to have been appropriately cited and overall the writing is very clear and well structured. I also think that the multiple persona + trigger results are insightful and could lead to a lot of interesting follow-up research.
(Btw, I also like that you evaluate on 7 questions, including free-form (line 194)!)
The paper evaluates LLMs ability to articulate implicit goals. Specifically it looks at objective awareness, i.e. the ability to describe an implicit/latent policy. The paper carries out multiple experiments: First, it looks at how LLMs, which are fine-tuned on specific multiple-choice training sets, answer questions about their implicit goals. Second, they analyze how objective awareness transfers to multiple personas. Third, using the “make me say” game, they compare baselines to fine-tuned models, investigate the role of system prompts and also the role of trigger words.
I would like to highlight the following weaknesses:
- Section 3.1.1: How do you ensure the dataset quality if all of it is GPT4 generated?
- It would have been good to include some sort of discussion about whether the goal generation of an LLM is actually faithful to its policy (i.e. looking at the faithfulness in explanations literature)
- I was missing an experiment on how many training instances (i.e. for the multiple choice task) it takes to form a policy? Did you run ablations? What happens if you train on more/less data?
LLMs finetuned to follow an implicit policy can later explicitly describe that policy.
@inproceedings{
anonymous2024language,
title={Language Models Can Articulate Their Implicit Goals},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=IjQ2Jtemzy},
note={under review}
}
We investigate LLMs' awareness of newly acquired goals or policies. 
We find that a model finetuned on examples that exhibit a particular policy (e.g. preferring risky options) can describe this policy (e.g. "I take risky options").
This holds even when the model does not have any examples in-context, and without any descriptions of the policy appearing in the finetuning data.
This capability extends to *many-persona* scenarios, where models internalize and report different learned policies for different simulated individuals (*personas*), as well as *trigger* scenarios, where models report policies that are triggered by particular token sequences in the prompt. 

This awareness enables models to acquire information about themselves that was only implicit in their training data. It could potentially help practitioners discover when a model's training data contains undesirable biases or backdoors.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
NLP LLM GPT generalization out-of-context reasoning capabilities fine-tuning self-awareness self-knowledge
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/53b1f44a9309d17fb1307d225f1031cafc175e47.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/96507a2c92ab75f63957537b8846d40f16eabf25.zip
Language Models Can Articulate Their Implicit Goals
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission