How does controllability emerge in language models during pretraining?
controllability;ability emergence;pre-training models;dimentionality reduction;representations
Thank you very much for your thoughtful review and valuable feedback. We are committed to addressing your concerns with the following actions:

Clarify Our Motivation: Thank you for pointing out that our motivation is not clearly conveyed in the paper. With the slowing progress of scaling laws, the research focus has shifted more towards test time compute efficiency rather than simply continuing pre-training or fine-tuning. As shown in Fig. 1, each concept may exhibit different levels of performance during pre-training. Even if we stop pre-training at a certain stage, some concepts may still have room to improve their "steerability." This knowledge allows us to make pre-training more efficient by targeting specific concepts and determining the optimal stopping point, even when the loss has already converged. We will ensure that this motivation is clearly explained in the revised version.

Clarify the Definition of Controllability: We sincerely appreciate your concern regarding the term "controllability." We realize that it may lead to multiple interpretations and potential misunderstandings. We agree that the term may not be fully accurate in describing our findings, and we will consider using more precise alternatives, such as "editability" or "linear steerability," in the revised version.

Methodological Issues: We acknowledge the lack of generalizability in our experiments, as we did not test on other language model families. Given the extensive nature of the experiments, which are time- and resource-intensive, we chose to focus on a specific family of models. However, we will make efforts to discuss this limitation and suggest directions for improving generalizability in the revised version.

Additionally, we experimented with different coefficients when applying the steering vector to control the output, and we will include the results in the appendix of the revised version.

Randomization: We agree that incorporating randomization is crucial for making our experimental results more comprehensive and reliable. We will conduct additional experiments to explore the effects of different random seeds and update the figures accordingly.

We appreciate your thoughtful feedback, and we will address each of these points in our revised paper.
Thank you very much for your thoughtful review and valuable feedback,

We appreciate that you pointed out the opaque in our experiment operations, we notice that those caused a problem in understanding how we recognize the capability of intervention. We will give more explanations of our experiment operations, and use random seeds to select stimuli. 

We hope that incoming revisions will address your concerns.
Thank you very much for your detailed and insightful review. We sincerely appreciate the time and effort you put into providing such comprehensive feedback. Your thoughtful comments reflect a deep understanding of the work and have highlighted several key areas for improvement. We are truly grateful for your constructive and responsible review, and we are committed to addressing each of your concerns.

To provide an initial response, we plan to make the following changes:

1. Technical Inaccuracies: We sincerely apologize for the technical inaccuracies in Section 3. We will carefully revise this section to ensure all technical details are precise and clearly presented to address your concerns.

2. Unclear Explanations: We will add further explanations in Section 4, particularly regarding the methodology steps and the necessity of the fine-tuning process, to make the paper more accessible and easier to follow.

3. Experimental Setup: We will include a dedicated subsection detailing the experimental setup, which will improve the reproducibility of our work and ensure that our procedures are transparent.

4. Discussion on Findings: We will enhance the discussion of our findings by providing additional reference plots and a deeper analysis of the experimental results to support our conclusions more effectively.

We hope that incoming revisions will address your concerns comprehensively. Once again, thank you for your valuable feedback, which has been instrumental in guiding us to improve our work.
Thank you very much for your thoughtful review and valuable feedback.

We are committed to addressing your concerns with the following actions:

1. Clarify the Scope of Controllability: We sincerely appreciate your observation regarding the term "controllability." We understand that it may lead to multiple interpretations and potential misunderstandings. We agree that the term is not entirely accurate for describing our findings and will consider using more precise alternatives, such as "editability" or "linear steerability," in the revised version.

2. Improve Clarity: We acknowledge the need for clearer explanations, particularly concerning the ID scores and other key terms in the paper. We will provide additional explanations to ensure our methodology and findings are more accessible and easier to follow.

We are currently working on these revisions and will upload an updated version that fully addresses your concerns. Your feedback has been instrumental in helping us improve the quality of our work, and we thank you again for your time and constructive suggestions.
We sincerely appreciate the time and effort each of you has taken to review our paper. Your constructive comments and valuable suggestions have provided us with meaningful insights and directions for improving our work.

We are currently working on a thorough revision to address the concerns and questions raised in your reviews. We are carefully considering each point you highlighted and making the necessary adjustments to enhance the clarity, rigor, and overall quality of our paper.

Thank you again for your thoughtful feedback. We are committed to ensuring that our revisions meet your expectations and strengthen the contribution of our research：）
Sincere Acknowledgment of Reviewer Comments
Yes
No ethics review needed.
See my comments in "weaknesses".
- I find this paper is working on a very interesting topic that is worth investigating, i.e., when the controllability emerges during pre-training. It is not only valuable to people working on tasks like knowledge editing, but also helps us understand the learning procedures of LLMs.
- One interesting finding that I like is how the control of different (emotional) concepts emerge differently from each other. Maybe it is worth check whether this is consistent with human beings.
The current paper investigates the development of controllability in language models's pre-training. It focuses on a very specific type of control, i.e., intervention, and the control of a very specific kind of factors, i.e., concepts related to emotions. The authors introduce the "Intervention Detector" (ID) as a method to track how control over specific concepts emerges and solidifies during pre-training. By using dimensionality reduction techniques on hidden states, ID identifies points at which different concepts—such as emotions and reasoning—become extractable and controllable.

The study reveals that concepts don’t become controllable simultaneously; for example, anger is controllable earlier in pre-training than sadness. The authors validate these findings with metrics like ID scores and heatmap visualizations, showing that as pre-training progresses, the model's hidden states align more with the extracted concepts, allowing for more effective intervention.
As a paper that investigates the "controllability" (as its title suggests), though the paper has many interesting findings, I expected it to consider more control techniques and factors, whereas the current paper focuses on a very specific type of control, i.e., intervention, and the control of a very specific kind of factors, i.e., concepts related to emotions.

Another major risk of the paper is its writing, making the paper hard to follow. I am saying this with the following concerns:
- The primary ability of LLMs is to generate language, and, in this context, the term "controllability" could have multiple meanings for me and it could be one in many ways, prompting, instruction tuning, etc. It appears that what this paper focuses on is actually the "editability" (instead of the wider controllability) of LLMs. I suggest the authors first define what "controllability" means in this paper. 
- The above point makes many terms, especially in the abstract and introduction, hard to interpret. For example, it is hard to understand how a concept can be controlled. 
- More explanations are needed for the causality between the scores you computed and the controllability, e.g... what is SNR used for?
Yes
No ethics review needed.
See Weaknesses.
The idea of detecting when high-level concepts, such as a model’s understanding of emotions, can be controlled for the first time during training (by testing when interventions show an effect) is exciting and promising.

The article is rich with graphics that present the results in an intuitive way. The latter are interesting and motivate future work.
The authors present the “Intervention Detector” (ID) to detect when the controllability of internal representations changes/emerges in Large Language Models (LLMs). The idea is to use a model’s hidden representations for suitable stimuli, which are inputs related to concepts such as emotions. They calculate an ID score to approximately detect when this controllability emerges during training. Empirically, they analyse the emergence of controllability across several checkpoints of the entire pre-training phase of an open-source model to pinpoint the emergence of concepts such as emotions.
Unfortunately, the article suffers from technical inaccuracies that make it very difficult to trace what exactly the authors did in their experiments.



Section 3:

Firstly, there are no definitions for $h_+, h_-$, the normalized() function, $S_{test}, A_i$, and checkpoints. The Appendix furthermore lists $H^+$ and $H^-$, which were never defined. Could the authors include a clear (sub)section for definitions, perhaps at the beginning or within an expanded notation subsection, to define all key terms? Could they also ensure consistency between the main text and the appendix regarding notation?

In step 1 of the ID method, there are positive and negative "stimuli". While the authors list a template to create positive and negative stimuli in the Appendix, it is unclear what the "{positive concept scenario}" is. Could the authors provide specific examples of positive and negative scenarios used in their experiments? Additionally, could they explain the source and selection criteria for these scenarios?

In step 2, the $v \in \mathbb{R}^{1 \times 4096}$ is supposed to be the first principal axis, correct? If so, there is no dimensionality reduction. Furthermore, the sets $S_{pos}$ and $S_{neg}$ contain positive and negative "samples", respectively. Are "samples" the stimuli from before?

At the end of step 2, the authors claim that "For a layer $l$, this vector $v_l$ is linked to a specific concept". Why? Could the authors provide empirical evidence or theoretical justification to support this claim? 

In step 3, equation (3), there is a "[-1]" missing, correct?

In Step 4: Is there any theoretical justification for adding $v_l$ to a layer?



Section 4:

Firstly, what is CrystalChat? There is no reference or information. Furthermore, the authors extracted checkpoints (I guess from the pre-trained LLM360 Crystal model) and then "fine-tuned each checkpoint" - do you mean fine-tuning the model based on the weights for these checkpoints? Moreover, why fine-tune the model in the first place? It is also the first step in Figure 3, but–--from my understanding—the ID method's first step is collecting hidden states based on prompting models at different stages of the pre-training phase. Could the authors clarify this process and add a dedicated subsection within Section 4 that outlines the model architecture, training process, and rationale for fine-tuning? If the models are fine-tuned before the analysis, how can it be ensured that whatever is measured did emerge during pre-training (and not fine-tuning)?

Overall, there is no information about the experimental setup in this section. Since this paper focuses on one specific model, it would be nice to have some reference background so readers do not immediately need to consult the LLM360 paper. Furthermore, the authors use ChatGPT to evaluate the model output after intervening and list a template in the Appendix. However, it is very difficult to grasp what the included "{CrystalChat intervention results}" are. It would be helpful to include some concrete examples to showcase the contents of all the templates.

The entropy plots in Figure 5 show variations in the range 4.89 - 4.96 (or smaller). I am wondering whether these variations are significant in absolute terms. Could the authors provide some reference plots or baselines to illustrate why these variations are representative of the described behaviour?

Figure 6 shows notable differences for the checkpoints at 78.11% and sometimes at 62.81% and 93.41%. Why do these occur? What makes these checkpoints unique?

The datasets used for the Supervised Detection Task should be referenced in the text and briefly explained. Also, models are now fine-tuned with different learning rates - however, it is unclear what the other hyperparameters are (learning rate scheduler, number of epochs, batch size, etc.). In addition, how many values of runs with different seeds were averaged to gain these results?

To summarise, while the results seem interesting and the overall approach promising, the paper lacks crucial details regarding how the experiments were conducted, which prevents reproducibility. There is also no code available. However, I am open to improving my rating if the authors provide sufficient details and polish their article.




Minor: 

The papers for specific Algorithms like PCA and t-SNE should be referenced; similarly, the model version (for example, for ChatGPT) should be mentioned.

Grammar and spelling mistakes need to be corrected. For example:

Line 068/069: “on model’s output” -> “[based] on a model’s output”

Line 080/081: “(Crystal (Liu et al., 2023))” -> needs rephrasing

Line 095/096: “be summarized thus:” -> “be summarized as:”

Line 145/146: “assessing model’s” -> “assessing a model’s”

Line 146/147: “figure 3” -> “Figure 3”

Line 161: “Tan et al. (2024) uses” -> “Tan et al. (2024) use”

Line 199/200: “hidden state value” -> “hidden states”

Line 210/211 and 214/215: “Detecting” -> “Detection”

Line 224/225: “at the -1 token position” -> “at the final token[’s] position”

Line 268: “Representation Vector” -> “Representation Vectors”

Line 279/280: “ID scores Across” -> “ID Scores Across”

Line 317: “Figure 4 use” -> “Figure 4 uses”

Line 318/319: “show ID score” -> “show ID scores”

Line 766/767 and 770/771: “Give the statement” -> “Given the statement” (Is this just an error in the article, or is it also present in the code for the experiments?)
Yes
No ethics review needed.
See weaknesses
The idea to explore the emergence of linear steerability over pre-training is quite novel; past works only focus on the final configuration of the LM (but this is for good reason, see weaknesses). Experiments test a broad range of concepts.
This paper explores when controllability (via representation steering) emerges in language model pretraining. To do so, the authors devise the Intervention Detector (ID) method, which, given a set of layer representations, identifies when a certain concept becomes linearly encoded. They find that controllability, for several concepts, emerges suddenly during pre-training.
I have several concerns about the paper, summarized broadly as follows:

**Major weaknesses (impacted score)**
1. __Motivation unclear/unconvincing:__ it was unclear why when controllability emerges should matter. In particular, 
    - While the authors state that past work focuses on steerability of already-trained language models, this is the actual use case of LM steering. In contrast, the authors would need to make a strong case for investigating the emergence of controllability-- to me, it is unclear why one cares about representation steering on a not-fully-trained model, as the model itself would never be deployed. I would suggest to clarify the context in which emergence of controllability is useful.
    - Controllability in and of itself doesn't mean much: what does it mean for concept representations to emerge "early in pretraining" when the LM presumably has not converged to a good distribution of language? 

2. __Imprecise definitions:__ the paper is about ``controllability" at large, yet only addresses a quite narrow domain within _linear_ control, adding an unscaled input to the representation. Controllability is a broad term that spans also nonlinear control-- I would change the emphasis to just additive representation steering. 

3. __Methodological issues__: 
    - Even though the authors state it is out of scope, it is important to test on another LM family for generalizability.
    - Did you try different scaling factors? In the literature, the scaling strength is crucial for performance [1-3]. If different scaling factors were not tried, could the authors please provide a justification?
    - Experiments are ideally run with several random seeds. For instance, it is hard to gauge the true effect for ARC Challenge and ARC Easy without error bars-- for OBQA, the gap is also quite small. Could the authors please provide error bars with the plots?

**Minor weaknesses (didn't impact score)**
1. Missing citations: https://arxiv.org/abs/2310.04444v3 https://arxiv.org/abs/2405.15454
2. l205: summarize the RepE method (e.g., are you referring to RepE linear combination, piece-wise operation, or projection?)
3. l210: the algorithm used for the Unsupervised task is unclear. What does it mean to obtain hidden values? If you mean the hidden states of the LM, which state are we using (e.g., last token)?

[1] Turner et al. 2023 https://arxiv.org/abs/2308.10248

[2] Li et al. 2023 https://arxiv.org/abs/2306.03341

Missing references:

[3] Cheng et al. 2024 https://arxiv.org/abs/2405.15454

[4] Soatto et al. 2023 https://arxiv.org/abs/2305.18449

[5] Bhargava et al. 2024 https://arxiv.org/abs/2310.04444v3
Yes
No ethics review needed.
1. I think there are some operations that you need to explain why:
- In Hidden States Collection, why do you collect the hidden states at the -1 token position?
- In Dimensionality Reduction, why do you compute the difference of hidden activations and compute PCA? How is this related to the latent concept?
- Where is the definition of $A_i$ in Analyzing ID scores Across Layers?

2. Have you considered other methods of selecting stimuli? For example, based on perplexity. 

3. You just randomly selected 256 stimuli, which is heavily influenced by the random seed, so I think you need to run different tests to see how different seeds affect the random selection.
1. The idea of investigating the concept of controllability within pre-trained language models is novel and interesting.
2. The paper has been experimented with fairly extensively, and the results show good potential.
The paper explores how controllability emerges during the pretraining process of language models. The authors propose a novel framework, named Intervention Detector (ID), which applies dimensionality reduction techniques (like PCA) to the hidden states of models under different stimuli, allowing for the extraction of concept-specific representations. This method allows for targeted interventions, helping models respond more predictably to specific prompts.
The paper finds that controllability in language models is an emergent property that develops as pre-training progresses.
The paper lacks clarity and most of the practical details and its design principles remain vague or unresolved. Please see the Questions.
discovering and analysing the reason that model's controllability towards different concept could emergence at different stage of training, and we use a method to detect this property
@inproceedings{
anonymous2024how,
title={How does controllability emerge in language models during pretraining?},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=egHptuv7hx},
note={under review}
}
Language models can be controlled by adjusting their internal representations, which alters the degree to which concepts such as emotional tone, style, truthfulness, and safety are expressed in their generative outputs. This paper demonstrates that controllability emerges abruptly during pre-training, and furthermore, even closely-related concepts (e.g. anger and sadness) can emerge at different
stages of pre-training. To understand how controllability of internal representations changes during pre-training, we introduce the “Intervention Detector” (ID), which applies dimensionality reduction to hidden states under different stimuli, and outputs concept representations that can be applied to control language models. Using these concept representations, we then compute an extraction score (ID score) that shows how well the extracted representations align with the model’s hidden states. This ID score can be used to approximately predict the time of emergence of controllability for different concepts, and the degree to which each concept is controllable. By analyzing ID scores across a longitudinal series of models taken at different stages of pre-training, we demonstrate that, as pre-training progresses, concepts become increasingly easier to extract via dimensionality reduction methods, which correlates with the emergence of controllability. For instance, in the CrystalCoder model, the controllability of the concept “anger” emerges at 68% of pre-training, whereas the controllability of the concept “sadness” emerges at 93% of the pre-training process. We use heatmap visualizations and other metrics (eg., entropy, cosine similarity, tSNE) to study these differences, and validate the reliability and generalizability of ID scores through model interventions using the extracted concept representations.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
controllability ability emergence pre-training models dimentionality reduction representations
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/542ff133b03fa9ae7a88e45572b7a805e3797381.pdf
interpretability and explainable AI
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
How does controllability emerge in language models during pretraining?
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission