True Counterfactual Generation from Language Models
Causality;language models;counterfactuals
Thanks for the in-depth review. We wanted to answer the three questions before we respond with more because we think we can address all your concerns in detail, but we want to be sure we understood it all. 

1. Our method is invariant to the exact method used to sample from the softmax in a specific sense. Non-identifiability when sampling from Categorical only arises from permutation non-invariance. See, for example, Oberst and Sontag (2019) Secton 3.1 [https://proceedings.mlr.press/v97/oberst19a/oberst19a.pdf].  We can give formal proof of this, which appears to be missing from the above reference. Thus, we chose the Gumbel-max trick for convenience as it is a simple, permutation-invariant sampling scheme. Would our spelling out this equivalence assuage some of your concerns? The reason we rule out permutation-dependent schemes, e.g., what we would call roulette sampling, is that the permutation is arbitrary. 

To be explicit, the result we will add, which we think will dramatically improve the paper is this. If one wishes to assume the sampling process from the Categorical is permutation-invariant, i.e., the order we assign elements of the categorical an integer does not matter, then the model is identifiable.  

2. This is a choice to reflect standard LM notation. We can change it it was confusing.

3. All of our experiments use hindsight sampling. Roughly speaking, the experimental design works like this:
a) Input a string
b) Apply an intervention on the network
c) Sample a counterfactual string using Algorithm 1.

So, our method is the first that can show how individual strings change under interventions, e.g., MEMIT. 

We specifically appreciated many of your comments about GSEMs. One quick note -- our GSEM is acyclic, so they have one solution. (An unrolled neural network-based language model is acyclic.) We only apply them because we have an infinite number of variables and SEMs do not apply. Apologies that we did not make this point clear. Also, we are not tied to GSEMs, as the reviewer infers, would it help the presentation to switch to causal dynamic Bayesian networks? We view our contribution as the first algorithm to give *string*-level counterfactuals under the model. We can elaborate more on why we call these "true" in the title.
quick initial comment
Yes
No ethics review needed.
See weakness.
The paper introduces a novel framework that reframes language models through the lens of causal inference, which offers a new perspective for analyzing the causal mechanisms within language models. Building on this new framework, the paper also proposes a novel approach to generate counterfactual strings for observed strings. Interestingly, the proposed method provides a systematic means to evaluate existing intervention techniques, revealing previously unrecognized side effects.
The paper first reframes the decoder-only language models as Generalized Structural Equation Models (GSEMs) with the Gumbel-max trick. Building on this framework, the authors propose an algorithm leveraging hindsight Gumbel sampling to infer and fix latent exogenous variables, enabling the generation of counterfactual versions of observed strings. Experimental results demonstrate that this method produces meaningful counterfactual pairs and further uncovers significant limitations of existing intervention techniques.
1. The presentation of the paper could benefit from clearer emphasis on its primary contribution: generating counterfactual pairs. It would be helpful if the authors clarified early on that the intervention models employed are drawn from existing methods. Before reaching Section 4, there may be some ambiguity regarding how to obtain the counterfactual encoder $\tilde{h}$, which seems the hardest part for counterfactual generations.
2. The way to find $U_t$ should be refined: 
     - Given the counterfactual string sampling method in Corollary 3.1, infinitely many vectors $U_t$ could theoretically be sampled, potentially resulting in different counterfactual generations. The paper currently presents only one counterfactual string per original string, and it would be beneficial to provide additional evidence demonstrating consistency across different samples of $U_t$.
     - While the Hindsight Gumbel Sampling can generate $U_t$ to ensure the next token is $\bar{w_t}$, there is limited theoretical or empirical discussion supporting this choice. Specifically, since $U_t(\bar{w_t})$ satisfies $\pi(\bar{w_t})+U_t(\bar{w_t})=\max_{\bar{w}\in\bar{\Sigma}}\pi(\bar{w})+U_t(\bar{w})$, the marginal distribution of $U_t(\bar{w_t})$ shouldn't be Gumbel(0,1) as mentioned in Proposition 3.1.
3. Additional experiments would enhance the persuasiveness of the results. For instance, demonstrating counterfactual generations across multiple Hindsight Gumbel Sampling trials would reinforce the approach’s robustness. Further, examining counterfactuals of counterfactual strings and comparing them to the original strings could offer valuable insights into the reliability and utility of the proposed method.
Yes
No ethics review needed.
1.  A fundamental question: why use Gumbel noise? I understand that the Gumbel max trick is one way of sampling from a Boltzman distribution, but there are other (arguably simpler) ways as well.  I vaguely see that Corollary 3.1 might be useful, but are there analogues for tother causal models? Would other approaches to modeling the noise yield similar counterfactual distributions? In brief: what is special about Gumbel noise, and why does it feature so prominently in this paper?

2.  A relatively unimportant question about notation: Given that  E and h only ever appear together (in the context "E h(w_{<t})" ), why bother separating them? Why not just define a new function of w_{<t} that is equal to this product?

3. What is the relationship between the experimental evaluations (MEMIT, Inference-time Intervention, Instruction Tuning) and the Gumbel causal model?  It looked as though Corollary 3.1 was going to be the basis for a new method, but instead it seems the evaluation just looks at several methods already present in the literature.  Do those have Gumbel noise?  Does Algorithm 1 make any appearance in the evaluation?  It appears not to.  Am I missing something?
The paper is beautiful, and very well-written.  The layout and math is clean, and the English is sharp.  The concepts are interesting, and the problem of generating counter-factual is well-motivated.  Many (but not all) unnecessary details are swept away.  It appears that the authors implemented their algorithms, and attempted several different evaluations with real-world data and models used in practice.   I think the presentation is so good as to be deceptive.  The vision is grand and compelling.
The paper proposes modeling a language model as a causal model with infinitely many variables (they use a GSEM, although I am skeptical that this choice is appropriate) with Gumbel noise.  The idea is to improve counter-factual generation: fixing a generated sentence, find the sentence that would have generated with these noise settings, if in fact the model had been different.  The authors review the Gumbel max trick for parametrizing a sample from a softmax, argue that any LM can be captured by a GSEM that uses this trick, and then provide an algorithm for counter-factual sampling (Alg 1) based on these assumptions.  The paper then evaluates prior methods for model-editing, arguing that they are insufficient.
Once we adequately pay our respects to the presentation and get to the substance of the paper, things start to fall apart quickly, on many fronts.  At a high level, the story is strong, but the math and evaluation simply does not back it up.  Below the surface, the conceptual, theoretical and empirical aspects of this work are all severely lacking. 

**Concepts.** 
- At a high level, I feel the authors do not really "get" what GSEMs are and how they fit into the causal literature.  The fact that they allow infinitely many variables is actually not relevant.  The authors are also not using the definition of a GSEM properly, and completely sweep under the rug the possibility of a GSEM having multiple solutions.  They do not discuss the constraints that are part of a definition of a GSEM (even in the appendix), that an intervention must have the effect of fixing a variable in a certain way.  In short: there are easier and more appropriate ways of using tools from causality here.  (Causal) dynamic Bayesian Networks are one.  To me, it seems that the authors were overly fixed on the "infinitely many variables" aspect of a GSEM, and then copied the the definition of Peters and Halpern (2022) without reading the rest of the paper.   

- The core contribution seems to be the understanding of the LLM noise variables through the Gumbel model, culminating in Corrolary 3.1 and Algorithm 1.  Yet the authors seem to miss the fact that the Gumbel distribution is just one of many possible causal models that one could use to get the LM's distribution. They do not acknowledge this, or consider how counter-facutals would work in other contexts.  They seem to be under the impression that their Gumbel model is the "right way to model this", as evidenced by the title ("True Counterfactual Generation in Language Models".)   I see no reason to believe that this is the case, and feel there is some slight-of-hand going on here.

- Finally, the evaluation does not even involve their own method! The conclusion states that these methods
   *"underline the need for more refined methods that can achieve targeted modifications with minimal collateral changes to the
model’s outputs."*
  But they do not evaluate their own solution to this problem! This seems hugely problematic. 
 


**Theoretical Results.** Proposition 2.1 is obvious, easy to prove, and a very weak result (as I describe below).  Proposition 3.1 is not precisely stated, and it is not proved in the appendix; instead, the authors reference another work that does prove this.  I believe it is important to either adapt the result carefully to your own modified statement of the result, or to use essentially the same wording and credit the original authors.  Corollary 3.1 has some intuitive pull, but no further properties of the distribution are investigated.  There are interesting questions here.  Is the Gumbel model the unique one with some property?  How do these counterfactuals differ from those in other SCMs? But those questions are not answered here.

In general, the wording around the probabilistic modeling does not inspire confidence.  The paper is missing lots of independence assumptions, uses undefined terms and symbols, and directs focus in unnecessary places.  I am particularly unhappy about the statement of Prop 3.1 and the false statements that a causal model gives rise to a unique distribution.  (See my comments below for more detail and further examples.) 

**Empirical Evaluation.**  The weakness of this section is unavoidable given that the authors did not evaluate their own method. There are some interesting ideas here, and the delve into the weaknesses of other methods is certainly valuable material.  But it is out of place, and being dramatically over-sold.  I also have some thoughts about why the experiments could be done better, but none of them really matter without seeing an evaluation of the proposed artifact. 

--------

Detailed Line-by-Line Comments and Concerns:

- 115-116**: Given the definition above, it is not true that a probability over \U induces a probability over \U \cup \V.  In general, there may be many such distributions or none at all.  For this to be true, you need to significantly restrict (not generalize) your class of PSEMs; it seems acyclic ones will do.  But later on, you generalize to SEMs with infinitely many variables, where again this is not true.  
(Finally, the notion $\mathbb P^{\mathcal S}$ is problematic; $\mathcal S$ is just the signature, and this distribution (if one exists) depends on not only \S, but also on \F, and the choice of fixed point.

- 131-133: Definition 2.2 is too vague to be numbered and italicized like this.  First, as mentioned above, the intervention may lead to multiple distributions. Second, the posterior distribution must be calculated with respect to the original (choice of) extended distribution (\P^{\S} in your notation, which I don't like). Finally, over what variables is this interventional distribution? Is it \U \cup \V?  Finally, some justification is required for this choice. Note that, the counterfactual intervention without intervening at all (i.e., taking I to be empty) need not be v.  Is this acceptable? If so, why? 

- 173: I recommend using a symbol other than \pi (perhaps \ell), for the logits, because \pi has a probabilistic connotation.  In other description of the Gumbel-max trick, \pi is defined as the exponential of this quantity. Also, it would be easier to parse if \pi were bound before the equation.

- 175: The M outcomes must be independent. 

- 184: This is the second "Let \Sigma be". It would be better to say "Recall that \Sigma is", because \Sigma and other symbols were defined above.  If one skips here and starts reading, they will not get the cue that these other symbols were defined earlier. 

- 187: Missing independence assumption (I can only assume), as this is required for everything to be defined. Specifically, { U_t(w) : w \in \bar\Sigma } must be mutually independent.

 - 189: Text inaccurately suggests that the language encoder is just h, but it is also E.  Given that 

 - 191: The quantity {\bf b} comes out of the blue.  Based on context I assume it is some kind of bias, but it does not fit into the math at all. Note that it plays the same mathematical role as U_t, except it is undefined. Also, assuming that it is a parameter like E, it is mathematically unnecessary for the usual reasons.  Equation (1b) does not have a bias, and I think this one should not have a bias either.

 - 200: Proposition 2.1 is extremely weak.  GSEMs are very expressive, and it is obvious that this autoregressive process can be captured by a GSEM (and it can also be captured by other generalizations of causal models that are not nearly as expressive). Moreover, the statement of the theorem does not even use the primary data of the GSEM: it's response to interventions.  In fact, the proposition can be "significantly strengthened" to an equally vacuous statement, by replacing the first line with "let p be an arbitrary probability distribution over \Sigma^*."


 - 223: What is "\Pi"? It has not been defined. 
 - 228: caption (b) I think it should be "a LM" not "an LM"

 - 240: If I understood correctly, you mean not that it can be sampled this way, but rather that \tilde W equals this quantity *as a function of the previous noise U*.  The wording needs to be tightened up a bit here.

 - **275: The term "truncated probability distribution" is not a formal concept in probability. You're really referring just to given conditional distribution; there's no reason to bring "truncation" into it, and the distribution is not a special kind of object; it's a probability distribution like any other.  Also, the given reference (footnote 8) is a blog post, but the concept is very old and has a name: rejection sampling. 

 - 293: This is not an algorithm, so much as a snippet of python code. 
 - 357. Why are you using just the lengths of prefixes for the evaluation? Why not compare, for example, edit distance, word movers' distance?
Yes
None
No ethics review needed.
- The formulation assumes the gumbel noise to be independent at each timestep. Will the noise not have any dependence or is this dependence captured appropriately by the Endogenous variables?
- Any limitations with using a Gumbel distribution to model sampling from softmax?
The paper makes key contributions that help advance the field of causal interpretability in language models:

$\textbf{Novel Framework:}$
- Reformulates autoregressive LMs as Generalized Structural Equation Models (GSEMs)
- Decomposes language generation into Deterministic computation (logits from model) and Stochastic elements (sampling noise as exogenous variables)
- Leverages Gumbel-max trick to establish equivalence with softmax sampling

$\textbf{Theoretical Foundations:}$
- Proposition 2.1: Proves equivalence between LMs and GSEMs through rigorous mathematical formulation
Shows $P(W = w_1...w_T) = P_E(W_1 = w_1, ..., W_T = w_T, W_{T+1} = EOS, ...)$

- Proposition 3.1: Derives hindsight Gumbel sampling for noise inference
Presents clear algorithm for counterfactual generation while preserving sampling noise

$\textbf{Integration with Causal Framework:}$

Enables true counterfactual reasoning through
- Noise inference from observed sequences
- Noise preservation during counterfactual generation allowing for "retrospection"

Maintains compatibility with existing and future intervention techniques through GSEM formulation

$\textbf{Empirical Validation:}$

Architectures used:
GPT2-XL, LLaMA3-8b

Interventions:
MEMIT (knowledge editing), Linear steering, Instruction tuning

$\textbf{Impact and Implications:}$

- Provides a foundation for analyzing intervention effects through counterfactual reasoning
- Offers to control generation stability
- Reveals unintended side effects in current intervention methods

The paper's primary strength lies in providing a theoretically sound solution to a fundamental challenge in causal interpretability: isolating intervention effects from inherent sampling randomness in language models.
This paper addresses a fundamental challenge in analyzing autoregressive language models with causal interventions: generating true counterfactual sequences that isolate the effects of model modifications from inherent sampling randomness. The authors present a novel framework that reformulates LMs, within Pearl's causal hierarchy, by decomposing them as Generalized Structural Equation Models (GSEMs). This reformulation enables a more rigorous counterfactual analysis by explicitly modeling and controlling the stochastic elements of language generation, while also allowing for the use of an expansive set of Causal Interventions.

The paper's primary theoretical contribution is the decomposition of LMs into deterministic and stochastic components through the Gumbel-max trick. Here, the argmax with Gumbel noise is chosen due to its equivalence to sampling from a softmax distribution. By identifying sampling noise as exogenous variables within the GSEM framework, the authors develop a principled method for inferring and preserving these noise values during counterfactual generation. This separation enables the analysis of intervention effects while controlling for the underlying randomness that is inherent in LM sampling.

Key Technical Contributions:

1. Establishing the equivalence between LMs and GSEMs, allowing for the analysis of Causal interventions using "retrospection" (Pearl's 3rd level of counterfactual reasoning)
2. Development of a hindsight Gumbel sampling algorithm that enables inference of latent noise variables from observed sequences
3. Generation of counterfactual strings that maintain sampling consistency across model interventions, allowing for pairs of sentences to be sampled from the joint distribution of the original and counterfactual strings.

The authors validate their framework through comprehensive experiments on LMs (GPT2-XL and LLaMA3-8b), utilizing and examining various intervention techniques such as knowledge editing (MEMIT), linear steering, and instruction tuning. Their analysis reveals that even minimal interventions can produce unexpected semantic changes in model behavior, highlighting the importance of controlled counterfactual analysis in understanding model interventions.

The work helps advance the field of Causal reasoning by allowing true counterfactual reasoning in LMs, providing tools to better understand and control the effects of model modifications.
$\textbf{Empirical Validation of Causal Framework:}$
While Proposition 2.1 provides a theoretical foundation for the LM-GSEM equivalence, several key empirical validations are missing even with section B in the appendix:

$\textbf{Noise Completeness}:$
The paper proves that $W_t = \text{argmax} {w\in\Sigma}(Eh(w_{<t}) + b)_w + U_t(w)$ captures sampling behavior, but doesn't empirically validate this captures all stochastic elements. 

Given that: $P(W = w_1...w_T) = P_E(W_1 = w_1, ..., W_T = w_T, W_{T+1} = EOS, ...)$

I would want to see further evidence that:

- Compounding effects at longer sequences are purely from intervention
- Quantification of any residual unaccounted noise

$\textbf{Counterfactual Quality}:$
Missing controlled experiments demonstrating the difference between the generation with and without preservation. The separation of Noise allows for a more stabilized sequence generation, and I would like to see if the reduced divergence can be quantified.

$\textbf{Secondary Concerns:}$
- Prefix length ($l_{\text{prefix}}$) is overly sensitive to early divergences
- E5 similarity ($\text{sim}_{\text{E5}}(x,y)$) may miss subtle linguistic changes

$\textbf{Ethical Considerations}:$
- No discussion of potential misuse in generating misleading content
Yes
No ethics review needed.
- Can the proposed method work with beam search decoding, which is commonly used in LLMs for generating sentences?
- How does the temperature parameter in the Gumbel-max trick affect the quality of the generated counterfactual sentences?
- The paper is well-motivated and clearly written. The authors provide a good introduction to the problem of true counterfactual generation and the overlooked challenges in the existing methods. The theoretical analysis is sound and well-presented. 
- The addressed problem is important and timely, as the true counterfactual generation is a crucial task for understanding the behavior of LLMs and the potential biases in their outputs. The proposed method is a step forward in this direction.
- The experimental results are insightful and the findings are interesting and can be useful for future research on true counterfactual generation.
The current paper stuides the true counterfactual generation of large language models: how a given sentence would have appeared had it been generated by the model following a specific intervention. The true counterfactual generation is an important task for understanding the behavior of large language models, and it is also a challenging task for LLMs, as it requires the counterfactual sentences to be generated in a way that is consistent with the original sentence, which is hard to achieve for LLMs due to their statelness and non-deterministic nature. 
In particular, this paper focus on the interventions of representation surgery, a technique that modifies the internal representations of an LLM to change its behavior. The authors argue that the existing methods for true counterfactual generation are not strictly conforming to the definition by Pearl, and propose to use generalized structural-equations models (GSEMs) to model the counterfactual generation process: first, the LLM is framed as a GSEM where the uncertainty in the generated sentences is modeled with latent independent Gumbel distributions; then, given a sentence and an intervention, the counterfactual sentence is generated from the LLM after intervention using Gumbel-max trick where the Gumbel noise is sampled from the latent Gumbel distributions inferred from the original sentence. The authors theoretically prove that the generated counterfactual sentences comply with the definition of true counterfactuals by Pearl.
In the experiments, the proposed method is used to analyze two aspects of the example interventions (MEMIT, inference-time intervention and Instruction Tuning): 1) the side effects of the interventions, which are the changes in the generated sentences that are unrelated to the intervention; 2) the effectiveness of the interventions, which are the changes in the generated sentences that are related to the intervention. The results show several interesting findings, e.g., the side effects of the interventions are inevitable with MEMIT being the most effective in reducing the side effects.
- There is no detailed discussion in the main body on the limitations of the existing methods for true counterfactual generation. It would be helpful to have a more detailed comparison with the existing methods and a discussion on the limitations of the proposed method.
- The experiments are conducted soly with the proposed method without comparison with the existing methods. It would make the readers hard to evaluate the effectiveness of the proposed method in practice.
We propose a framework to generate true counterfactuals from language models by reformulating them as Generalized Structural Causal Models using the Gumbel-max trick.
@inproceedings{
anonymous2024true,
title={True Counterfactual Generation from Language Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=TUC0ZT2zIQ},
note={under review}
}
Understanding and manipulating the causal mechanisms in language models is essential for controlling their behavior. Previous work has primarily relied on techniques such as representation surgery---e.g., model ablations or manipulation of linear subspaces tied to specific concepts---to \emph{intervene} on these models. To understand the impact of interventions precisely, it is useful to examine \emph{counterfactual} strings---e.g., how a given sentence would have appeared had it been generated by the model following a specific intervention. We highlight that counterfactual reasoning is conceptually distinct from interventions, as articulated in Pearl's causal hierarchy. Based on this observation, we propose a framework for generating true string counterfactuals by reformulating language models as Generalized Structural-equations Models (GSEMs) using the Gumbel-max trick. This allows us to model the joint distribution over original strings and their counterfactuals resulting from the same instantiation of the sampling noise. We develop an algorithm based on hindsight Gumbel sampling that allows us to infer the latent noise variables and generate counterfactuals of observed strings. Experiments demonstrate that our approach produces meaningful counterfactuals while at the same time showing that commonly used intervention techniques have considerable undesired side effects.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Causality language models counterfactuals
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/340210ee9fd17ad5673cd0222c5dbc47cb8d51d8.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/d5f4691c178f6a5386c29aba92a18f71b4749d0c.zip
True Counterfactual Generation from Language Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission