Markovian Transformers for Informative Language Modeling
Chain of Thought Reasoning;Reinforcement Learning;Scalable Oversight;Language Modeling;Proximal Policy Optimization
Yes
No ethics review needed.
- Is there a way to combine the interpretability with actual human perception? Though informativeness here can be used to improve model quality, it is also very helpful from human level. This is probably mentioned in F. But it seems to me F is more about how to encode human interpretability in training.
- In F it is mentioned that "optimal CoT would be a compression of the question, which can potentially be difficult for humans ". Is this observed in your experiments?
- Were there more ablation study or comparison conducted?
- The Markovian framework blocks the model from attending back to the original question and force it to use the CoT context for generation. This provides a new view and framework for analyzing CoT effects.
- The reinforcement learning-based approach demonstrates improved performance on tasks requiring multiple steps, 
- The CoT steps generated from this method seem to be more interpretable, from two dimensions: 1) pertubation of the CoT could lead to more model errors 2) the reasoning can be carried over to another model.
This paper proposes a framework where the reasoning steps are used as fixed-size states, which limits the model’s context (text bottleneck), and force the model to use the reasoning steps as input. This method design is inspired by the fact that past CoT literature find that the final answer might not be sensitive to the CoT trace. In the experiment, the authors show that the model trained with this method is indeed more fragile against CoT pertubations.
-  While the approach improves the model’s reliance on CoT, it’s uncertain if this CoT is genuinely interpretable by humans. The transferrability between Mistral and Llama should only serve as an indirect proof.
-  It is actually fine to focus on QA task, but probably we'd like to see how this can generalize to more domains other than arithemtic. Would this paradigm also work for other reasoning task as well?
- There seem to be no baselines and ablation designed, so it is a bit hard to position the effectiveness of the method against other methods.
- The fragility analysis is insightful but lacks depth. A more detailed investigation into which types of perturbations impact CoT reliability most could provide valuable insights.
-  Writing-wise, the paper writing is clean, but probably some adjustment of the sections flow and emphasis would be nice.
   -  For instance, while the method section is quite detailed, it’s presented before establishing the limitations of existing CoT techniques clearly, which makes it harder to understand the innovation.
   - There are few tables but quite a few definitinos and equations. I'd suggest consider streamline the method descriptions and move some of them to the appendix, while adding more discussion and insights in the main body.
Yes
No ethics review needed.
Have you tried other open-source models like llama? Not use CoT of Mistral in it, but use your method to finetune Llama.
The technical ideas, including the proposed metric and RL methods, are new, well-motivated, and technically reasonable.  

The experiments in random addition and GSM8K are positive.
This paper proposes a metric to measure the informativeness of CoT tokens, and then uses RL to train the model to generate highly informative CoT tokens, in order to improve the correctness of the final answer. Experiments in random addition problems and GSM8K math problems demonstrate the effectiveness of the proposed metric and RL methods. The paper also shows that more informative tokens will bring gains in interpretability.
The experiments are limited to a synthetic math problem setting and GSM8K, and the only trained model is Mistral 7B. 

The presentation needs a better organization. E.g., some major results are placed in the appendices, but training details are in the main paper.
Yes
No ethics review needed.
* Is the space of states task-conditional? This isn’t apparent based on the formulation in Section 3.1 (and is unclear by the wording in line 162). If not, then it would seem that the set of relevant “CoT states” would be very sparse relative to the complete space. 
* As posed in the weaknesses section, does this method extend to other reasoning tasks (e.g. in natural language or code) whose structure is less “linear”?
* The setup of this causally-guided model is pretty novel, and the finding that this improves performance by optimizing on an “information” metric is an impactful finding. 
* The selection of the RL training technique (PPO) is supported by highlighting the limitations of other considered methods (expert iteration and policy gradient).
* The work is written in a way that was simple to follow, which I appreciated.
* The gains on GSM8K (24.64% --> 35.71%) are meaningful (although, a bit tucked away in the paper's text).
This work introduces and explains the construction of a Markovian Language Model to study causality in chain-of-thought reasoning. With a limited state (the previous state and its observation) on which to condition, the model is trained (fine-tuned) to maximize an informativeness objective through PPO, and is empirically shown to improve performance on mathematics tasks such as GSM8K and toy addition problems.
* I understand the general intuition surrounding the design of the informativeness function, but it would be good to add some discussion on why the expected reward over the trajectory actually constitutes / addresses “informativeness” under your construction. 
* While math tasks have a more well-defined structure (the order in which their steps may be pursued), this is less clear for other tasks without such a clear structure in natural language, for instance. It would be good to examine this approach on at least one such task to further support the method’s general efficacy.
* Despite the intuition-based process of selection for the RL training strategy, there are recent works that advocate in favor of expert iteration and REINFORCE / vanilla policy gradient for LLM reasoning and RLHF [1, 2]. To this effect, including such approaches for comparison in the results section (or in the appendix) would strengthen the defense of the PPO method chosen. It would be helpful to include some evidence supporting the limitations posed.
* While I appreciate documenting the design choices in Section 4.3, some justification behind them would be beneficial, either through ablations (it’s fine for these to be in the appendix) or relevant references. 

1. Teaching Large Language Models to Reason with Reinforcement Learning. Havrilla et al. 2024 
2. Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs. Ahmadian et al. 2024
Yes
No ethics review needed.
Please use different line styles in your figures so colorblind people can make sense of them. Otherwise Figure 2 and 3 are really hard to parse!
* The paper addresses an important limitation in Chain-of-Thought reasoning, which is of relevance to the broader ICLR community.
* The core idea is intuitive and simple.
* The paper is well written.
* The results on the simple arithmetic task and the math task are promising.
* The claims that the proposed method improves the generated CoTs in terms of interpretability and informativeness are well supported.
The paper addresses an issue of Chain-of-Thought (CoT) reasoning in LLMs where the LM's final answer does not always depend on the CoT. The paper's idea is to enforce informativeness by conditioning the answer model on the generated CoT only without other context. To this end, the paper formally defines Markovian Language Models and (informative) update functions, from which the policy gradient procedure is derived. Applying the framework to the specific use case of CoT reasoning, the paper experiments with several RL techniques such as expert iteration, policy gradient, and PPO. The model is applied to a simple arithmetic task of adding 15 numbers as well as GSM8K, and shows that the model a) improves performance on the task b) is sensitive to perturbation in the CoT reasoning and c) produces CoTs that are sensible to a different language model such that its performance on the task is improved.
The method is evaluated only on few tasks and models, limiting how sure we can be that this is a useful method. Especially an application to language modeling would be very insightful and potentially extremely impactfull. However, while more is always better when it comes to experimental results, I think that this initial set of experiments support the ideas presented well and should suffice for publication.
We introduce "Markovian" RL-training that ensures that LM Chain-of-Thought is causally load-bearing, while improving performance on many-step arithmetic and GSM8K over baselines, thereby advancing more transparent and interpretable AI reasoning.
@inproceedings{
anonymous2024markovian,
title={Markovian Transformers for Informative Language Modeling},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=s5N7p5UjgR},
note={under review}
}
Chain-of-Thought (CoT) reasoning holds great promise for explaining the outputs of language models, but recent studies have highlighted significant challenges in its practical application for interpretability. We propose to address this issue via two key components: a technique to factor next-token prediction through intermediate CoT text, ensuring the CoT is causally load-bearing, and a reinforcement learning approach to train CoT to predict future tokens independently of other context. This results in "Markovian" language models, where CoT serves as a fixed-size state for future token prediction. Our approach optimizes for "informativeness" – the improvement in next-token predictions using a trained CoT compared to a baseline. We demonstrate our method's effectiveness using Proximal Policy Optimization (PPO) on arithmetic problems and achieve an 11% performance boost on the GSM8K benchmark using Mistral 7B Inst V2. The increased sensitivity of model performance to CoT perturbations provides strong evidence of CoT reliance. This work advances the development of more transparent and interpretable language models, potentially enabling their extension to arbitrarily long contexts and enhancing AI reasoning capabilities across various domains.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Chain of Thought Reasoning Reinforcement Learning Scalable Oversight Language Modeling Proximal Policy Optimization
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/a79caf50e0c520c209bb406385368c4127f93770.pdf
reinforcement learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/e73ba559b132b247843e7c743557c4da6fa2ca64.zip
Markovian Transformers for Informative Language Modeling
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission