Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solver
LLM;Reasoning
Yes
No ethics review needed.
Answering the questions 1, 2 above would help.
The experimental results point to much improved performance of the model under reasoning.
The paper puts forward an approach to improve reasoning capabilities of LLMs. The approach is based on MCTS with a discriminator model selecting the most promising answers. Experiments appear to demonstrate superior reasoning capabilities of the approach compared to fine-tuning and otherapproaches.
This is not my area, hence low confidence, however as someone outside the LLM area I struggled to see the arguments put forward in the paper. I recommend the author to put more emphasis into making the key parts of the technical material more accessible by providing more explanations. Here are the key problems I had.

1 I did not understand how the MCTS approach works here both in terms of generation of the next steps. What simulation is performed, how this is tuned and where the MCTS itself results from (is it the model itself). Equally I found no explanation as to why this would/should lead to increased performance in the step generations.

2 I did not understand why a discriminator would have better performance than the original model and why it is the case that these capabilities cannot or should not be employed directly by the reasoning model. 

I think that if I had more explanations on the points above I would have been to understand the technical contribution more.
Yes
NA
No ethics review needed.
Reasoning capabilities are often studied in terms of generalizability How would you study generalization capabilities of your method? Could you transfer between tasks/benachmarks?
- The paper is relatively well written making it easy to follow..
-  the methodology is clearly explained and differences as well as similarities to competing methods are made explicit.
- In the experimental evaluation it is clearly shown that the proposed method out-competes state-of-the-art approaches. This is substantiated by a couple of informative ablation studies.
The paper describes a method to improve "reasoning capabilities" of small language models. This is done by generating trees of prompt using MCTS and by using a second small language model that "critiques" MCTS rollouts. After introducing the method the paper than show in its experimental section that on a suite of benchmarks the new method, dubbed rStar, outperforms existing methods.
The paper claims to improve reasoning capabilities of small language models. However, by training/augmenting a SML with MCTS on a specific dataset we now end up with a model that is informed by the statistics of the dataset. In the end there is no reasoning happening but the proposed method allows the SML to better exploit statistical patterns in the data-query pairs that it was trained on.

In the end the proposed methods does not allow SMLs to reason but is a method to perform prompt-engineering in an automated fashion using the statistics of the benchmark in question.


One of the problems I have is the anthropomorphism present in the paper. Specifically, with using "rich set of human like reasoning actions" already in the abstract and continuing it throughout the paper.


While I do not see how the paper enables SMLs to reason better, I can see that the introduces techniques have clear experimental advantages over competing methods. This leads me to tend towards arguing towards accepting the paper ever so slightly. .
Yes
No ethics review needed.
1) The authors introduce a set of five human-like reasoning actions as the action types for MCTS. This design requires manual selection and experimental validation, which may not be optimal. Could the authors provide any experiments or analysis comparing their manually selected actions to other potential action sets? Did the authors consider automating the action type design process?


2) A very interesting aspect of OpenAI's o1 model is its self-reflective behavior. Did the authors consider integrating self-reflection as an action type in their framework? What potential benefits or challenges do the authors expect with such an addition?


3) The authors mention that temporal order constraints exist for different action types. Could the authors provide pseudocode or a more detailed explanation of how the temporal order constraints are implemented in their MCTS algorithm?


4) In Section A.3, the authors mention the high token cost associated with this method. For instance, the average number of generated tokens per question on GSM8k is 367.1k, which could limit the method's practical applicability. Have the authors considered optimization strategies to address this issue? While distributed inference can reduce processing time, it does not reduce the overall computational cost.


5) The authors emphasize that this method is designed for SLMs. Have the authors conducted experiments or analysis comparing rStar's performance on SLMs versus on LLMs? What’s the expectation of the method's effectiveness to change with model size?


Reference:

[1] Huang, Jie, et al. "Large language models cannot self-correct reasoning yet." arXiv preprint arXiv:2310.01798 (2023).

[2] Chen, Ziru, et al. "When is tree search useful for llm planning? it depends on the discriminator." arXiv preprint arXiv:2402.10890 (2024).
The idea of introducing human-like reasoning actions and using mutual consistency between SLMs to verify the results during inference time is intriguing. The writing is well-structured and clear. Empirical results with details are given.
The paper introduces rStar, a self-play mutual reasoning approach designed to improve the reasoning capabilities of small language models (SLMs). This method enhances SLMs with prompting engineering. The key mechanism involves a generation-discrimination process where the target SLM creates reasoning trajectories using Monte Carlo Tree Search (MCTS) enriched with human-like reasoning actions. Another SLM, with similar capabilities, acts as a discriminator to verify the generated trajectories, ensuring they are mutually consistent, which increases their likelihood of correctness. Experiments show that rStar effectively boosts performance on challenging reasoning benchmarks.
There are still some design and implementation details related to discriminator dependence, manual action selection, self-reflection, token cost, scalability to larger models that need further clarification.
Yes
No ethics review needed.
- When does performance improvement drop off with increased roll outs? The paper stops at 32, but performance seems to still be improving linearly.
- How does this approach compare to other MCTS methods recently proposed?
The paper's primary originality lies in its novel two-phase approach to improving small language model reasoning. Rather than relying on conventional methods, it creatively combines an enriched Monte Carlo Tree Search with a mutual consistency verification phase using a second SLM. Particularly innovative is the expansion of the MCTS action space to include human-like reasoning operations such as problem decomposition, question rephrasing, and stepwise thinking. The proposed "mutual consistency" verification approach, while used before, is an interesting application that seems very effective.

The authors conduct a comprehensive evauation across a diverse set of benchmarks and models. Additionally, they perform convincing ablation studies that validate their key design choices, particularly the importance of both the expanded action space and mutual consistency verification. The provided baselines are also good.

The majority of the paper is well written and clearly expressed.

The significance of this work is particularly noteworthy. It demonstrates that SLMs possess stronger latent reasoning capabilities than previously believed and provides a practical method for improving SLM reasoning without requiring expensive fine-tuning or supervision from larger models.
This paper tackles the challenge of improving small language models' reasoning abilities without fine-tuning or larger model supervision. The key innovation is a two-phase approach called rStar: first, a generator phase uses Monte Carlo Tree Search with an expanded set of reasoning operations (like decomposing problems, rephrasing questions, and proposing intermediate steps) to create potential solution paths. Then, a discriminator phase uses a second small language model to verify these solutions through "mutual consistency" (checking if the model can arrive at the same conclusion given partial steps). The approach is notable because it improves performance through better inference-time decision making rather than parameter updates or knowledge distillation from larger models. The empirical results demonstrate substantial improvements across multiple reasoning benchmarks and model sizes, suggesting that smaller language models have more latent reasoning capability than previously thought, but need better mechanisms to access it. The authors validate their approach through extensive ablation studies showing the importance of both the expanded MCTS action space and the mutual consistency verification.
The description of rStar in the introduction, starting on line 71, is hard to follow. Perhaps breaking the algorithm down into bullet points would help make the process more explicit and easier to digest than the wall of text. Moving figure 3 to the beginning would also be effective.

On line 255 you say "based on whether it reaches the correct answer" (which would imply you're using the ground truth answer during your search), but it seems like it is simply based on the likelihood of self-consistency majority voting mentioned on line 259.

It seems one potential limitation of mutual reasoning consistency is if an early action makes an incorrect statement that dramatically simplifies the problem. In this case it is likely that $SLM_2$ matches $SLM_1$. Given that this method works so well, this clearly isn't a critial issue, but certainly worth addressing/exploring more (at the very least mention this potential limitation).

Comparisons to other MCTS methods would be nice to have. A quick google search found (on top of the couple cited in the paper) "Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning" and "Improve Mathematical Reasoning in Language Models by Automated Process Supervision". How do these approaches performance compare to yours?
We introduce a self-play mutual reasoning approach that significantly improves SLM reasoning capabilities without fine-tuning or superior models.
@inproceedings{
anonymous2024mutual,
title={Mutual Reasoning Makes Smaller {LLM}s Stronger Problem-Solver},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=6aHUmotXaw},
note={under review}
}
This paper introduces rStar, a self-play mutual reasoning approach that significantly improves reasoning capabilities of small language models (SLMs) without fine-tuning or superior models. rStar decouples reasoning into a self-play mutual generation-discrimination process. First, a target SLM augments the Monte Carlo Tree Search (MCTS) with a rich set of human-like reasoning actions to construct higher quality reasoning trajectories. Next, another SLM, with capabilities similar to the target SLM, acts as a discriminator to verify each trajectory generated by the target SLM. The mutually agreed reasoning trajectories are considered mutual consistent, thus are more likely to be correct. Extensive experiments across five SLMs demonstrate rStar can effectively solve diverse reasoning problems, including GSM8K, GSM-Hard, MATH, SVAMP, and StrategyQA. Remarkably, rStar boosts GSM8K accuracy from 12.51\% to 63.91\% for LLaMA2-7B, from 36.46\% to 81.88\% for Mistral-7B, from 74.53\% to 91.13\% for LLaMA3-8B-Instruct.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
LLM Reasoning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/04ac610460616726cc504c653169a46735d95400.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solver
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission