DOTS: Learning to Reason Dynamically in LLMs via Optimal Reasoning Trajectories Search
large language model;reasoning
Yes
No ethics review needed.
See weaknesses.
1. The authors propose a dynamic reasoning method which can enable the model to decide the appropriate atomic actions based on the characteristics of the input question.
2. The authors conduct comprehensive experiments to prove the effectiveness of the proposed method, containing in distribution, few-shot, and out-of-distribution settings.
3. The proposed method can be used on both open-source and close-source models.
This paper proposes DOTS, an approach enabling LLMs to reason dynamically via optimal reasoning trajectories search, tailored to the specific characteristics of each question. DOTS first defines atomic reasoning action modules, then searches for the optimal action trajectory for each training question through iterative exploration and evaluation for the specific task-solving LLM, and uses the collected trajectories to train a model to plan for the reasoning trajectories of unseen questions.
1. The method proposed in this paper does not show significant improvement on out-of-distribution (OOD) tasks, and it incurs additional computational overhead compared to prompt engineering methods.
2. The baseline for Vanilla SFT only used the training data from CoT. I believe it should be compared with baselines from other reasoning formats to demonstrate the effectiveness of the proposed method, such as using the program reasoning format and mixed training data from CoT and Program.
3. The experiments demonstrating the effectiveness of the Search section are compared with randomly selected reasoning paths. I am curious about how the results would compare if we contrasted it with reasoning paths generated by combining non-empty actions from each layer.
4. Using PoT prompts on non-code models may yield suboptimal results. Would combining PoT with code models improve the results of the PoT baseline?
Yes
No ethics review needed.
Q1. How many trajectories are required to improve performance substantially at FT-time? A more thorough discussion would be useful.

Q2. How does the proposed approach scale to more complex problems where process rewards would be beneficial?
1. Adaptability of the method: The method can be used either on the planner or on the task solver.

2. The method shows lower cost than non-CoT methods.

3. Improved Reasoning ability: The paper demonstrates improved reasoning ability by allowing for dynamic selection for a given question, outperforming static methods as well as self refinement (most but not all scenarios).
The paper proposes a new way to learn reasoning through two learning paradigms. The paper (DOTS) proposes both a way to FT an planner LLM and in directly FT-ing the task solver.
1. The proposed method is complex and involves several steps. It is not clear if the complexity is warranted and ablation studies on the various aspects can help.

2. The paper does not explore decomposition or tool use which would be crucial for complex tasks.

3. The paper glosses over how the atomic trajectories are collected.
Yes
No ethics review needed.
+ (Type) Equation 2: $T$ has not been defined.
+ (Clarification) For the "few-shot" settings, do you use the planner trained on MATH dataset? If so, they should be "near-distribution + few-shot", since those "few-shot" datasets are math-related.
+ Since DOTS has marginal performance on OOD tasks (despite DOTS need additional training), what's the performance on near-distribution (e.g. the few-shot datasets in the paper) but zero-shot settings?
+ In Table 3, 4, 5, on a few datasets DOTS is not the best among methods. Does it mean in some datasets, it's better to choose one single baseline method instead of choosing modules dynamically as in DOTS?
+ DOTS select one reasoning method (one choice for each module). Would it be better to attempt multiple choices and choose the best one (and this would align more with "trajectory search")?
+ (Minor) It would be beneficial to also include the computing efficiency of external DOTS.
DOTS unifies many of recent LLM prompting reasoning methods, and can dynamically choose better method for each module for each data sample. It borrows the strength of various solutions in literature, and can possibly incorporate future works.

The paper contains comprehensive experimental results, on various datasets across domain, and in/near/out-of-distribution experiments. The experiments show superior performance of DOT to previous prompting methods consistently.

The paper also includes ablation studies to verify the importance of each of the breakdown modules.

It also includes analysis of preferences of DOTS on different tasks, with insightful explanations. It also includes an analysis of efficiency, showing comparable computing costs to other advanced prompting techniques.
The paper proposes DOTS, which can dynamically choose prompting methods to do reasoning, by dividing reasoning into analysis, solution, and verification modules, and choosing from recent popular methods for each module.

The dynamic planner can be either external or internal to incorporate various LLMs with difference sizes and openness.

Experiments are conducted on in-distribution, near-distribution, and out-of-distribution settings, and zero-shot and few-shot evaluations, where DOTS outperforms most recent LLM prompting reasoning methods.
Although DOTS has higher average performance, DOTS cannot consistently beat baselines in all tasks. This is a little surprising as DOTS should be able to be considered as superset of all baselines. On those dataset DOTS fall behind baselines, it would mean it's not necessary to do dynamical reasoning (as one can choose the baseline instead of choosing different modules for each data sample).

This phenomenon is more often in out-of-distribution settings. The average score on out-of-distribution also shows smaller margin. This means lack of training (in a domain) can lead to much worse performance of DOTS. However, as also mentioned by author, such training set may be not available, casting generalizability concern on DOTS.

It would be beneficial to show the comparison on near-distribution zero-shot settings (e.g. the few-shot datasets in the paper) to verify that only near-distribution training is needed, but the paper only includes few-shot settings on these datasets. (also see questions)

To clarify, the few-shot experiments with consistent outperformance can partly address the point, but I would expect better zero-shot performance since training is involved.
Yes
No ethics review needed.
1. could you present the distribution of the path the planner will choose in the experiment? (i.e. x% of the path will choose Empty + POT + self-verifications, y % of the path will choose Decomposition + COT + Empty, etc.) (This is different from Table 7 in 2 parts: 1. It shows the distribution of the 12 types of combination of 3 layers in the path, 2. It needs to contain more benchmarks besides the MATH dataset)

2. Can 2 findings in Section 3.7 be generalized to other datasets?  

3. (From weakness 1) The authors should add a comparison with the baseline with 3 layers (and the choice of the layer should be reasonable towards a given benchmark) to make the comparison more fair.

4. From my perspective, the OOD benchmark is more important than the IND benchmark, however, the authors only show the ablation study in the IND benchmark. Why?

5. The improvement of the DOTS seems to come from the explanation (if you compare the result in Table 6's "-w/o Explanation" line with the second to the last line in Tables 3 and 4). So, does the prior knowledge in the prompt cause the improvement? 

6. (minor) The author claims that "the fine-tuned LLMs are constrained to follow the same reasoning format of the training data (e.g., CoT (Luo et al., 2023)) and lack the flexibility to adopt other reasoning strategies" (in line 53,77-78). However, FireAct[1], shows that the fine-tuned LLMs can change the reasoning strategies (COT or ReAct) by itself without any additional prompt. Maybe training COT and POT at the same time can help the model to be more flexible. The authors should revise this point to make it more accurate.

7. In Internalized setting, I'm curious that if llama3-8b is trained only in the Reasoning Process and Answer (like FireAct), what performance will be, will the trained model choose the Reasoning Process automatically? (It should also be a baseline), although I admit this can't be used in external setting.

8. (minor) In Table 2, "Few-shot" is not a kind of Distribution, maybe 4 columns (add example number column) can be more clear.

[1] FIREACT: TOWARD LANGUAGE AGENT FINE-TUNING
1. The paper is easy to read and follow. Its ideas are presented straightforwardly along with illustrations.

2. It simplifies the planning prompt construction process. 

3. The authors study the DOTS over a wide range of benchmarks.
The author creates a planning prompt strategy named DOTS. It can choose the path to solve the problem via a small model (as the planner) and prompt the large model to solve the problem. The path contains 3 parts: the Analysis Layer, the Solution Layer, and the Solution Layer. The experiment shows that both external and internalized planner can help the large model to solve the problem more effectively and efficiently. The ablation study also shows that the searching and explanation in planner's plan is useful.
1. The comparison between the DOTS and other methods is not fair. The DOTS's prompt contains 3 layers, however, the baselines only contain 1/2 layer. The authors should compare the baseline with 3 layers (for example compulsory use of Decomposition + POT + self-verifications at the same time in MATH) to make the comparison more fair.

2. The method is not generalizable enough. It needs to SFT a new model for each large model / new atomic action in layers / new layers. (Although the authors claim that this is an advantage to make the prompt more suitable for a given model, it is still a limitation for the method.)

3. The improvement of the DOTS is not significant, especially in the OOD dataset. 

4. The ablation study only shows the result in the IND benchmark 

5. The ablation study in Table 6 does not explain the inherent reason that DOTS can improve performance, it's not enough for ICLR. (For example, can a large model have a similar performance with DOTS by only adding prior knowledge in the explanation?). (see question 3,4,6 below for more details)
We propose a method to dynamically select an optimal trajectory of reasoning actions tailored to the specific characteristics of questions and LLMs.
@inproceedings{
anonymous2024dots,
title={{DOTS}: Learning to Reason Dynamically in {LLM}s via Optimal Reasoning Trajectories Search},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=tn2mjzjSyR},
note={under review}
}
Enhancing the capability of large language models (LLMs) in reasoning has gained significant attention in recent years. Previous studies have demonstrated the effectiveness of various prompting strategies in aiding LLMs in reasoning (called "reasoning actions"), such as step-by-step thinking, reflecting before answering, solving with programs, and their combinations. However, these approaches often applied static, predefined reasoning actions uniformly to all questions, without considering the specific characteristics of each question or the capability of the task-solving LLM. In this paper, we propose DOTS, an approach enabling LLMs to reason Dynamically via Optimal reasoning Trajectories Search, tailored to the specific characteristics of each question and the inherent capability of the task-solving LLM. 
Our approach involves three key steps: i) defining atomic reasoning action modules that can be composed into various reasoning action trajectories; ii) searching for the optimal action trajectory for each training question through iterative exploration and evaluation for the specific task-solving LLM; and iii) using the collected optimal trajectories to train an LLM to plan for the reasoning trajectories of unseen questions. In particular, we propose two learning paradigms, i.e., fine-tuning an external LLM as a planner to guide the task-solving LLM, or directly fine-tuning the task-solving LLM with an internalized capability for reasoning actions planning. Our experiments across eight reasoning tasks show that our method consistently outperforms static reasoning techniques and the vanilla instruction tuning approach. Further analysis reveals that our method enables LLMs to adjust their computation based on problem complexity, allocating deeper thinking and reasoning to harder problems.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
large language model reasoning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/8c92bc7b0e48e618efe25f04138a5db0fee97dcf.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
DOTS: Learning to Reason Dynamically in LLMs via Optimal Reasoning Trajectories Search
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission