Learning to Plan Before Answering: Self-Teaching LLMs to Learn Abstract Plans for Problem Solving
LLM;self-training;high-level abstraction;self-reflection;meta learning;anticipatory plans
Yes
No ethics review needed.
1. Will it work on other LLM?
2. How to deal with questions which have no correct answer after data creation pipeline?
3. How to better choose data?
1. Novel self-improvement method to improve the LLM's reasoning capability.
This paper introduces a self-training approach aimed at enhancing large language model (LLM) reasoning.
LEPA addresses the need for LLM self-generated data to contain meta-knowledge, rather than merely task-specific steps, enabling better generalization. By integrating anticipatory plans, LEPA provides a structured approach to problem-solving, reducing cognitive load, enhancing solution quality, and preventing the generation of incorrect rationales for correct answers.
1. Only improve in-domain performance. 
2. No OOD experiments. 
3. Only one LLM (Llama 3 8B Instruct) is evaluated here.
4. Data generation pipeline is not clear illustrated. 
5. Experiment section is not convincing with so few benchmarks.
Yes
No ethics review needed.
1. Presentation of Table 3 is confusing and cannot figure it out in the title. Could you please explain the metrics in the Table and the relation of three variants of tokens to the three methods?
2. Have you ever tested the effectiveness of LEPA on other LLMs?
3. How do you think we can optimize the model when the solution to a reasoning problem is open-ended such that the evaluation of solution correctness is hard?
1. The paper addresses an important problem that what essential information of self-generated data should an LLM learn during self-training, and considers an anticipatory plan as the essential information.
2. Experiment results show the the anticipatory plan can guide an LLM to generate correct solutions after the plan is refined iteratively through self-reflection and added to the training data for further fine-tuning.
3. The ablation studies validate the usefulness of key components of the proposed method.
The paper addresses the problem of what information should self-generated data encapsulate in the LLM self-training, by prompting an LLM (specifically Llama 3.1-8B-Instruct in this paper) to first generate an anticipatory plan before generating solutions to reasoning problems. Then optimize the anticipatory plan via self-reflection using the LLM itself until the plan leads to a correct solution. Once the corrected solution is generated, the reasoning problem and the plan as well as the solution are added to the training dataset to fine-tune the LLM.

Experimental results across four reasoning benchmarks show better performance compared to three baselines, especially after more training iterations.
1. Although the paper claims that the proposed method is also compatible with more sophisticated RL algorithms, the current version does not support the claim. If an RL algorithm is incorporated, the method will be quite different from the proposed one as the key components of current methods (e.g., self-reflection and fine-tuning method) should be significanly altered. So the performance of current method is limited to the LLM performance via prompting and self-reflection.
2. The justification of the anticipatory plan in Section 2.3 is not solid theorectically by simply referring to some work in cognitive science and the example shown in Figure 1.
3. The idea of generating a plan before solutions is similar to recent work on generating a plan using an LLM before actions for emobodied agents (e.g, [1][2]), however, the paper does not address the related work in Section 4. Also Section 4 lacks related work on self-reflection of LLMs (e.g, [3][4]), which is a key component of the proposeed method.

References:

[1] Wang, Z., Cai, S., Chen, G., Liu, A., Ma, X., and Liang, Y. Describe, explain, plan and select: interactive planning with llms enables open-world multi-task agents. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.

[2] Yao, Shunyu, et al. "React: Synergizing reasoning and acting in language models." arXiv preprint arXiv:2210.03629, 2022.

[3] Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023.

[4] Shinn, N., Labash, B., and Gopinath, A. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366, 2023.
Yes
No ethics review needed.
1. In Table 1, the zero-shot CoT performance for llama3-8b-instruct is only 19%, which is a little bit too low. I acknowledge that the result would be different due to the prompts or something, but 19% is not very convincing to me. I would recommend you use other evaluation methods, e.g., simple-eval (https://github.com/openai/simple-evals) to re-evaluate the answer of the MATH dataset.

Also, see the comments on weaknesses. If issues are solved, I will improve my score.
1. The highlighted advantage is that the idea is simple and straightforward. It’s also quite clear why this would help performance, and the authors present it well with clear figures and examples. The method is easy to implement and can be easily generalized to other tasks, which makes this paper impactful.

2. The experiments are reasonable and effectively demonstrate the superiority of the method. There is sufficient analysis, including an analysis of inference computation.
This paper introduces a novel training method LEPA for reasoning tasks. Unlike previous approaches that only focus on optimizing answer generation, LEPA also fine-tunes the LLM for plan generation. This is because planning represents a high-level abstract concept (meta-knowledge) that presents generalizable patterns across similar problems. LEPA prompts the LLM to generate an anticipatory plan and refine it until the plan can guide the LLM to a correct answer. After gathering the training data, LEPA uses SFT to optimize both the planning and answering generation. LEPA significantly enhances performance across various reasoning tasks.
1. My biggest concern is that this method may negatively affect the experience of using LLMs for simple problems. The current approach encourages the LLM to have careful planning before answering, which is unnecessary for simple problems and incurs unnecessary costs. In contrast, ReSF does not have this issue.

2. Secondly, I am concerned that this merely replaces the solution with an easily memorable method. I am curious whether it will provide sufficient improvement for few-shot scenarios, e.g., trained within a few MATH instances and tested with the others, or out-of-distribution reasoning problems, e.g., trained within the MATH dataset and tested with math questions in MMLU-pro.

3. More LLMs in experiments would be better. But this is not a big issue.
Yes
No ethics review needed.
1. In line 224, you claim that LEPA can "preventing the creation of false-positive data". However, even STaR can't prevent the creation of (step-level) false-positive data. It's even harder to judge whether the planning is correct/incorrect/meanless. So maybe you should give a evaluation standard to judge the quality of the planning data (and talking what is the defination of false-positive in planning).

2. Why not compare with other baselines mentioned in related work, especially those used MCTS?

3. Add some OOD benchmark (like MATH).

4. Discuss the reason why the planning data can improve the reasoning ability of the LLMs.
1. The paper is easy to follow.

2. The results show that planning data improves the performance of the LLMs.

3. The results in Figure 3 are impressive, the LEPA seems to continuously improve the performance even after 15 iterations.
The author present LEPA, a data generation pipeline. The data will not only contain the COT answer but also the planning process. The experiment shows that the LEPA beat other baselines in Hellaswag, Hendrycks MATH, BoolQ, and PIQA. The ablation study shows the importance of the self-peflection process and planning data in the data generation pipeline.
1. Training planning data is not a new idea, paper like LUMOS[1] (specifically LUMOS-O) has already shown that planning data can improve the performance of the LLMs, although I admit that the LEPA is more simple than LUMOS-O.

2. There is only outcome judgment in data selection. Actually, LEPA doesn't judge the quality of the planning as well as the quality of the COT before the answer.

3. The comparison in Table 2 is a bit unfair. The LEPA's training data is (likely) bigger than the "Without Plan/Without Self-Reflection" settings if the settings are aligned with lines 349-360.


[1] LUMOS: LEARNING AGENTS WITH UNIFIED DATA, MODULAR DESIGN, AND OPEN-SOURCE LLMS
We propose a novel self-training algorithm that self-teaches LLMs to generate high-level abstract plans before solving problems.
@inproceedings{
anonymous2024learning,
title={Learning to Plan Before Answering: Self-Teaching {LLM}s to Learn Abstract Plans for Problem Solving},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=KmmNb7631I},
note={under review}
}
In the field of large language model (LLM) post-training, the effectiveness of utilizing synthetic data generated by the LLM itself has been well-presented. However, a key question remains unaddressed: what essential information should such self-generated data encapsulate? Existing approaches only produce step-by-step problem solutions, and fail to capture the abstract meta-knowledge necessary for generalization across similar problems. Drawing insights from cognitive science, where humans employ high-level abstraction to simplify complex problems before delving into specifics, we introduce a novel self-training algorithm: LEarning to Plan before Answering (LEPA). LEPA trains the LLM to formulate anticipatory plans, which serve as abstract meta-knowledge for problem-solving, before engaging with the intricacies of problems. This approach not only outlines the solution generation path but also shields the LLM from the distraction of irrelevant details. During data generation, LEPA first crafts an anticipatory plan based on the problem, and then generates a solution that aligns with both the plan and the problem. LEPA refines the plan through self-reflection, aiming to acquire plans that are instrumental in yielding correct solutions. During model optimization, the LLM is trained to predict both the refined plans and the corresponding solutions. By efficiently extracting and utilizing the anticipatory plans, LEPA demonstrates remarkable superiority over conventional algorithms on various challenging natural language reasoning benchmarks.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
LLM self-training high-level abstraction self-reflection meta learning anticipatory plans
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/59dbe6afa90c5b404e91d99d3075ca1856033585.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/a392fe8a91e3c12cbad4064870b7112644b1123d.zip
Learning to Plan Before Answering: Self-Teaching LLMs to Learn Abstract Plans for Problem Solving
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission