Unveiling the Magic of Code Reasoning through Hypothesis Decomposition and Amendment
Code Reasoning;Hypothesis Decomposition;Reflection
Yes
No ethics review needed.
Can you provide any empirical evidence to back up the many claims made in the article?
The approach taken has significant potential.
This article presents a dynamic prompting method that automatically adjusts the number of prompting steps based on task complexity and model performance in real time.
This paper is not suitable for a computer-science/ML conference like ICLR. It seems best suited to a cognitive psychology or philosophy conference.
The paper starts off in line 42 with "From the perspective of human cognitive psychology, reasoning can be viewed as a process of memory retrieval," and this is the perspective taken. There is insufficient mathematical theory or implementation to show results relevant to ICLR.

There are several claims made without defintion and/or validation. Examples include:

 - LLMs perform better on System 1 tasks than on System 2 tasks.    ???? Must define System 1 tasks and System 2 tasks for this to make any sense.

 - l. 52: "We believe that, similar to humans, there is no significant distinction between memorizing and reasoning tasks for LLM

----this is a scientific document NOT a statement of unprovable beliefs.

 - l. 112: what is "the execution space"?

 - l. 115: what is "abstract level understanding of function’s behavior" or "perform abductive inference"  ?
Yes
No ethics review needed.
1. Example analysis is said to be shown in Appendix E, but Appendix E is empty. 
2. How sensitive is the method to the quality of the initial decomposition? Are there cases where poor initial decomposition cannot be recovered through amendments?
3. When and why does the RHDA pipeline fail - are the failures primarily due to poor initial decompositions, ineffective amendments, or do different tasks exhibit different failure patterns?
4. The computational overhead analysis is missing:
 -  What is the computational cost compared to baseline methods? 
 -  What is the trade-off between the number of iterations and performance gains?
5. While VirtualHome examples demonstrate RHDA's potential for real-world tasks, can you provide quantitative metrics and more complex scenarios (e.g., handling multiple interdependent tasks or recovering from failures) to better evaluate its practical capabilities?
1. Novel formulation and pipeline:
    - The paper introduces an intermediate task between reasoning and recall, giving us the ability to understand LLMs more deeply
    - RHDA pipeline demonstrates robustness in handling complex reasoning tasks
    
2. Comprehensive Evaluation: 
    - Three types of meta-framework with 8 specific benchmarks
    - Evaluation across different model types (GPT-4o and Claude 3.5)
    - Thorough ablation studies show the importance of each component.
3. Strong Results:
    - On inductive reasoning: RHDA outperforms baselines by 18.45% (List Function), 5.89% (MiniARC), 33.31% (RobustFill), and 12.02% (DeepCoder)
    - On deductive reasoning: Achieves 90.62% on CRUXEval and 84.16% on LiveCodeBench
    - On abductive reasoning: Achieves 83.75% on CRUXEval and 71.57% on LiveCodeBench
4. Extensibility: Successfully demonstrated application to VirtualHome environment, showing the framework can handle real-world complex scenarios like household tasks.
The paper presents "code reasoning" as a novel task requiring both reasoning and memory to explore the capabilities and boundaries of LLMs. The authors propose three new meta-benchmarks based on different forms of logical reasoning, namely: inductive code reasoning (predicting program from input-output pairs), deductive code reasoning (predicting output from input-program), and abductive code reasoning (predicting input from output-program). They instantiate these meta-benchmarks into eight concrete benchmarks: List Function, MiniARC, RobustFill, and DeepCoder for inductive reasoning, and CRUXEval and LiveCodeBench for both deductive and abductive reasoning. They propose a novel pipeline, "RHDA" (Reflective Hypothesis Decomposition and Amendment), that iteratively decomposes complex problems into simpler steps, verifies the results, and based on this feedback proposes amendments. This pipeline leads to as much as 3× performance gains over baseline models.
1. Limited Theoretical Analysis: 
  - While the paper positions code reasoning between memory and reasoning tasks, it lacks a deeper theoretical analysis of why this intermediate position is beneficial or how it relates to existing theories of reasoning.
2. Limited Real-World Validation: 
 - While they show the program is capable of performing in VirtualHome, having quantitative benchmarks and more complex scenarios would be helpful to understand real-world applicability.
3. Incomplete Error Analysis: 
- Additional discussion on the failure modes of the RHDA pipeline would be helpful. Analysis of whether failures occur due to incorrect hypothesis formation, ineffective amendment etc is missing.
4. Complexity Overhead: 
- The paper doesn't analyze the computational cost of running multiple iterations of RHDA compared to single-pass methods like CoT or PoT. This is particularly important given the iterative nature of the approach.
Yes
No ethics review needed.
1. What is the difference between Sub-Hyp and Chain of Thought?

2. I didn't get the rationale of why the IO prompt is the best in terms of accuracy. More specifically, what does it mean when you say this? On page 6, the end of the paragraph preceding the ablation study says "This may be a misunderstanding... than RHDA."?

3. At the bottom of page 7, "...we present a quantitative...", did you mean qualitative?

4. Did you conduct a full-scale experiment on Virtual Home? It seems that you only present two examples in the paper, which I'm not sure can generalize to the conclusion that RHDA is scalable and flexible.

5. What is the cost of RHDA? Does it generalize to other non-symbolic domains?

6. How does your method compare to self-refine?
originality: RDHA is a novel method. It is interesting to see such a method to be applied to code reasoning and even planning tasks (e.g., Virtual Home).

quality: the paper conducts a systematic investigation of code reasoning tasks and an additional planning task, which is a very comprehensive task setting.

clarity: the paper is generally easy to follow.

significance: it is important to understand LLMs' capabilities and limitations.
This paper evaluates LLMs on code reasoning tasks, which require both recall and reasoning capabilities. The authors also propose a method RDHA which generally improves their performance in these tasks.
Generally, the contributions of this paper are not strong enough.

1. If the main contribution is the evaluation of LLMs on code reasoning, it is not enough to test only two LLMs. Plus, it is not surprising that LLMs fail in complex reasoning tasks as widely testified in many existing works ([1,2], inter alia). 

2. If the strongest contribution is RHDA, I wonder how it compares to other prompting methods such as self-refine [3].

2. Some arguments are a bit loose or controversial. For instance, the authors "...position the code reasoning task between memory and reasoning" because it requires memory of syntax from pre-training and reasoning about current question and context. In this sense, most, if not all of the reasoning tasks can be put in the middle of the Figure 1 spectrum, which has one extreme of reasoning and another extreme of recall. For instance, math, which the authors put on the reasoning extreme of the spectrum, requires the LLM to understand what operators mean as well (i.e., syntax, which also needs to come from training memory). The authors are encouraged to be more cautious in statements.

[1] Zhuo, T. Y., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., ... & Von Werra, L. (2024). Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877.

[2] Dziri, N., Lu, X., Sclar, M., Li, X. L., Jiang, L., Lin, B. Y., ... & Choi, Y. (2024). Faith and fate: Limits of transformers on compositionality. Advances in Neural Information Processing Systems, 36.

[3] Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., ... & Clark, P. (2024). Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36.
@inproceedings{
anonymous2024unveiling,
title={Unveiling the Magic of Code Reasoning through Hypothesis Decomposition and Amendment},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=kN25ggeq1J},
note={under review}
}
The reasoning abilities are one of the most enigmatic and captivating aspects of large language models (LLMs). Numerous studies are dedicated to exploring and expanding the boundaries of this reasoning capability. However, tasks that embody both reasoning and recall characteristics are often overlooked. In this paper, we introduce such a novel task, code reasoning, to provide a new perspective for the reasoning abilities of LLMs.
We summarize three meta-benchmarks based on established forms of logical reasoning, and instantiate these into eight specific benchmark tasks. Our testing on these benchmarks reveals that LLMs continue to struggle with identifying satisfactory reasoning pathways.
Additionally, we present a new pathway exploration pipeline inspired by human intricate problem-solving methods. This \textbf{R}eflective Hypothesis Decomposition and Amendment (RHDA) pipeline consists of the following iterative steps: (1) Proposing potential hypotheses based on observations and decomposing them; (2) Utilizing tools to validate hypotheses and reflection outcomes; (3) Revising hypothesis in light of observations. Our approach effectively mitigates logical chain collapses arising from forgetting or hallucination issues in multi-step reasoning, resulting in performance gains of up to $3\times$. Finally, we expanded this pipeline by applying it to simulate complex household tasks in real-world scenarios, specifically in VirtualHome, enhancing the handling of failure cases. We release our code and all of results at \url{https://anonymous.4open.science/r/code_reasoning}.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Code Reasoning Hypothesis Decomposition Reflection
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/711dc12dde189f8254a99090b90fd903e89dfb82.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Unveiling the Magic of Code Reasoning through Hypothesis Decomposition and Amendment
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission