Can LLMs Solve Long Math Word Problems Better?
Large Language Models;Math Reasoning;Long Math Word Problems
Yes
No ethics review needed.
- How is E-GSM different from GSM-IC[1]? 

[1] Large Language Models Can Be Easily Distracted by Irrelevant Context. ICML 2023. https://arxiv.org/abs/2302.00093
- This paper introduces E-GSM, a dataset with lengthy, distracting sentences that make it considerably more challenging than the original GSM. This dataset offers a valuable tool for evaluating the robustness of LLMs.

- The approach used to create E-GSM can also be applied to expand existing math training datasets, providing new supervised fine-tuning (SFT) data in the math domain.
This work examines the effect of extended contexts on mathematical reasoning and introduces the Extended Grade-School Math (E-GSM) dataset, featuring math problems with lengthy narratives. Analysis reveals that current LLMs struggle with E-GSM, prompting the authors to propose new methods to address these challenges. 

For proprietary LLMs, they introduce a new instructional prompt, while for open-source LLMs, they develop a novel auxiliary fine-tuning task. These approaches aim to enhance model performance in handling extended-context MWPs.
- The augmented math questions may include contradicting sentences. The augmented math questions  may become unsolvable or yield answers that differ from the original ones. 
Although human evaluations on 200 samples suggest that “94.5% of questions meet acceptable quality,” this accuracy may still be inadequate, particularly given that the labels in the GSM8K test set might contain errors.
An alternative could be to release these 200 samples as a verified subset of the E-GSM dataset. Reporting CoLeG-E and CoLeG-R results on the 200 samples, both with and without verification, would also be helpful.

- In Table 2, the higher results w/ $\mathcal{D}$ (compared to w/ $\mathcal{D_0}$) may because the size of  $\mathcal{D}$ is larger than $\mathcal{D_0}$.
Yes
No ethics review needed.
1. According to “Evaluation results shows that 94.5% questions possess accepatable quality”, the total questions from rounds 1 to 4 should be about 5K. But in Table 1, it is only 4.5K.

2. As shown in Table 1, different rounds have different numbers of questions, what’s the impact on the defined metrics? namely CoLeG-E and CoLeG-R?

3. In Table 2, were the fine-tuned models evaluated with the CoRe method? can they be tested in the same way as those proprietary models? 

4. “Apart from 7,473 annotated examples available in GSM8K training set, we get D0 that incorporate 38,507 valid CoT data points …”, the numbers here confused me. If the authors generated five reasoning paths for each question in the training set, at most, D0 can have 7,473*5 questions, less than 38,507.

5. In Section C.2, “The results suggest scaling up model scales and SFT dataset can further improve CoLeG.”, this conclusion may not be valid. Under CoLeG-R, after the SFT on D0, D1, and D2, the performance is not improved.
The paper explored the impact of question length on LLMs’ performance and proposed a method to extend the length of GSM questions. The paper presented a method called CoRe to help proprietary LLMs better handle these long-form questions. For the open source LLMs, the authors fine-tuned them with a fine-tuning dataset comprising 65K CoT data, created by the authors.
In this paper, the authors investigated the performance of LLMs in solving long math problems. They first examined the performance discrepancy of ChatGPT 3.5 in solving two versions (i.e. long form v.s. short form) of the same questions and concluded that LLMs struggle to answer math word problems with longer context. 

Then, they propose an automatic approach to extend the GSM questions into their long versions (named the obtained dataset E-GSM), with the same computation logic remaining, as far as they could. 
After that, the paper presented a method called CoRe to help proprietary LLMs better handle these long-form questions. 
For the open source LLMs, the authors fine-tuned them with a fine-tuning dataset comprising 65K CoT data, created by the authors. 


The paper introduced E-GSM containing artificial long math problems, but in real cases, there are seldom questions written in the way that the authors presented, i.e. very verbose questions talking about a relatively simple math problem. Therefore, it is unknown whether the conduct here can help in solving real-world long math problems where although the question is quite long, it already describes the problem in a succinct way that it could. Better solving them is our goal, rather than solving the artificial verbose problems that are less likely to exist in the real world. Although they show the same characteristic length-wise, the capability of solving the latter is not necessarily helpful for solving the former.
1. The paper explores the artificial long math problems, but in real cases, there are seldom questions written in the way that the authors presented, i.e. very verbose questions talking about a relatively simple math problem. Therefore, it is unknown whether the conduct here can help in solving real-world long math problems where although the question is quite long, it already describes the problem in a succinct way that it could. Better solving them is our ultimate goal, rather than solving the artificial verbose problems that are less likely to exist in the real world.

2. Besides the above major point, there are more points:
- In Section 2.1, the authors examined the performance discrepancy of ChatGPT 3.5 in solving two versions (i.e. long form v.s. short form) of the same questions and concluded that LLMs struggle to answer math word problems with longer context. However, ChatGPT 3.5 is a relatively weak model now, I would suggest the authors do the same analysis with stronger open-source and proprietary LLMs. 
- Still in Section 2.1, the analysis here is based on real math questions, but the long questions in E-GSM are artificial. Therefore, it is not convincing to me that the conclusion in Section 2.1 can provide a solid foundation for the subsequent conduct. 

3. Many parts are not clear, see the questions section.

4. The writing needs a thorough improvement:
- “Human evaluation details are provided in Appendix A.4.” has a wrong reference. 
- In the first paragraph of Section 3, the subsections should be introduced in order. 
- The second sentence of Section 3.1 has redundancy. 
- The first two sentences of Section 3.2 are not about open-source LLMs, therefore, they cannot help develop this section. The third sentence is redundant. In the fourth sentence, “their generated reasoning paths” should be referred to the place that telling how it is done. The loss function has a typo, should be “ (q, e, a)”.
- Section 3.3, “To negate the influence of few-shot demonstrations”, should be specific, what is the influence? 
- Repeated sentences in the third paragraph of Section 4.1.
Yes
No ethics review needed.
see weaknesses
Strong motivation through rigorous statistical analysis shows LLMs struggle with longer MWPs (Section 2.1)

Proposes creative solutions (CoRe prompting and extension fine-tuning) to address identified limitations

Well-designed metrics (CoLeG-E and CoLeG-R) that capture both efficacy and robustness of LLMs on long MWPs

Sufficient experiments have proven the effectiveness of the method
This paper investigates the ability of LLMs to solve math word problems (MWPs) with longer contexts, introducing the concept of Context Length Generalizability (CoLeG). The key contributions are:(1) Creating Extended Grade-School Math (E-GSM), a dataset of MWPs with extended narratives. (2) Proposing two metrics to evaluate LLMs' efficacy and resilience on E-GSM. (3) Developing tailored prompts for proprietary LLMs to improve CoLeG. (4) Using extension as an auxiliary fine-tuning task for open-source LLMs. (5) Analyzing the impact on semantic understanding vs reasoning efficacy.
The paper focuses on LLMs tackling longer math word problems, rather than genuinely difficult ones. Addressing truly challenging problems would likely yield more impactful and valuable research insights.

A deeper analysis of the types of errors LLMs make on extended MWPs would strengthen the paper. This could shed light on whether mistakes stem from misinterpreting context, losing track of key information, or actual computational errors.

The authors don't explore whether breaking down problems into atomic facts could help solve extended MWPs. It would be worthwhile to compare their methods against a baseline that first extracts crucial information from the lengthy context before attempting a solution. The techniques discussed in https://arxiv.org/abs/2305.14251 could be relevant here.

The table captions should be placed above the tables, not below, to comply with ICLR's official template guidelines.

The "Experimental Setup" section doesn't belong under Methodology. It should be moved to the Experiments section, alongside the results analysis.
Yes
N/A
No ethics review needed.
1. How does the performance degradation on longer MWPs correlate with specific architectural features of different LLMs, such as context window size or attention mechanisms?

2. The extension approach shows promise for open-source LLMs. Have you considered how this might be adapted for extremely long MWPs or multi-step reasoning problems that span multiple pages?
1. The paper addresses a gap in current research by focusing on LLMs' ability to handle longer MWPs, which is more reflective of real-world mathematical reasoning tasks. The focus on CoLeG provides insights into the limitations of current LLMs and pathways for improvement.

2. The creation of the E-GSM dataset through a systematic extension process is another contribution. By maintaining problem difficulty while increasing context length, the authors have developed a framework for evaluating LLM performance on longer MWPs.

3. The introduction of CoLeG-E and CoLeG-R metrics offers a more comprehensive evaluation framework than traditional accuracy measures. These metrics provide insights into both the consistency and robustness of LLM performance across varying context lengths.

4. The proposed methods, CoRe and extension-based fine-tuning, show consistent improvements across different LLM types and generalize well to other benchmarks.
This paper investigates the performance of LLMs on Math Word Problems with extended narratives, introducing the concept of Context Length Generalizability (CoLeG). The authors created a new dataset, Extended Grade-School Math (E-GSM), by iteratively extending problems from GSM8K. They propose two novel metrics, CoLeG-E and CoLeG-R, to evaluate efficacy and robustness respectively. The study reveals that existing LLMs struggle with longer MWPs, showing a consistent performance decline as context length increases. To address this, the authors introduce Condition-Retrieving Instruction (CoRe) for proprietary LLMs and an extension-based fine-tuning approach for open-source LLMs. These methods demonstrate improvements in CoLeG across various LLM types and generalize well to other MWP benchmarks as well.
1. The paper lacks a detailed exploration of why longer contexts impact LLM performance. While the authors mention potential working memory limitations, a deeper analysis could provide valuable insights. For instance, examining how performance correlates with the models' context window sizes or investigating the behavior of attention patterns in different layers could shed light on where breakdowns occur. Additionally, analyzing how different positional encoding schemes (e.g., rotary position embeddings vs. absolute position embeddings) affect performance on longer MWPs could offer insights into architectural considerations for improving CoLeG.

2. The E-GSM dataset creation process, while systematic, may introduce biases that aren't adequately addressed. Using GPT-4 for extensions could potentially lead to biases in language style, problem structure, or even subtle cues that GPT-4 uses for reasoning. For example, GPT-4 might consistently use certain phrases or sentence structures that inadvertently serve as hints for other GPT models. Additionally, there's a risk of amplifying any biases present in the original GSM8K dataset. The authors should consider analyzing the distribution of problem types, linguistic patterns, and solution strategies in E-GSM compared to the original dataset to identify any systematic biases introduced during extension.

3. The evaluation of open-source LLMs is limited to LLaMA-2 and Mistral-7B families. To provide a more comprehensive assessment, the authors should consider including models specifically designed for mathematical reasoning, such as MathGPT, GPT-f, or MetaMath. Additionally, evaluating performance on models with different architectural choices, like PaLM or BLOOM, could offer insights into how various model designs handle longer MWPs. This broader evaluation would strengthen the claims about the generalizability of the proposed methods.

4. While the paper shows improvements on other MWP benchmarks, it doesn't explore how the proposed methods perform on problems significantly longer than those in E-GSM. This leaves questions about the scalability of the approaches to even more complex, multi-page word problems. The authors could consider creating a small set of extremely long MWPs (e.g., 1000+ tokens) to test the limits of their methods and provide insights into scaling challenges.

5. The use of GPT-3.5-turbo for answer extraction in the evaluation process introduces a potential confounding factor. The paper doesn't adequately address how this might impact results, especially for non-OpenAI models. The authors should consider comparing this extraction method with simpler rule-based approaches or using model-specific output parsing to ensure fair comparison across different LLM families.
We investigate the impact of long contexts on mathematical reasoning abilities of large language models.
@inproceedings{
anonymous2024can,
title={Can {LLM}s Solve Long Math Word Problems Better?},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=C9ju8QQSCv},
note={under review}
}
Math Word Problems (MWPs) play a vital role in assessing the capabilities of Large Language Models (LLMs), yet current research primarily focuses on questions with concise contexts. The impact of longer contexts on mathematical reasoning remains under-explored. This study pioneers the investigation of Context Length Generalizability (CoLeG), which refers to the ability of LLMs to solve MWPs with extended narratives. We introduce Extended Grade-School Math (E-GSM), a collection of MWPs featuring lengthy narratives, and propose two novel metrics to evaluate the efficacy and resilience of LLMs in tackling these problems. Our analysis of existing zero-shot prompting techniques with proprietary LLMs along with open-source LLMs reveals a general deficiency in CoLeG. To alleviate these issues, we propose tailored approaches for different categories of LLMs. For proprietary LLMs, we introduce a new instructional prompt designed to mitigate the impact of long contexts. For open-source LLMs, we develop a novel auxiliary task for fine-tuning to enhance CoLeG. Our comprehensive results demonstrate the effectiveness of our proposed methods, showing improved performance on E-GSM. Additionally, we conduct an in-depth analysis to differentiate the effects of semantic understanding and reasoning efficacy, showing that our methods improves the latter. We also establish the generalizability of our methods across several other MWP benchmarks. Our findings highlight the limitations of current LLMs and offer practical solutions correspondingly, paving the way for further exploration of model generalizability and training methodologies.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Large Language Models Math Reasoning Long Math Word Problems
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/afbf68f976ad02252ff67e1474fa3d887541eb74.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Can LLMs Solve Long Math Word Problems Better?
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission