HARDMath: A Benchmark Dataset for Challenging Problems in Applied Mathematics
math;benchmark;dataset;few-shot learning;reasoning
Yes
NA
No ethics review needed.
1. The description in Section 3.2 is not clear. For example, the paper presents that `sympy` and `scipy` are used to implement the mathematical procedures required for obtaining approximate, analytical solutions. In Appendix A, it only includes some examples with questions and solutions, and I did not see the role of `sympy` and `scipy`.

2. As mentioned in Weakness 2, besides constructing the benchmark, I think this work has the potential to be used for constructing large scale training dataset, teaching the LLMs to solve such hard math problems with the suggested thoughts in CoT, or generated `sympy` code in PoT. What do the authors think about this direction?

3. As my concern of diversity in the Weakness 1, I also have the same concern on the solution ideas. It seems that many problems share the same solution idea by just randomly changing the numbers in the problem. This may be verified by the large performance increase from 0-shot to 5-shot. As the 5 demonstration examples may have already provide the correct solution path, the LLMs only need to change some numbers. Can the author introduce how many solution strategies are used for each type in your dataset?
1. This paper presents a method to generate graduate level math problems (in some pre-defined areas) and corresponding solutions.
This work presents a new math dataset that targets on applied mathematics and requires graduate level knowledges. The problems in this dataset demand a combination of mathematical reasoning, computational tools, and subjective judgment. For example, it may need checks for self-consistency and the use of numerical methods.
1. The type of the math problems in this dataset is limited. First, only three main subjectives are discussed, ODEs, integrals and polynomial. Second, most of the problems are related to calculations, and it seems none of them is proof related. For such problems, the existed numerical computational software should be able to solve most of them. In summary, the diversity of this dataset is limited.

2. Since the dataset can be generated with python tools, it is possible to generate more examples for training. I didn't see any experiment about training on such dataset.
Yes
No ethics review needed.
Have the authors evaluated larger open-source models such as Llama 3.1 405B?
This paper makes an important contribution to the LLM community. The LLM community needs a harder math benchmark with the growing capability of the models. I think this benchmark will be useful.
This paper proposes a new math benchmark for evaluation of LLMs.
I don't see obvious weaknesses.
Yes
No ethics review needed.
Refer to the weakness of paper section.
1- Introducing a dataset that requires approximate analytical solutions is innovative, addressing a gap in current benchmarks. This approach aligns with many real-world problems and reflects how scientists typically tackle them.

2- The dataset is larger than similar graduate-level math datasets. It can be utilized to develop novel prompting techniques or for fine-tuning models.
The author introduces a challenging math dataset focused on asymptotic reasoning, highlighting the use of mathematical approximations to address complex problems that commonly arise in real-world scenarios.
The authors suggest that their dataset could be utilized for fine-tuning LLMs to enhance their capabilities. However, it would be valuable to see empirical results demonstrating this improvement.

The authors explore one-shot and five-shot CoT prompting, noting a substantial performance increase across most models. It would be worthwhile to investigate more complex CoT prompting techniques
Yes
No ethics review needed.
Please see weaknesses.
The topic is interesting and the dataset (at least, a good portion of it) can be useful for the community.

Paper is well written and the experiments are insightful.
This paper proposes a dataset of mathematical problems taken from a graduate level engineering course, and evaluates the ability of several LLMs on solving the problems in the dataset. The problems include topics related to polynomials, root finding, integrals, etc. Paper considers some base problems and modifies them randomly to generate a relatively large dataset. The dataset also includes some manually crafted problems. The accuracy of best LLM goes above 62% in solving the problems in the test set of this dataset. The paper goes on to analyze the failures of LLMs.
In my view, the dataset could have consisted of more difficult problems. From what I read in the abstract and introduction, I was expecting a much harder set of problems to be included in the dataset. The relatively high accuracy of the GPT model on the dataset (above 60%) is also indicative that a considerable portion of the dataset consists of problems that are not as challenging.

Dataset generation method, described in Section 3.2, is still not completely clear to me, even after reading Appendix A. I think the generation procedures need to be explained in more detail such that one can reproduce the generation method by reading the paper and the appendices. Appendix A merely provides some examples of problems in the dataset which I was hoping to see at least one sample in page 1 or 2, rather than the appendix.

The topic of non-dimensionalizing of a polynomial does not strike me as interesting as some of the other topics. In the appendix A, paper provides exact formulas for non-dimensionalizing a polynomial - an exact formula does not need numerical methods as the paper suggests are the basis of all problems in the dataset. Quoting the claim from page 1: “most datasets focus on grade school- to high school-level mathematics problems whose solution methods only involve direct, ‘clean’ calculations. In contrast, HARDMATH targets applied mathematics problems that require approximate analytical solutions.”

When it comes to the so-called Word problems in the dataset, the paper crafts them manually, which is fine. But, it does not sit well with earlier claims of the paper such as “Rather than relying on the typical approach of collecting problems from textbooks, standardized tests, or competitions, as seen in most existing datasets, we developed algorithms to automatically generate problems and their step-by-step solutions.”

Randomly modifying the initial conditions of an ODE or coefficients of a polynomial can of course be done automatically. However, I think it is not fair to contrast such automation with the manual work that was the basis for some of the benchmarks in the literature.
We introduce a new dataset of difficult graduate-level applied mathematics problems; evaluations demonstrate that current leading LLMs exhibit low accuracy in solving these problems.
@inproceedings{
anonymous2024hardmath,
title={{HARDM}ath: A Benchmark Dataset for Challenging Problems in Applied Mathematics},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=nDTvP6tBMd},
note={under review}
}
Advanced applied mathematics problems are underrepresented in existing Large Language Model (LLM) benchmark datasets. To address this, we introduce $\textbf{HARDMath}$, a dataset inspired by a graduate course on asymptotic methods, featuring challenging applied mathematics problems that require analytical approximation techniques. These problems demand a combination of mathematical reasoning, computational tools, and subjective judgment, making them difficult for LLMs. Our framework auto-generates a large number of problems with solutions validated against numerical ground truths. We evaluate both open- and closed-source LLMs on $\textbf{HARDMath-mini}$, a sub-sampled test set of 366 problems, as well as on 40 word problems formulated in applied science contexts. Even leading closed-source models like GPT-4 achieve only 43.8% overall accuracy with few-shot Chain-of-Thought prompting, and all models demonstrate significantly lower performance compared to results on existing mathematics benchmark datasets. We additionally conduct a detailed error analysis to gain insights into the failure cases of LLMs. These results demonstrate the limitations of current LLM performance on advanced graduate-level applied math problems and underscore the importance of datasets like $\textbf{HARDMath}$ to advance mathematical abilities of LLMs.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
math benchmark dataset few-shot learning reasoning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/f02124a1c41644fb1413d17c5053210ea9daf6e5.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/dbeefdc0ef993a80ef8d3f1ac345838c348de1c4.zip
HARDMath: A Benchmark Dataset for Challenging Problems in Applied Mathematics
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission