How to Get Your LLM to Generate Challenging Problems for Evaluation
Evaluation;Synthetic data;Benchmarking;Question Answering;Code Generation;Math Reasoning
Yes
No ethics review needed.
None.
--The paper is written at an impressive quality, especially the figures and the elucidation of the problem's motivation and challenges. 
--The authors consider three sufficiently diverse tasks and benchmarks to showcase the utility of their approach.
--The results are fairly compelling, and the benchmark indeed succeeds in yielding performance drops even from advanced models.
The authors introduce CHASE, a unified framework to synthetically generate challenging problems 
using LLMs without human involvement. For a task that is given, the approach builds a hard problem
 in a bottom-up manner from simpler components. It decomposes the generation process into
independently verifiable sub-tasks to ensure a high level of quality and correctness.

CHASE is designed to address two challenges that the authors state succinctly on pages 1 and 2:
first, how can it be used to create hard and realistic problems, and secondly, how can it be used
to automatically verify the correctness of the generated data? This second challenge is especially
prevalent in other work of this nature that is attempting to construct synthetic evaluation benchmarks.
--Experimental results could have been deeper in the main text. It is for this reason that I am not inclined to give the paper a stellar rating. 
--The approach is simple and has some nice properties, but I am not too sure about its sensitivity and robustness. I felt inadequate attention was paid to this in the paper.
Yes
No ethics review needed.
1. Why does Figure 1 only provide an overview of constructing CHASE-QA and CHASE-Math, but not CHASE-Code? I believe all three should be at the same hierarchical level.
2. Without human verification, how can we ensure that the data in the CHASE-QA, CHASE-Code, and CHASE-Math datasets are correct? Is there a possibility that the ground truth/golden answer in the dataset themselves are incorrect?
3. In lines 333-340, you mentioned that approximately 33% of the data was filtered out through sampling and self-consistency, and subsequent experiments (e.g., Table 2) suggest that CHASE-QA generates more challenging data. I think it is unconvincing. If the 33% of the data were added back, how would the experimental results change? Would you still claim that CHASE-QA is a more challenging dataset?
4. From the examples given in the paper, CHASE-Math seems to concatenate a series of atomic problems. Intuitively, if the tested LLMs reason and calculate sentence by sentence, the accuracy may be significantly higher than under your current naive prompt. Could you elaborate further on how CHASE-Math is more challenging, given the point I raised?
5. What is the motivation behind the experiments in lines 469-477 and lines 486-493? In my understanding, the "Impact of context size" is not the focus of this paper. Also, the experiment in lines 486-493 only fine-tuned weaker models. Would the same conclusion apply to fine-tuning stronger models?
6. Could you provide some comparative experiments between the CHASE dataset and other synthetic datasets, such as a comparison between CHASE-QA and existing long-context benchmarks?
- The problem addressed by this paper is critical to the evaluation of current LLMs -- the lack of comprehensive and challenging datasets.
- The paper is well-structured, with comprehensive appendices, such as a detailed list of prompts used in CHASE.
- This paper presents a novel paradigm for data construction, which may have significant potential in the field of synthetic data.
To address the high cost of manually labeled data and the low quality of synthetic data, this paper proposes the CHASE framework. CHASE is a framework for automatically generating QA pairs, code problems, and math problems. It adopts an innovative bottom-up structure and divides the overall task into individually verifiable sub-tasks. This allows a seed problem to progressively increase in difficulty through multiple rounds of generation and verification. Experimental results show that the data generated by the CHASE have a certain degree of difficulty.
- Some issues with the details of the paper. For example, in the main figure (Figure 1), the bottom-right corner should say "12 pens" instead of "18 pens."
- The current dataset is relatively small, which may result in a high degree of randomness in evaluation results when using this dataset.
- The experiments are not sufficiently thorough. Some experimental designs lack strong motivation, and there is a lack of experiments that demonstrate the advantage of CHASE over other synthetic data generation methods.
Yes
No ethics review needed.
1. How to organize functions into files to build repositories from scratch in CHASE-CODE?
2. Could you specify more details on rejection sampling?
1. The experiments are comprehensive, with a good set of LLMs covering representative proprietary and open-source models. 
2. The paper is well-written, which clearly describes the methods, experiments and results.
This paper introduces CHASE (CHallenging AI with Synthetic Evaluations), a framework for generating challenging evaluation benchmarks using large language models (LLMs). The authors implement CHASE to create benchmarks in three domains: document-based question answering, repository-level code completion, and math reasoning. Experiments with 15 LLMs show that the generated benchmarks are challenging, with even top models achieving only 40-60% accuracy across domains. The authors demonstrate CHASE's utility in differentiating between state-of-the-art models and revealing performance drops with increasing context length. They argue that this approach offers advantages in scalability, renewability, and ability to evaluate tasks difficult for humans to assess, while providing high-quality, challenging problems for LLM evaluation.
1. Although overall I believe it is valuable to explore data synthesis for benchmark construction, I think the authors should be more careful in selecting appropriate settings. I think the most important motivation for this paper is that it is expensive and sometimes impracticable to create benchmarks with challenging problems. However, in some settings present in the paper, I feel that this may not be the case. For example, SWE-bench [1] also focuses on repo-level code generation, and they take existing Github issues as queries, and the modifications made by real users as the ground truth. The current state-of-the-art performance is only 43% in the leaderboard, which indicates its difficulty. Compared to CHASE-CODE, I think the pipeline used in SWE-bench is a better way to collect repo-level code generation data.
2. To demonstrate that this pipeline is scalable, I think it is important to generate data of large size and apply it to training. If the API cost is a concern, I think the authors can use open-source models, e.g., Llama-70B. 
3. Typo in Figure 1: Jill has 12 pens in the bottom right corner.  
4. In line 443-444, I don’t quite understand why better performance of models different from the generator and verifier can indicate better data quality.
5. One advantage of the CHASE claimed by authors is to mitigate data contamination, but I think this may not be a big concern for challenging benchmarks that involve intensive reasoning. For example, even if codellama [2] has been intensively trained on Github data, its performance is still low on SWE-bench (which uses the Github data).

[1]. Jimenez, Carlos E., et al. "Swe-bench: Can language models resolve real-world github issues?." arXiv preprint arXiv:2310.06770 (2023). \
[2]. Roziere, Baptiste, et al. "Code llama: Open foundation models for code." arXiv preprint arXiv:2308.12950 (2023).
We propose a framework for synthetically generating challenging problems to evaluate LLMs.
@inproceedings{
anonymous2024how,
title={How to Get Your {LLM} to Generate Challenging Problems for Evaluation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=2VhFZPYqjE},
note={under review}
}
The pace of evolution of Large Language Models (LLMs) necessitates new approaches for rigorous and comprehensive evaluation. Traditional human annotation is increasingly impracticable due to the complexities and costs involved in generating high-quality, challenging problems, particularly for tasks such as long-context reasoning. Moreover, the rapid saturation of existing human-curated benchmarks by LLMs further necessitates the need to develop scalable and automatically renewable evaluation methodologies. In this work, we introduce **CHASE**, a unified framework to synthetically generate challenging problems using LLMs without human involvement.  For a given task, our approach builds a hard problem in a bottom-up manner from simpler components. Moreover since we want to generate synthetic data for evaluation, our framework decomposes the generation process into independently verifiable sub-tasks, thereby ensuring a high level of quality and correctness. We implement CHASE to create evaluation benchmarks across three diverse domains: document-based question answering, repository-level code completion, and math reasoning. The performance of state-of-the-art LLMs on these synthetic benchmarks lies in the range of 40-60\% accuracy, thereby demonstrating the effectiveness of our framework at generating hard problems. Our experiments further reveal that the Gemini models significantly outperform other LLMs at long-context reasoning, and that the performance of all LLMs drastically drops by as much as 70\% when we scale up the context size to 50k tokens.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Evaluation Synthetic data Benchmarking Question Answering Code Generation Math Reasoning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/d39fa44d1794322cd7d223fc07d5079aed7e9ba0.pdf
datasets and benchmarks
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/98873f3e6e494ea208ecdc1dbc67d23a6b6bbae7.zip
How to Get Your LLM to Generate Challenging Problems for Evaluation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission