DataGen: Unified Synthetic Dataset Generation via Large Language Models
large language model;evaluation;synthetic data
Yes
No ethics review needed.
- What is the performance of each module given different generator models?
- There's comprehensive work into each of the target attributes (generalization, controllability, diversity, and truthfulness)
- The methodology is highly detailed, including comprehensive ablations, evaluations, and cost details.
- The details about what synthetic generations other LLMs perform well and poorly on are helpful for further work into synthetic benchmarks.
The authors present a framework for generating synthetic datasets that focus on generalization, controllability, diversity, and truthfulness by guiding the generation with attributes, checking diversity within a clique, performing code-based verification for reasoning tasks, and performing RAG to verify facts. The authors also show what types of synthetic benchmarks LLMs excel at and fail at.
- There could be more side-by-sides of questions from the original dataset and each generated dataset.
Yes
No ethics review needed.
**Section 3.8 Benchmarking LLM**. The paper mentioned "challenging nature of Claude-3 generated dataset". Do different LLM uses the same / different prompts? 

**Section 3.3 Effectiveness of Modules in DataGen**. 
- Can you explain more on remote-clique score? Why is it a good metric, how exactly is it calculated on the generated dataset (using embeddings, or other representation of the dataset)?
- The delta of remote-clique score of HellaSwag is significantly higher than other datasets. Why is that?
**Novelty and Significance**. The paper presents a novel technique and artifact for the field of synthetic data generation. DataGen is generalizable to other domains and tasks, though with additional overhead. Compared to other related work, DataGen is able to cover a wide range of features in real settings. The artifact is available and runnable. 

**Writing**. The writing is clear and well-organized, with clear visual / tables to summarizes the comparison, methodology, evaluation, and ablation studies. The motivation of the paper is very clear, the problem is well-defined, key contributions are listed and aligned with the structure of the paper. The visual elements in the paper are very helpful to understand the paper.

**Methodology**. The proposed framework is well-motivated, and the framework design is simple and easily generalizable to different domains and setups.

**Evaluation**. The evaluation is very comprehensive. It covers a wide range of tasks, models (open and closed source), and domains.
This paper introduces DataGen, a comprehensive framework for generating high-quality (diverse, accurate, and controllable) datasets using large language models (LLMs). DataGen accepts diverse dataset and constraints as input, a comprehensive set of generation hints to reduce computational cost, augment diversity with hyperparameter setting / attribute guided generation, and increase evaluation quality using various reasoning techniques (self-refine) / strong code-based verifier / RAG. Evaluation shows DataGen is able to generalize to diverse set of domains and tasks, models.
**Data formatting.** In section 3.5 (error analysis), the paper mention sometimes the dataset strggles to follow instruction / format the data correctly. Using constrained decoding and similar techniques, this is a very much solved problem, but produce result that the LLM itself may not follow (hence potentially dropping quality of response). I recommend checking out related works in this field (e.g. Guidance[1], AICI[2], LMQL[3], etc.) to improve the data formatting issue. Further more, LLM engiens such as vLLM[4], SGLang[5] and other proprietary engines (Anthropic, Gemini) have provide structured output generation to support constrained decoding at the time of data generation.  

[1] https://github.com/guidance-ai/guidance
[2] https://github.com/microsoft/aici
[3] https://lmql.ai/
[4] https://vllm.ai/
[5] https://github.com/sgl-project/sglang


**Section 3.3 Effectiveness of Modules in DataGen**. 
- Can you explain more on remote-clique score? Why is it a good metric, how exactly is it calculated on the generated dataset (using embeddings, or other representation of the dataset)?
- The delta of remote-clique score of HellaSwag is significantly higher than other datasets. Why is that?
Yes
No ethics review needed.
My questions are included in the weakness section.
1.  DataGen introduces novel elements like attribute-guided generation and the RAG-based validation, which distinguish it from existing synthetic dataset generation frameworks.
2. The modular design allows for customization and adaptability across diverse datasets.
3. The experiments with improved reasoning and agent-oriented tasks performance shows potential in this data generation framework.
The paper introduces DateGen that uses LLM to generate synthetic dataset. DateGen overcomes limitations in generalization, controllability, diversity, and truthfulness, DATAGEN supports a variety of dataset formats and includes mechanisms like attribute-guided generation and group-checking to enhance diversity. It also employs mathematical code-based assessment and Retrieval-Augmented Generation (RAG) for accuracy and truthfulness. Experimentation confirms superior data quality, with applications in benchmarking LLMs and data augmentation, leading to improved model performance in domains like reasoning and agent capabilities
1. RAG-based validation is very high in cost (raising cost from 0.038 to 0.19, almost 5x increase). However, it is unclear how it affects the final data generation quality (like the results in Table 7). In other words, it would be nicer to ablate the modules in terms of metrics in Table 7, instead of the current reports in Table 4.
2. I am not convinced that the performance decline on GSM8K in your experiments can be concluded to that many LLMs may be overstated and overfit on the GSM8K dataset, would you please elaborate more on this?
3. Have you tried any experiments with open-sourced LLMs such as LLaMA 405B being the generating LLM? Would it be as beneficial as the GPT-4 or Claude?
Yes
No ethics review needed.
please refer to weaknesses.
- By integrating different modules, DataGen ensures the quality of generated datasets, considering the diversity, truthfulness, controllability, and so on.
- To assess the quality of generated datasets, the authors design curated experiments and evaluate key factors including length distribution, semantics, diversity, and knowledge richness.
- Experiments on DataGen's effectiveness for data augmentation demonstrate significant benefits across various tasks, particularly in instruction-following scenarios.
This paper proposes a new framework to synthesize high-quality datasets across various types. To ensure dataset quality, the framework integrates an attribute-guided generation module and a group-checking feature to enhance diversity and controllability. It also includes a code-based mathematical assessment and a retrieval-augmented generation module to improve truthfulness and factuality. Experimental results demonstrate the superior quality of the generated datasets in terms of semantics, diversity, and length distribution. By applying the framework to two scenarios, benchmarking LLMs and data augmentation, it validates the effectiveness of this framework.
- In Table 1, the authors list a series of current dataset generation frameworks, highlighting that DataGen considers a broader range of factors. However, for downstream applications, especially data augmentation section, none of these methods are compared, which limits the demonstration of their effectiveness.
- As shown in Figure 4(a), while the length distribution of the generated data tends toward a normal distribution, longer-length samples are missing for HellaSwag and MMLU. In Figure 5, although the generated examples align with the original datasets, it is evident that the generated dataset represents only a partial subset of the originals.
- Table 7 shows that, without difficulty enhancements, LLMs perform better on generated benchmarks compared to the originals, which reduces DataGen's effectiveness and practical value.
- The proposed framework is complex, and in the ablation study presented in Table 4, the analysis may be too simplified to fully validate the effectiveness of each module.
@inproceedings{
anonymous2024datagen,
title={DataGen: Unified Synthetic Dataset Generation via Large Language Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=F5R0lG74Tu},
note={under review}
}
Large Language Models (LLMs) such as GPT-4 and Llama3 have significantly impacted various fields by enabling high-quality synthetic data generation and reducing dependence on expensive human-generated datasets. 
Despite this, challenges remain in the areas of generalization, controllability, diversity, and truthfulness within the existing generative frameworks. To address these challenges, this paper presents DataGen, a comprehensive LLM-powered framework designed to produce diverse, accurate, and highly controllable datasets. DataGen is adaptable, supporting all types of text datasets and enhancing the generative process through innovative mechanisms. To augment data diversity, DataGen incorporates an attribute-guided generation module and a group checking feature. For accuracy, it employs a code-based mathematical assessment for label verification alongside a retrieval-augmented generation technique for factual validation. The framework also allows for user-specified constraints, enabling customization of the data generation process to suit particular requirements. Extensive experiments demonstrate the superior quality of data generated by DataGen, and each module within DataGen plays a critical role in this enhancement. Additionally, DataGen is applied in two practical scenarios: benchmarking LLMs and data augmentation. The results indicate that DataGen effectively supports dynamic and evolving benchmarking and that data augmentation improves LLM capabilities in various domains, including agent-oriented abilities and reasoning skills.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
large language model evaluation synthetic data
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/452a957a646e20230e6cd90fee47cd73928c5125.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/3952b7d1f10b54998a87b5c2c11c0e6685774694.zip
DataGen: Unified Synthetic Dataset Generation via Large Language Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission