Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification
Learning with Synthetic Data;Data Curation;Avoiding Model Collapse
Yes
No ethics review needed.
- I am interested in whether the proposed methods can be applied to other modalities (e.g., VLMs, Vision-Language-Action Models in robotics) - many problems face more challenges in data annotation cost and also resort to synthetic data.
- Many LLMs are trained with synthetic data specifically on preferences (e.g., RLHF, DPO). I wonder if parallels can be drawn between the verifiers and a reward model and if model collapse is particularly severe with Vicuna-style finetuning.
- Paper is very well written and easy to understand
- Both synthetic data and model collapse are very important topics in the literature and are of great interest to the audience
- Theoretical derivation is complete and appears to be correct
- Very thorough quantitative experiments with strong results
The work investigates synthetic data in LLM training and concerns about model collapse (catastrophic drop in model performance) when LLMs are trained with model-generated labels. Specifically, the work investigates model collapse as a decision problem and utilizes a framework where a verifier (pruner in the manuscript) vets synthetic examples in an attempt to limit the risk of model collapse. Experiments have demonstrated that this verification-based approach can prevent collapse even when the verifiers are imperfect and are not oracles.
- In terms of LLMs, qualitative examples showing model collapse in some of the tasks may benefit the delivery further
Yes
No ethics review needed.
There are a few minor writing inconsistencies in the introduction (and more generally, I feel that the writing of the intro could be improved): 
- "We validate our theoretical findings in three empirical settings" and "We conduct two large-scale experiments to test our theoretical insights :" are somewhat at odds. And if they aren't, then the writing for this part should be clarified. 
- Ranking is only mentioned once, at the very end of the intro. It is unclear how ranking fits into the verification process -- data selection does not necessitate ranking, although ranking is indeed a way to perform data selection. What is meant by this? 
- Is verification really all we need? Some form of "verification is all you need" is mentioned throughout the paper, but it seems like there is a long list of things that we need. 
- While I imagine that this would be computationally expensive, it would be useful for the authors to include a scaling law analysis under synthetic data, and for that matter, an example of pretraining on synthetic general data. The proposed technique indeed improves over the collapsed baseline, but it does not necessarily approach the performance of training on real data. There are many questions related to scale that I don't think this paper addresses, but I am not going to factor this into my evaluation, as this seems like obtaining an answer to this would be quite expensive. In any case, I would be interested to hear any speculative thoughts that the authors might have about this.
This paper covers an important topic, and establishes some of the fundamentals for effectively training on synthetic data -- this is likely going to be an important area of study as we continue to scale foundation models. 
- The authors perform a theoretical analysis that characterizes how high-quality the synthetic data needs to be in order for it to be a useful stand-in for real data, i.e., in order to avoid model collapse. 
- The authors present practical results for fine-tuning Llama-2 on a news summarization task as well as a pretraining task in which the transformer is trained to predict matrix eigenvalues. For the news summarization task, the authors find that Llama-3 is a worse verifier than Llama-2, which is a peculiar finding. 
- For the real data experiments, the authors confirm that the model collapse phenomena actually occurs, and that their verification procedure actually improves over the model collapse baseline.
This paper studies a phenomenon known as "model collapse," in which LLMs that are trained on synthetic (LLM-generated) data suffer drops in performance compared to LLMs trained on real data. The authors propose a verification technique for selecting synthetic data, and analyze cases in which this process can improve over collapsed LLMs trained on randomly selected synthetic data. The theoretical analysis is done using a simplified setting of a Gaussian mixture and linear classifiers, and the results are verified empirically in the same setting. Finally, the authors present empirical results using a pretraining setting for eigenvalue prediction and news summarization.
- The theoretical analysis could be improved with some form of a finite-sample result. The current result only applies in the infinite sample limit in a simplified setting, although the result is validated empirically (using finitely-many samples, of course). 
- The writing in Section 3 can be improved substantially. The purpose of this section and its contribution to the overall narrative are currently unclear, which can be mostly fixed by leading with a crisp motivational statement. More generally, I'm still not sure that I understand the takeaways of this section -- why is it that the model "lacks the inherent capability to autonomously select the best predictions," when using beam search? Is this a statement about decoding, or about the model? Earlier, in the intro, the authors say that models "cannot intrinsically identify the best solution using perplexity alone," yet beam search *is* essentially based on maximizing the likelihood of outputs. 
- 4.1 in the theoretical analysis says that the $y'$ variables "[have] been generated by an AI model," but it's unclear what this statement means mathematically. This should be more formal, particularly since it's part of the theoretical analysis. 
- Assumption 4.1 introduces "bits" $q_1, ..., q_N$. I assume these represent whether or not a sample has been pruned, however this is not defined until after it's already mentioned, and it is up to the reader to infer what these bits represent. 
- The authors compare to "selection with weak supervision," but there is little elaboration about what this actually means. Weak supervision is an overloaded term and refers to many things -- one reading of this could be that the authors refer to programmatic weak supervision, which could make sense because programmatic weak supervision in some sense generates synthetic training data and verifies it via abstain votes.
Yes
No ethics review needed.
Figure 5 shows when more selected synthetic data are trained on, both the performance of Oracle and $p_{*}$ become worse, why does this happen? Does it mean synthetic data can still potentially make the model collapse?
1. The paper provides a theoretical framework for understanding the role of verification in preventing model collapse.
2. The paper is well-written and easy to follow. 
3. The method has been demonstrated effective via empirical experiments in different domains.
The paper presents a verification method to prevent model collapse that happens when models are trained on synthetic data. The proposed proxy $p_{*}$ offers a practical way to estimate the potential performance of models trained on verified synthesized data. Based on it, the synthetic data can be selected to avoid degrading the performance of the model that is trained on mixed real and synthetic data. Tested across three different tasks, the method shows its effectiveness and practical applicability.
1. Model collapse not only happens when trained on synthetic data but also occurs when synthetic data is involved in the training set iteratively. This paper fails to address model collapse from this dimension. 
2. As stated in the paper, Llama 3 performs worse than Llama 2 when as a verifier. Therefore, an advanced model is not necessary to be a good verifier. There could be more discussion on the reason behind it and how to choose verification models. 
3. This paper primarily focuses on accuracy for analysis, yet it would be insightful to consider cases where accuracy alone is insufficient for assessing model performance, such as alignment in large language models (LLMs), and explore how the synthetic data selection strategy could be applied in such complex scenarios.
Yes
No ethics review needed.
- How might this generalize to language modeling / task-agnostic pretraining, where there is not necessarily a sense of "correctness" for verification?

- Lines 390-395 point out that when using a weak verifier, there's a dependence on $n'$, the amount of synthetic data to train on. This seems practically quite important to establish. Can the authors comment more on how to "take into account the quantity of data selected" (line 395)?

Misc suggestions for a final version:

- I would suggest moving Appendix C into the main text. If necessary for space, I would move Section 3 to the appendix, since these results are reasonably well-known from the test-time compute literature.

- https://arxiv.org/pdf/2209.03942 is a related reference

- Line 365 typo: "and the ground truth $w_*$. 5Having the verified"
- Line 12 typo: "other LLM[s]"
This paper was well-written, flowing through a toy but well-executed theory section (Section 4), and then a set of three nice experiments (Sections 5-6). The grounding in actual experiments on transformers was great; in particular, the news summarization experiment for tasks without exact match notions of correctness improved the generality of this work. I appreciated that the appendices were very thorough, and I was able to find answers to most of my questions by looking through these. 

I'm not confident in my familiarity with related work, but I believe that the setting studied here (and the style of analysis, beginning with a toy model and ending with empirical work) is novel and helpful to the synthetic data community.
The authors study whether filtering synthetic data (collected from a generator) using verifiers can prevent model collapse. In a binary classification setting, they relate downstream model performance to two quantities: the generator's error $p$ and a quantity related to the verifier's accuracy $p_*$. When the generator's error surpasses the threshold related to the verifier's accuracy ($p > p_*$), model collapse occurs. They then conduct three empirical experiments (binary classification w/ linear models, eigenvalue prediction w/ transformers, and news summarization with Llama-2) in which they show that downstream performance correlates with the verifier performance $p_*$, which they can estimate using some labeled training data.
*Novelty of results.* From a practitioner's point of view, the conclusion from these results is simple: when it is possible to evaluate a verifier on labeled data, picking a more accurate verifier gives better results on that same task. This seems to be obvious / empirically quite clear in existing work (e.g. Appendix A.1), and I'm not sure if the specific definition of $p_*$ offers gains over just picking by existing accuracy metrics.
We theoretically and empirically demonstrate that leveraging verifications to select synthesized data can prevent model collapse.
@inproceedings{
anonymous2024beyond,
title={Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=MQXrTMonT1},
note={under review}
}
Large Language Models (LLM) are increasingly trained on data generated by other LLM, either because generated text and images become part of the pre-training corpus, or because synthetized data is used as a replacement for expensive human-annotation. This raises concerns about *model collapse*, a drop in model performance when their training sets include generated data. Considering that it is easier for both humans and machines to tell between good and bad examples than to generate high-quality samples, we investigate the use of verification on synthesized data to prevent model collapse. We provide a theoretical characterization using Gaussian mixtures, linear classifiers, and linear verifiers to  derive conditions with measurable proxies to assess whether the verifier can effectively select synthesized data that leads to optimal performance. We experiment with two practical tasks -- computing matrix eigenvalues with transformers and news summarization with LLMs -- which both exhibit model collapse when trained on generated data, and show that verifiers, even imperfect ones, can indeed be harnessed to prevent model collapse and that our proposed proxy measure strongly correlates with performance.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Learning with Synthetic Data Data Curation Avoiding Model Collapse
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/2596c64ad77aa5b199722900cb67da73cdec45f0.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission