Leveraging Knowledge Distillation to Mitigate Model Collapse
computer vision;natural language processing;generative models;diffusion;vae;text summarization;model collapse;synthetic data;distillation
Yes
No ethics review needed.
- (Writing) Please modify sty file for ICLR 2025.
- (Writing) The line style looks unorganized
- The proposed approach is highly intuitive. (but, too obvious.)
The authors use a knowledge distillation framework that utilizes a model trained on real data as a teacher to address the issue of model collapse in models trained on synthetic data.
- Lack of novelty
  - They merely used the conventional knowledge distillation (KD) method. I think they should have compared various KD methods to identify a more suitable approach for resolving model collapse on the same task.
- Lack of references
- Design of experiments
  - Their experimentation is limited to adjusting the hyperparameters of the model.
  - For iterative cases (repeated cycles of data generation and model retraining, as you mentioned in Introduction), I think it is necessary to assess how severe the model collapse becomes and to what extent it can be resolved.
Yes
No ethics review needed.
Is there no existing baseline for comparison?
-	This paper uses knowledge distillation as a solution to address model collapse.
-	This paper conducts experiments on image generation using VAE and DDPM, as well as text summarization using the T5 model.
This paper presents a method that utilizes knowledge distillation to mitigate the adverse effects of synthetic data by enhancing the transfer of knowledge from high-performing teacher models to student models. Through extensive experiments, they improve the robustness and generalization capabilities of models trained on synthetic data.
-	The template of the article is not officially provided.
-	The models used in this paper are VAE and DDPM. Can more advanced models be used for image generation, and can the resolution of the generated images be improved? This can better prove the generalization of the proposed method.
-	Lack of comparison with existing approaches to mitigate model collapse.
Yes
No ethics review needed.
NA
1. Applying KD to mitigate model collapse might be a possible solution.
The paper explores the issue of model collapse. The authors propose a solution using knowledge distillation, Experiments across image generation tasks, including Variational Autoencoder (VAE) and Denoising Diffusion Probabilistic Model (DDPM), and text summarization show performance gains.
This paper’s format does not follow the ICLR requirements. Additionally, the presentation is poor, lacking clear motivation and an introduction to the methodology. Many unnecessary figures that should have been placed in the appendix occupy a large portion of the main text, making the paper resemble an experimental report. Even so, it fails to reach ten pages. This submission appears extremely unprofessional.
Yes
N/A
No ethics review needed.
Please refer to the Weakness section.
1. The proposed method is easy to follow.
2. The structure of the paper is clear.
This paper provides a contribution to addressing the problem of model collapse using synthetic data. The proposed method leverages knowledge distillation to address this problem. Experiments on multiple image genearion models and text generation model are conducted to indicate the effectiveness of the proposed method.
1. There is no formal definition about "Model Collapse" indicated in the paper, the author should describe it for both text model and image model in more details. Also I do not agree that the test set loss and ROUGE scores are a good metric for model collapse indication.
2. The adopted datasets for image generation are quite simple. The authors should use more complex datasets.
3. There is no theoretical/empirical analysis about the results and findings, the authors should think about adding these.
4. The proposed method is still worse than $M_0$ after training with longer steps for language models. The authors should analyze this more.
5. The authors have modified the template style, which could be problematic.
Knowledge distillation between model, trained on real data, and on synthetic data, improves the performance of the last one.
@inproceedings{
anonymous2024leveraging,
title={Leveraging Knowledge Distillation to Mitigate Model Collapse},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=8TbqoP3Rjg},
note={under review}
}
Since the amount of data generated by neural networks on the Internet is growing rapidly due to widespread access to corresponding models, it is logical to inquire about the impact of this surge in synthetic data on the training of subsequent models that will utilize it during training.  Previous work has demonstrated a concerning trend: models trained predominantly on synthetic data often experience a decline in performance, which can escalate to a complete loss of the ability to reproduce the initial distribution of real-world data. This phenomenon, now referred to as model collapse, highlights the potential pitfalls of over-reliance on synthetic datasets, which may lack the diversity and complexity inherent in genuine data. To address this issue, we propose a novel method that leverages the well-established technique of knowledge distillation. Our approach aims to mitigate the adverse effects of synthetic data by facilitating a more effective transfer of knowledge from high-performing teacher models to student model. By doing so, we seek to enhance not only the qualitative aspects—such as the richness and variability of the generated outputs—but also the quantitative metrics that gauge model performance. Through extensive experimentation, we demonstrate that our method improves the robustness and generalization capabilities of models trained on synthetic data, for instance, for DDPM enhancement is 68.8%, in terms of the FID metric, contributing to a more sustainable and effective use of synthetic datasets in machine learning applications.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
computer vision natural language processing generative models diffusion vae text summarization model collapse synthetic data distillation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/48f34f8d3ce3a30f2e6a80fac0b8d1079804eed5.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Leveraging Knowledge Distillation to Mitigate Model Collapse
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission