Maximizing the Potential of Synthetic Data: Insights from Random Matrix Theory
Synthetic Data;RLHF;Generative Models;Statistical Models;Random Matrices
Yes
No ethics review needed.
Though I think the paper is potentially a very good paper, I still have a few worries as outlined in the **Weaknesses** section of this review and also the questions listed below (I'm open to changing my mind if addressed).

- Clarify relevance of the consider model (see **Wicknesses** section above)
- Clarify connection to literature, especially Jain et al. (2024) "Scaling laws for learning with real and surrogate data" and Dohmatob et al. (2024) "Strong Model Collapse" (see **Wicknesses** section above)
- Can you instantiate your results on a concrete setup and show the corresponding scaling laws (perhaps with some approximations of Gaussian CDF) ?
- A nice theoretical setup for analysing the impact of synthetic data and pruning strategies is introduced.
- A complete analytic theory for this toy setting is obtained and its phenomological clearly discussed.
- Theoretical results from previous work Feng et al. (2024) are recovered as special cases.
- The calculations provided in the appendix could be of independent usefulness, beyond the specific problem considered in the paper. I should however immediately nuance this praise by noting that similar calculations have been done in  Liao and Couillet (2019) "A Large Dimensional Analysis of Least Squares Support Vector Machines", for more general kernel-based models.
- The paper is very clearly written.
This work considers the problem of maximizing the usefulness of synthetic data in the modern era of generative modelling whereby at least part of the training data is not from the real data distribution, but synthesized from another AI model. In the solvable setting of a mixture of isotropic Gaussians, the authors consider a linear model whose weights vector is a regression-style ridge estimator. The synthetic data distribution is taken as yet another Gaussian mixture whose parameters (i.e shape) is given by empirical estimates of the ground-truth Gaussian mixture: the amount of data on which these parameters are estimated serves as a proxy for the quality of the synthetic data. Optiinally, this synthetic data is mixed with data from the real data distribution.

In the "proportionate" high-dimensional scaling limit, the authors use classical tools from random matrix theory (RMT) to obtain exact analytic expressions for the classifier and its accuracy. In particular, they paper recovers the theoretical results of Feng. et al (2024) as special case (corresponding to training only on synthetic data).
- **The setup.** Fitting a regression model to use its weights as a linear classifier, nobody does this in practice. The authors should justify why such a model is reasonable / relevant, beyond the fact that it leads to a tractable analysis. Note that the underlying estimator is a linear version of the so-called least-squares support vector machine (LS-SVM) which was analyzed in Liao and Couillet (2019) "A Large Dimensional Analysis of Least Squares Support Vector Machines".
- **(Ir)relevance to practice.** In the asymptotic regime considered in this paper, mixing real and synthetic data only helps when the proportion of real data is bounded-away from zero. Also, the practitional needs to know which samples are real and which samples are synthetic. These constraints might not match what happens in practice. This observations have also been made in Dohmatob et al. (2024) "Strong Model Collapse".
- **Non-optimal mixing.** The weights in mixing strategy considered is a bit naive / sub-optimal. The right thing to do would be to consider a general mixing constant $\alpha \in (0,1)$ (the papers uses $\alpha=1-\alpha=1/2$), and derive a theory as a function of $\alpha$, alongside the other  constants in the theory, namely $\pi$, etc.. In general the optimal value of $\alpha$ will depend on the other constants. See Jain et al. (2024) "Scaling laws for learning with real and surrogate data" and Dohmatob et al. (2024) "Strong Model Collapse"
Yes
No ethics review needed.
1.Figure 1: $\hat{\eta}$ = 0.1, $p$ = 500, and $\hat{n}$ = 500, given that $\hat{\eta}$ = $\frac{p}{\hat{n}}$, is this a typo?              
2. Figure 3, in the figure of weak supervision, when the proportion of synthetic data is higher than 0.8, the performance increases, what is the reason for this? This trend is opposite to that when the proportion of synthetic data is lower than 0.8.  Similar observations also happen in Figure 6.      
3. Can authors explain why $\hat{\eta}$ can represent a distribution shift?
1. This work presents analysis and insights with theoretical justification. 
2. Through multiple tasks, the approach is shown its empirical effectiveness.
Synthetic data has been used in training large language models, but concerns about its quality have emerged in practical applications. This paper works on data pruning to select high-quality data and analyzing the performance of trained on a mixture of real and synthetic data in a high-dimensional setting. The authors provide both theoretical and experimental results for the benefits of using and verifying synthetic data.
1. The paper lacks a clear definition of the quality of synthetic data. 
2. The derived theory in the paper is strongly assumed based on Gaussian distribution, however, non-Gaussian data are common in text or image data used for LLMs.
3. The paper only focuses on label verification for synthetic data. In real-world applications, synthetic data generation often struggles with feature fidelity, addressing feature consistency or the quality of synthetic data in feature space is more beneficial.
Yes
No ethics review needed.
1) How does your work compare to Jain et al. [1] specifically? (It’s a different data model, of course, but qualitatively, are there similarities/differences and how do the methods differ?)
2) I am confused about the "No supervision" case in Fig. 6 (the Amazon data). While in all other experiments more synthetic data doesn't help, here the U-shape of the curve stands out. Can you discuss a little more why in this case the red curve starts going up?
The paper lays out a coherent program and goes ahead to prove it with relatively sophisticated tools. The message is conveyed very clearly, the extensions over prior work, and in particular Feng et al 24 are clearly demarcated. 
The consequences of theorems are explained well, despite the necessarily heavy math. This paper thus provides further evidence to what was laid out in Feng et al 2024: synthetic data can only boost performance when coupled with verification (pruning).
The paper does a masterful job of presenting the results and the experiments are a nice addition. The paper is also well written and is fun to read.
This paper extends important work by Feng et al. 24 and studies the effect of synthetesized data as part of the training data of models. It has two parts, a theoretical part that significantly extends the analysis of a mixture of two isotropic Gaussians from Feng et al. 24 from the case of infinite data at finite dimension to the case where both dimension and data tend to infinity with fixed ratio. Using tools from RMT they thus uncover a richer structure of transitions in a parameter space defined by the quality of synthetic data, the generator, label noise and verifier strength. A key novel element (which facilitates the analysis) is the assumption that the synthetic data comes from a model that estimates the mean and covariance matrix from initial (real) data (as opposed to a more general estimator). This allows to nicely study distribution shift of the features (not just the labels). Another extension over Feng et al.24 is the analysis of mixtures of real and synthetic data (though this is also done in "Strong Model Collapse" by Dohmatob et al., in a different and more general setting). 4 experiments nicely support the theory.
The main weakness is that a few crucial references are missing. In particular there are three papers mixing real and synthetic data that should be cited, [1] and [2], and [3], a more recent one (there is no fault in not citing the latter, as it is recent, 
but the former date back over 8 months). In particular, a careful brief comparison to the [1] setting and results should be added, and [2] should be mentioned. Another paper that should be cited among the theoretical works for model collapse is [4]. On the other hand, 
your reference to [5] should probably be completely removed as it is misleading (the analysis of combination of real and synthetic data doesn't compare to adding the same amount of real data properly - this has been debunked on several occasions and should not be propagated).

Specifics (mostly minor):
1. line 252: typo "usefull"
2. line 402: "emphasize on" - sounds wrong, reformulate
3. line 406 use \citep for Malartic
4. line 408: "from the the Antropic's" --> "from Antropic's"
5. line 483-484: discuss Llama versus Gemma to make clear that you regard one as a weaker supervisor than the other
6. line 518: "Conclusionn" - typo
7. line 527-28: you might want to discuss that label pruning can induce a distribution shift in the features (because the pruning could be biased towards keeping certain features alive) - see (and cite) the discussion in Feng et al. 2024 (https://arxiv.org/pdf/2406.07515 version bottom of p. 10 - top p 11)

I will be happy to reconsider my score once these have been addressed (and the questions).

[1] Ayush Jain, Andrea Montanari, Eren Sasoglu: Scaling laws for learning with real and surrogate data, https://arxiv.org/abs/2402.04376

[2] Bertrand, Q., Bose, A. J., Duplessis, A., Jiralerspong, M., and Gidel, G. On the stability of iterative retraining of generative models on their own data, ICLR 2024 spotlight

[3] Elvis Dohmatob, Yunzhen Feng, Arjun Subramonian, Julia Kempe: Strong Model Collapse, https://arxiv.org/abs/2410.04840

[4] Elvis Dohmatob, Yunzhen Feng, Pu Yang, Francois Charton, Julia Kempe: A Tale of Tails: Model Collapse as a Change of Scaling Laws, ICML 2024

[5] Gerstgrasser et al. 2024
Yes
No ethics review needed.
1. In Equation (2), the generative model assumes a prior with symmetric data sharing the same $\hat{\mu}$. How would the theory change if one Gaussian per class was fit instead?
2. On line 129, label noise is introduced as purely random noise. What if the label error depended on the input? For instance, in safety applications, a generative model might introduce label errors in specific regions of the distribution due to bias. Could such input-dependent noise be analyzed?
3. It would be helpful to discuss how label noise and feature noise correspond to various types of errors and shifts in real settings.

Missing details from the experiment:

4. What value of $\hat{n}$ is used across all experiments? The authors mention $\hat{n}$ varies in the MNIST experiment (line 400) but don’t provide further details.
5. In the LLM Safety experiment (line 411), it is stated, "we focus only on label noise." However, since data is generated by another language model, feature noise is likely present. 
6. In Figure 7 (left), both feature noise and label noise should be included, as data are generated by a Gaussian fitted to the distribution. Could the authors provide error bars here? Given the simplicity of the MNIST setup with two-layer networks, this should be feasible.
7. In Figure 8, why does using $(\rho, \phi) = (0.5, 0.5)$ as weak supervision? According to the definition in Equation (5), the data selection remains random regardless of label accuracy, which seems equivalent to no supervision.
8. In the LLM Q&A Safety Generation experiment, is accuracy evaluated by the Llama-Guard-3.1 model? What weak supervision method is employed here? Why do the authors use two models to annotate labels? It would be helpful to include the evaluation results comparing the two labeling models with the “ground truth” from Llama-Guard-3.1 to assess the quality of synthetic labels. Additionally, how is supervised fine-tuning conducted with data containing both safe and unsafe answers?
1. The theoretical section is well-written, offering strong motivations and clear explanations.
2. Analyzing distribution shifts in the feature space is a valuable contribution compared with the previous work.
3. The paper provides a precise analysis of performance when training on mixed real and (selected) synthetic data with random matrix theory, expanding beyond previous work.
This paper builds upon Feng et al. that examined how using synthetic data with selection impacts final performance. Here, the authors extend that research to a setting that considers both distribution shifts in the feature space and the combination of real and synthetic data. This approach provides precise results without requiring an infinite number of synthetic samples. The experiments corroborate some theoretical insights.
My primary concern is the applicability of the theoretical insights in practical settings. The theoretical predictions are based on binary Gaussian mixture data, with synthetic data generated by fitting another Gaussian mixture. While the authors provide performance predictions for models trained on mixed data, it is unclear how well these predictions, derived from Gaussian mixture models, will generalize to real-world data. Specifically, how can we estimate some of the key theoretical variables? How can we ensure that the supervision quality is sufficient to mitigate the negative effects of distribution shift?

Currently, the experimental section could be improved by drawing more explicit connections to the theoretical insights. It is generally expected that better supervision and a more accurate generator would lead to improved performance with selected data. However, what practical insights does the theory provide? Can the theoretical findings be leveraged to inform the design of good data selection methods?

The experiment part is not well-written, with lots of details missing. I will elaborate more in the questions.
@inproceedings{
anonymous2024maximizing,
title={Maximizing the Potential of Synthetic Data: Insights from Random Matrix Theory},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=I9Dsq0cVo9},
note={under review}
}
Synthetic data has gained attention for training large language models, but poor-quality data can harm performance (see, e.g., Shumailov et al. (2023); Seddik et al. (2024)). A potential solution is data pruning, which retains only high-quality data based on a score function (human or machine feedback). Previous work Feng et al. (2024) analyzed models trained on synthetic data as sample size increases. We extend this by using random matrix theory to derive the performance of a binary classifier trained on a mix of real and pruned synthetic data in a high dimensional setting. Our findings identify conditions where synthetic data could improve performance, focusing on the quality of the generative model and verification strategy. We also show a smooth phase transition in synthetic label noise, contrasting with prior sharp behavior in infinite sample limits. Experiments with toy models and large language models validate our theoretical results.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Synthetic Data RLHF Generative Models Statistical Models Random Matrices
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/711cbe9db2a40855d0ee31fc2272d19413d4c361.pdf
learning theory
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Maximizing the Potential of Synthetic Data: Insights from Random Matrix Theory
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission