Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws
Data selection;Pre-training;Curriculum learning;Language Models;Scaling laws
Yes
No ethics review needed.
1. Figures 6, 7, and 8 show the training loss for ADO only. I'd like to see how quickly ADO's training loss decreases compared to the baselines. I assume that ADO converges faster, but it would be interesting to observe how it performs relative to the baselines.

2. Can ADO be effectively applied to vision data and tasks? It would be of great importance if ADO could be used in training vision models that require multiple domains, such as large vision-language models (LVLMs) and diffusion models for image generation.
1. The motivations presented in Section 2 highlight significant concerns, and the proposed method properly addresses these issues.
2. ADO outperforms the baselines overall, offering online data selection without the requirement of external knowledge or proxy models.
3. The authors present several interesting observations that align with our intuition and the literature (e.g. line 428, 409, 431).
The authors propose ADO, an online data selection method that adjusts data distribution dynamically across various domains by leveraging domain scaling laws. Without requiring a proxy model or external knowledge, ADO forecasts the model's loss on different data domains and automatically modifies the training data distribution according to each domain's learning potential. The experiments indicate that ADO surpasses baseline methods overall, with only a minimal increase in clock time.
1. Although the 1.3B model is not small, ADO is especially practical and relevant for LLMs with at least 8B parameters—the size typical of many popular "small" LLMs such as Llama 3.1 8B. The paper's relevance and importance might have been enhanced if the authors had conducted experiments with language models of at least 8B parameters
2. The performance improvement achieved by ADO does not seem significant enough, especially on the 1.3B model. More problematically, Table 1 shows diminishing returns as the model scale increases from 124M to 1.3B parameters. For example, 1.3B-ADO only outperforms 4 out of 7 downstream tasks, whereas 124M-ADO outperforms 6 out of 7 tasks. Additionally, the gap between the average score of ADO and the second-best baseline is less pronounced in the 1.3B model. This suggests that ADO may not scale well as the number of parameters increases.
3. Although the authors propose interesting future research directions (e.g. Section 6), incorporating some of these suggestions—such as learning rate scheduling—into the current paper would make it more complete and thorough.
Yes
No ethics review needed.
- The smoothing parameter $s$ for computing the credit assignment scores seems a bit confusing. By using $s<1$, I suppose $\lambda_k$ indeed changes drastically when $h_k$ is close to 0? Should using $s<1$ encourages $\lambda_k$ to be larger than $h_k$ especially when $h_k$ is originally close to 0? Some further discussion on the effect of this hyper-parameter may be necessary. 
- Also, can we consider other types of functions to compute $\lambda_k$ from $h_k$? Some empirical comparison on different types of functions may help others better understand how the credit assignment score affects model training, and why the authors reject simply using $\lambda_k=h_k$ here. 
- Experiments are limited to medium scale models (124M and 1.3B), and the performance gain seems to decrease with increasing model size. Such a tendency can naturally make one wonder if the performance gain can be smaller or even diminish on larger models (e.g., 2.3B or 7/8B)? Some more empirical results may be necessary to better support the proposed method. 
- While the proposed method includes many hyper-parameters (e.g., the smoothing parameter $s$ mentioned above), the authors did not provide sufficient analysis on how these hyper-parameters can affect the final performance. These results should be necessary to demonstrate that ADO can be robust to different values of these hyper-parameters. 
- Despite showing the final sampling probabilities in Figure 5, the authors may also consider further showing how $\lambda_k(t)$ and $\frac{\partial}{\partial n} \hat{L}_k(n)$ changes with training iteration $t$. These results may help others better understand how these two terms affect the sampling probabilities of different domains and how they can affect model training.
The proposed method is introduced clearly and easy to understand
This paper proposes a novel method ADO to adaptively sample data from different domains based on their contribution to model training, which is estimated from scaling law in an online manner. Empirical results on different data sets demonstrate the effectiveness of the proposed method.
- Several parts of ADO lack sufficient motivation
- Empirical results seem not supportive enough to fully evaluate the proposed method
Yes
No ethics review needed.
1.	In ADO's data selection process, some datasets might receive minimal training due to low selection probabilities, potentially leading to under-representation. Have any additional measures been implemented to balance data distribution and ensure adequate representation across datasets?
2.	How sensitive is ADO to changes in its hyperparameters, such as the smoothing coefficient (γ1) and update intervals? Would different tasks or datasets require significant parameter tuning?
3.	Given the current limitation in modeling solely intra-domain interactions, have the authors considered potential methodologies to incorporate inter-domain interactions within the ADO framework? What approaches might be feasible for capturing these cross-domain dynamics to enhance the model’s adaptability and performance in tasks where inter-domain relationships play a significant role?
4.	ADO is designed to be task-agnostic, but certain applications may benefit from targeted data selection. Could ADO be extended to incorporate task-specific objectives without compromising its efficiency?
1.	ADO utilizes online scaling laws to dynamically adapt data selection during training, replacing the need for pre-trained proxy models or multi-staged processes. It requires no prior domain knowledge or external models, making it highly practical and versatile.
2.	The paper presents empirical evaluations demonstrating ADO's effectiveness across multiple benchmarks and datasets. ADO outperforms or matches existing baselines on several downstream tasks while incurring a small computational overhead (e.g., less than 0.4% additional wall-clock time).
3.	ADO offers a practical, scalable solution for data selection in large model training, with the potential to significantly reduce computational waste. The method aligns with the growing emphasis on efficient resource utilization in AI research, especially as models scale in size and cost.
The paper introduces ADO, a novel online algorithm for optimizing data distributions during the training of large-scale foundation models. ADO addresses the challenge of efficiently managing data mixtures in pretraining by continuously adapting the data distribution based on scaling laws, without requiring proxy models or significant computational overhead. The primary contribution lies in the use of per-domain scaling laws to estimate learning potential and dynamically adjust the data mixture during model training. Experimental results demonstrate that ADO can achieve comparable or better performance than prior methods, while maintaining computational efficiency across various model scales.
1.	Figure 4 reveals that fewer than half of the datasets achieve minimal perplexity under the ADO algorithm, indicating potential limitations in the algorithm’s generalization capability across diverse scenarios. The paper would benefit from a more detailed analysis of ADO's applicability.
2.	The algorithm employs heuristic choices for key parameters, such as the exponential moving average in credit assignment and smoothing coefficients, which may impact the robustness and consistency of ADO across different datasets and tasks due to the lack of a more systematic basis for parameter selection.
3.	Symbols such as $\gamma_2$ appears in key formulas but might be clarified further with more detailed context, especially for readers unfamiliar with its role in the algorithm.
4.	Although ADO highlights low computational overhead as a strength, the paper lacks a systematic computational complexity analysis and comparative experiments. The absence of quantitative and visual evidence of its efficiency limits the clarity of its computational advantages.
Yes
No ethics review needed.
- About Figure 2, I think the point on the variance is clear. But what’s more important is the agreement of the relative order of data strategies: if a data strategy is better than another on a smaller model, is it also better on a larger model? Did the authors study that? Also, what are the blue dots? I guess they are the actual validation loss of different model sizes, but would be nice to make it clear in the caption.
- The authors also write “they find $\gamma_1 = 0.1$, $s = 0.5$, $\gamma_2 = 0.1$” works well. How exactly is the process of finding those values?
- The authors write in line 480: “they are accurate locally for much of training and thus can act as a learning signal for the data policy”. On what basis do the claims are made?
- Novelty. It’s a novel idea to improve over online data optimization (ODM, Albalak et al. (2023)) on 3 aspects. 1) Using scaling law to predict training loss on each domain and the derivative of the loss w.r.t. to the number of data points has intuitive interpretation as the loss reduction per data point. 2) The weighting also considers the number of data points used for a domain. 3) A combined weighting with the above two and temporal average.
- Readability. The paper not only shows their method works but also spends a reasonable amount of space discussing perspectives, hence better readability. For example on the curriculum learning (Section 2) and on the requirements (Section 3, online and agnostic to downstream tasks) of data optimization for pretrained models.
- Besides their proposed method, the authors also discover a strong baseline called “natural” policy, that depends on the number of tokens in each domain.
This work proposes an online adaptive data optimization (ADO) method for finding a policy (weighting) for different domains when pre-training a language model. The core idea is to use scaling law to model training loss. The derivative of the loss w.r.t. to the number of data points can be understood as loss reduction per data point locally, i.e. outline the potential for a domain. The final scoring also considers the contribution of the domain itself from the last update and makes it stable with a moving average. Their experiments show the ADO brings improved downstream performances on 7 common-sense reasoning tasks.
- Many heuristics are used such as fitting of scaling law (Section 3.1 eq1), credit assignment (Section 3.2, eq2 ), preference distribution (Section 3.3, eq3) and temporal average (eq4 and eq5). The authors try to motivate those choices from related works and intuitions, but only eq1 is adequately explained and validated. 
- Following the above, are all the heuristics (eq 1- 5) necessary? How important are they? This is a missing part in the paper. An ablation on them can validate if the invented heuristics are actually all useful.
The paper presents a scalable and proxy-free algorithm that dynamically optimizes data distributions during model training.
@inproceedings{
anonymous2024adaptive,
title={Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=aqok1UX7Z1},
note={under review}
}
The composition of pretraining data is a key determinant of foundation models' performance, but there is no standard guideline for allocating a limited computational budget across different data sources. Most current approaches either rely on extensive experiments with smaller models or dynamic data adjustments that also require proxy models, both of which significantly increase the workflow complexity and computational overhead. In this paper, we introduce Adaptive Data Optimization (ADO), an algorithm that optimizes data distributions in an online fashion, concurrent with model training. Unlike existing techniques, ADO does not require external knowledge, proxy models, or modifications to the model update. Instead, ADO uses per-domain scaling laws to estimate the learning potential of each domain during training and adjusts the data mixture accordingly, making it more scalable and easier to integrate. Experiments demonstrate that ADO can achieve comparable or better performance than prior methods while maintaining computational efficiency across different computation scales, offering a practical solution for dynamically adjusting data distribution without sacrificing flexibility or increasing costs. Beyond its practical benefits, ADO also provides a new perspective on data collection strategies via scaling laws.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Data selection Pre-training Curriculum learning Language Models Scaling laws
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/4ce822b2af064fe586f838858eb49436e389434b.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/a1e7d3ad7e5dab6b58c8b06e14844ec1eada8e94.zip
Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission