Equally Critical: Samples, Targets, and Their Mappings in Datasets
Data-efficient Learning
Yes
No ethics review needed.
Applicability to Larger Datasets: Have you considered applying STRATEGY C to larger-scale datasets

============ Revise according to the Associate Program Chair's comments ================ $\searrow$

"Have you explored applying STRATEGY C to larger datasets like ImageNet? What computational or methodological challenges do you anticipate in scaling up this approach?"

Teacher Model Selection: How does the choice of teacher model architecture impact the student model's performance under STRATEGY B and STRATEGY C? $\searrow$

"Have you considered comparing the impact of different teacher model architectures (e.g., ResNet vs. Vision Transformer) on student performance under STRATEGY B and STRATEGY C?
Comprehensive Analysis: The paper provides a thorough investigation of how different sample-to-target mappings and data augmentation strategies affect training efficiency, offering valuable insights.

Novel Perspective on Targets: By highlighting the often-neglected role of targets in dataset design, the paper contributes to a more holistic understanding of data-efficient learning.

Unified Loss Framework: The introduction of a unified loss function that separates the backbone training from the classifier training allows for a clearer evaluation of the representational capacity influenced by different strategies.

Practical Implications: The findings offer actionable guidance on selecting target types, teacher models, and augmentation strategies to enhance training efficiency, which can be beneficial for practitioners.

Extensive Experiments: The use of multiple datasets and varied experimental settings strengthens the validity of the conclusions drawn.
This paper investigates the often-overlooked role of targets in data-efficient learning, particularly in the context of neural scaling laws. Neural scaling laws indicate that achieving lower test errors typically requires exponentially more data and computational resources, leading to inefficiencies and sustainability challenges.

The authors observe that using soft targets on a smaller subset of data can outperform using one-hot targets on the full dataset under the same training budget. Motivated by this, they explore the impact of different sample-to-target mapping strategies on training efficiency. They categorize these strategies into three types:

STRATEGY A: Multiple augmented samples within the same class are mapped to a single one-hot target (conventional supervised learning).
STRATEGY B: Each augmented sample is mapped to a unique soft target generated by a teacher model (knowledge distillation).
STRATEGY C: Multiple augmented views of a single sample are mapped to the same soft target (proposed method to reduce noise in soft targets).
Theoretical Analysis: It would be ideal to provide a theoretical framework or intuition to explain the empirical observations, especially concerning why weaker teacher models can aid early learning and why STRATEGY C effectively reduces noise.

This addition would be a nice enhancement rather than any requirement, but I am not allowed to leave this section blank.🥸
Yes
No ethics review needed.
Figure 1: How are the results obtained? For example, how were the number of epochs determined?
Section 3.2: I do not fully understand the motivation behind the design of a unified loss function. It is said that CE is unable to exploit information from soft targets, but CE can be used with soft targets, so it’s not clear to me what is meant with that statement. Also, why is it not a valid evaluation strategy to evaluate strategies B and C by training the network exclusively using the soft labels with cross-entropy?
Later it is said that KL divergence is used to leverage the information in the soft targets, but this seems to contradict what was said above, i.e. that CE cannot make use of soft labels, but minimizing CE and KL divergence is equivalent.
Figure 5:
Which strategy was used to train the student?
On line 405 it says: “MixUp-trained teacher models achieve superior performance compared to standard augmentation”, but this is not supported by Figure 5a?
Why are the results of the experiments from the appendix not summarized in the main paper, i.e. whether they support the six key findings?
They pose an interesting question how different target encodings influence training efficiency of neural networks
Interesting experiments are designed that investigate questions such as how the quality of labels affects the accuracy of the student during different stages of the training, whether better teacher performance always entails better student performance or the interplay of data augmentation for the student and the teacher.
All experiments are repeated at least five times.
The paper is well structured and easy to read
The main goal of the paper is to investigate the influence of different target encodings on learning efficiency in an exploratory way that generates some interesting new hypotheses.
The authors categorize three types of target encodings: hard labels (A), soft labels based on the un-augmented input (B) and soft labels from the augmented input (C).
To empirically compare the three approaches, they define a teacher-student training setup.
The teacher is trained using approach A and is used to generate soft labels for approaches B and C. Student networks are then trained using approaches A, B and C and they are evaluated with respect to top-1 accuracy on three image classification tasks. The network structure consists of a backbone and two heads, one for the soft labels and one for the hard labels respectively. For approaches B and C, the hard labels are only used to train the network head for the hard labels while the backbone and head for the soft labels are trained using the soft labels. As the authors are interested in disentangling the influence of the soft and hard labels on the representational capacity of the backbone, this training setup prevents the hard labels from influencing the structure of the backbone. In the remainder of the paper they evaluate the three methods in different experimental setups, where they use teachers with different quality, select different numbers of observations per class and also vary the data augmentation schemes for the teacher and the student. From these experimental results they are able to derive six findings that relate the varied experimental factors to the accuracy of the student model. For example, soft targets can speed up early training and student networks – when training using approach B and C – are limited by the capacity of the teacher.
The experiments fail to consider other possibly relevant factors. For example, it is possible that strategy A in the results from figure 3a) simply needs a different learning rate
While the experiments are repeated at least five times, no uncertainty quantification (such as standard error) is included in the plots or the analysis.
Research data:
No code is provided
Experiment results are not included, e.g. as csv files
While interesting experiments are designed and phenomena are observed, little explanation is offered as to why these patterns are being observed.
For example, in the context of Figure 7 it would be interesting to discuss the effects the different augmentation strategies have on the class probabilities, which might explain why some augmentation methods perform worse with strategy C than with strategy B. 
When an augmentation method actually changes the class probability of an image (such as random cropping), using strategy C to train the network does not reduce noise but instead feeds the network wrong soft labels.
Yes
No ethics review needed.
Can the authors explain why the results/insights from the paper are relevant to neural scaling laws at the practically relevant scales discussed in the introduction?
Data and computational efficiency are highly relevant practical problems. As we reach fundamental upper limits on the possible size of training datasets, finding ways to improve the neural scaling laws that have been observed until now will be essential for continuing to improve model capabilities. Thus, the stated problem under study is relevant to the ICLR community.
The authors study how three different categories of data augmentation (dubbed Strategies A, B, and C) affect the rate of improvement in model test accuracy with respect to training budget and training dataset size. The methods are grouped by how the data augmentation affects the sample *label* rather than the features: the first group uses standard one-hot labels, the second uses soft labels from a teacher model but recomputed on each augmented feature vector, and the final group computes a soft label only on the pre-augmented feature vector and uses this for all augmented samples. They conduct experiments to determine which augmentation strategy is optimal in different settings and provide general recommendations based on these results.
The main thrust of the paper is that, in the context of improving neural scaling laws, their "finding underscores the significance of the exploration the target component, a frequently overlooked aspect in the deep learning community." The discussion of neural scaling laws centers of the fact that exponentially larger datasets are needed to achieve only marginal performance improvements; in particular, these scaling laws are a problem only once we have reached the "extremely large dataset regime." On the other hand, the insights the authors provide showing the advantage of augmented targets all occur in low data regimes. As they return to the size of the full CIFAR-10 dataset, regular 1-hot labels have the best performance. Thus, it's unclear what relevance the insights in the paper have to the stated practical problem of interest, i.e., datasets at a scale far beyond that of CIFAR-10.

Related to this first point, Claim 3 on the efficacy of the different strategies is misleading. In particular, the best overall performance is in fact obtained by Strategy B with a 90% accurate teacher model (Table 2 in the appendix).

The choice to separate the model training into the "backbone" (feature extractor) and classifier is motivated by the claim that the cross-entropy loss cannot handle soft labels, but this is not true. The cross-entropy can be computed between any two discrete distributions with the same support (https://en.wikipedia.org/wiki/Cross-entropy). In fact, the KL divergence and cross-entropy differ by a quantity which is constant w.r.t. the trained model, so the gradients for the proposed training strategy for the backbone are the same as the standard CE loss. It should also be noted that previous data augmentation methods which use soft labels (such as MixUp) also apply CE with the soft labels directly.

The description of the results in Fig. 5a seems to be incorrect. This paragraph states that MixUp > standard augmentation > no augmentation, but the plot has standard augmentation with the highest performance with MixUp and no augmentation approximately equal for the relevant purple and blue lines.

Minor: A more descriptive name than "Strategy A/B/C" would make it easier for the reader to remember the salient features of the different augmentation methods.
Yes
No ethics review needed.
My questions that need clarification are included in the weaknesses section.
1. Studying neural scaling laws and exploring how to overcome them to achieve a balance between efficiency and performance is both practical and meaningful.
2. The paper presents extensive experimental results from multiple perspectives.
The paper investigates the effects of soft target formats, teacher model performance, data quantity, and data augmentation on the training process, within a framework similar to knowledge distillation. The paper primarily presents extensive experiments on CIFAR outputs and summarizes the key findings derived from these experiments.
1. For scaling law research, the experimental scale is too small, even with the so-called “large-scale” Tiny-ImageNet dataset mentioned in the paper.
2. The paper merely lists observations without extracting the underlying insights or potential implications for practical applications.
3. I have concerns about the experimental results. For instance, a ResNet-18 model trained on CIFAR-10 typically achieves over 95% accuracy, yet the best result reported in the paper is only around 80%.
4. The so-called key findings are also trivial. For example, “Soft targets expedite early-stage training”: it’s well-known that knowledge distillation from a teacher model accelerates model convergence. Additionally, “one-hot targets yield superior final accuracy” is unsurprising, as the teacher models in the paper are weak and unable to match the performance of traditional supervised learning, thus hindering final accuracy. I don’t believe this conclusion would hold for more challenging tasks and strong teacher models.
5. I believe the statement “this study is the first to emphasize the critical role of targets in breaking neural scaling power laws” is an overclaim. As mentioned in lines 50-57, there are already existing works on it.
6. The paper does not provide the experimental setup for Figure 1, and the conclusions drawn are inconsistent with those shown in Figure 6(a), where even a 100% subset of soft targets fails to outperform hard targets.
1. Some works regarding offline and online data selection on efficient learning, such as those listed below, should be discussed.

[1] Spanning training progress: Temporal dual-depth scoring (TDDS) for enhanced dataset pruning. CVPR 2024.

[2] Data selection for language models via importance resampling. NeurIPS 2023.

[3] Diversified batch selection for training acceleration. 2024. ICML 2024.

[4] Towards accelerated model training via bayesian data selection. NeurIPS 2023.

[5] Coveragecentric coreset selection for high pruning rates. ICLR 2023.

[6] Beyond neural scaling laws: beating power law scaling via data pruning. NeurIPS 2022.

I’m not sure if I have misunderstood certain parts of the paper. However, based on my current assessment, I believe this paper is not suitable for publication at ICLR. I will adjust my score accordingly, depending on the authors’ clarifications and modifications during the rebuttal phase.
@inproceedings{
anonymous2024equally,
title={Equally Critical: Samples, Targets, and Their Mappings in Datasets},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=FM21yYBhuE},
note={under review}
}
Neural scaling laws highlight the trade-off between test error reduction and increased resources in machine learning, revealing diminishing returns as data volume, model size, and computational power increase.
This inefficiency poses sustainability challenges, as marginal performance gains necessitate exponential resource consumption.
Recent works have investigated these laws from a data-efficient standpoint, primarily concentrating on sample optimization, while largely neglecting the influence of target.
In this study, we first demonstrate that, given an equivalent training budget, employing soft targets on a 10% subset can outperform the use of one-hot targets on the full dataset. Building on this observation, we review existing paradigms in the sample-target relationship, categorizing them into distinct sample-to-target mapping strategies.
Subsequently, we propose a unified loss framework to assess their impact on training efficiency. Finally, we conduct a comprehensive analysis of how variations in target and sample types, quantities, and qualities influence training efficiency across three training strategies, providing six key insights to enhance training efficacy.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Data-efficient Learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/e62543b758f1092f5b78271850289d8ddcac40a9.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Equally Critical: Samples, Targets, and Their Mappings in Datasets
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission