GIFT: Unlocking Full Potential of Labels in Distilled Dataset at Near-zero Cost
Dataset Distillation;Soft Label
Yes
No ethics review needed.
- Can you explain theoretically why different optimizers have such a big impact on DD?
- It is interesting to find that models trained on synthetic datasets show high sensitivity to the choice of loss function for soft label utilization.
- The authors reveal that traditional loss functions suffer from significant robustness deficiencies when applied across different optimizers.
- They propose GIFT and prove its effectiveness through experiments.
- The experiment is well designed.
This paper conducts a comprehensive comparison of various loss functions for soft label utilization in DD and find that the model trained on the synthetic dataset exhibits high sensitivity to the choice of loss function for soft label utilization. Building on these insights, the authors introduce GIFT, which encompasses soft label refinement and a cosine similarity-based loss function to use full label information. Extensive experiments demonstrate the effectiveness of GIFT.
- In Table 1, the results of ResNet-18 on CIFAR-100 are not very significant, which may not reflect the effect of the proposed algorithm.
- The current synthetic datasets on Tiny-ImageNet and ImageNet-1K are only generated using Conv-4. Can we experiment with structures such as transformer?
Yes
No ethics review needed.
- Could you share the results of your label utilization method on a randomly selected data subset?  
- How does GIFT perform with higher IPC values, such as 100 or 200?  
- Is it possible to achieve lossless dataset distillation using GIFT on any dataset?  
- It would be interesting to see how the Jensen-Shannon divergence loss function performs within the GIFT framework.
[S1] The paper dives into a crucial part of dataset distillation, specifically how to choose effective loss functions for label utilization in DD frameworks.  
[S2] The writing is clear and easy to follow.  
[S3] The use of cosine similarity is backed by solid theoretical reasoning.  
[S4] The ablation studies are detailed, and the method shows promising results in continual learning.  
[S5] GIFT consistently improves state-of-the-art dataset distillation methods across various datasets without adding any extra computational cost.
This paper introduces a novel method for dataset distillation that fully leverages soft labels generated by pre-trained teacher models. The authors highlight the critical role of selecting the right loss function, showing that the model's performance on synthetic datasets is highly dependent on this choice. They propose **GIFT**, a simple yet effective method that refines soft labels and uses a cosine similarity-based loss function to boost performance across various tasks. The approach also enhances generalization across optimizers and architectures. Extensive experiments demonstrate that **GIFT** significantly improves performance without adding computational costs, making it scalable even for large datasets like ImageNet-1K.
[W1] From a technical standpoint, the contribution of the paper feels somewhat limited. Both the hard-label refinement approach and the cosine-similarity loss function have already been explored in many knowledge distillation studies. Additionally, the paper lacks convincing evidence or justification that these techniques are specifically suited for the synthetic datasets used in dataset distillation.

[W2] Regarding the experimental results, the improvements brought by the proposed label utilization algorithm to existing DD methods are relatively modest. However, since the approach incurs no additional computational cost and offers general performance boosts, the results are still reasonable and acceptable.
Yes
No ethics review needed.
1. How would cosine similarity perform if it is applied directly on hard labels / smoothed labels? Will it still outperform the other compared loss functions?
2. The performance improvement is generally higher on small datasets / poor architectures compared with the other end. Can the authors explain the potential reason?
1. The motivation is clear and the designed model has correspondence to the motivation. 
2. The experiments results show uniformly performance improvement over multiple distilled datasets and multiple benchmarks.
3. It is valuable to look into the validation for dataset distillation.
This work focuses on the problem of soft label utilization in dataset distillation. Currently, multiple different loss functions are employed to use the soft labels generated by pre-trained models. The choice of loss function can have large influence on the training effects for different distilled datasets. Therefore, the authors propose to refine the generated soft labels, and adopt a unified cosine similarity-based loss function to leverage the soft label information. The experiments suggest that the proposed method uniformly improve the validation performance on multipld benchmarks.
1. The authors claim that GIFT can correct erroneous signals from the teacher model, particularly in cases where the teacher assigns an incorrect label of the highest value. However, there is no qualitative or quantitative result supporting it. Since the $\gamma$ parameter is only set as 0.1, it remains unclear how it can correct potential soft label error. Can the authors summarize how much the soft labels change after the refinement?
2. The authors introduce a new evaluation scenario named cross-optimizer generalization. But it is not defined until the end of the introduction section. It is recommended to raise the issue of poor cross-optimizer generalitzation earlier. Does it serve as one of the motivations, or is it simply empirical results after the method is designed?
3. The authors show in Table 6 the cross-architecture generalization results on Tiny-ImageNet. In many cases the performance gain of large models is on par or lower than that on small models. Is the improvement benefited simply from better training effects of cosine similarity or better generalization?
4. Although the proposed method uniformly improves the validation results with soft labels on different distilled datasets, the soft label requires enormous storage space. While it is feasible to use currently available pre-trained models for popular dataset distillation benchmarks, it is not practical for actual use on new application scenarios. Thus, the overall practicality slightly degrades the value of this work for the dataset distillation society.
Yes
No ethics review needed.
Please refer to weaknesses.
1. Unlike previous methods, this paper shifts focus from the distillation process itself to the model training phase using synthetic data, presenting a novel perspective for dataset distillation. The paper proposes a universal loss function that enhances the performance of most dataset distillation methods. 
2. Currently, most state-of-the-art methods incorporate soft labels into distillation, complicating unified evaluation when training models with synthetic data. The approach in this paper could potentially help establish a standardized evaluation benchmark for dataset distillation products.
3. Extensive experimental results demonstrate that GIFT significantly improves state-of-the-art dataset distillation methods on large-scale datasets, such as ImageNet-1K, while also enhancing cross-optimizer and cross-architecture generalization. GIFT’s plug-and-play nature makes it versatile across various datasets and network architectures.
4. The presentation of the paper is great. It provides a detailed discussion and clearly showcases the motivation and methodology.
This paper proposes a simple but effective plug-and-play dataset distillation method to leverage both hard and soft labels at near-zero cost. 
It identifies that existing dataset distillation methods are highly sensitive to different loss functions, highlighting the need for a universal loss function. Based on this insight, GIFT introduces a label refinement process that incorporates hard label smoothing, thereby correcting any erroneous signals from the teacher model and enhancing inter-class separation. Additionally, the authors use a cosine similarity-based loss function to fully exploit label information during training.
1. Although the method offers a novel perspective, its lack of improvement in the distillation process itself somewhat limits the contribution of the paper. 
2. I suggest the authors consider incorporating GIFT into [1], i.e., replacing the loss with GIFT's loss in [1], or adding a loss term in other trajectory-matching methods (as long as they can access a well-trained model during training), I am interested to see if this combination could further improve the effectiveness of dataset distillation.


[1] Can pre-trained models assist in dataset distillation? arXiv:2310.03295
We propose an effective plug-and-play approach to consistently enhances the SOTA dataset distillation methods without incurring additional computational costs.
@inproceedings{
anonymous2024gift,
title={{GIFT}: Unlocking Full Potential of Labels in Distilled Dataset at Near-zero Cost},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=FoF5RaA3ug},
note={under review}
}
Recent advancements in dataset distillation have demonstrated the significant benefits of employing soft labels generated by pre-trained teacher models.
    In this paper, we introduce a novel perspective by emphasizing the full utilization of labels.
    We first conduct a comprehensive comparison of various loss functions for soft label utilization in dataset distillation, revealing that the model trained on the synthetic dataset exhibits high sensitivity to the choice of loss function for soft label utilization.
    This finding highlights the necessity of a universal loss function for training models on synthetic datasets.
    Building on these insights, we introduce an extremely simple yet surprisingly effective plug-and-play approach, GIFT, which encompasses soft label refinement and a cosine similarity-based loss function to efficiently leverage full label information. 
    Extensive experiments indicate that GIFT consistently enhances state-of-the-art dataset distillation methods across various dataset scales without incurring additional computational costs.
    Importantly, GIFT significantly enhances cross-optimizer generalization, an area previously overlooked.
    For instance, on ImageNet-1K with IPC = 10, GIFT enhances the state-of-the-art method RDED by 30.8\% in cross-optimizer generalization.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Dataset Distillation Soft Label
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/d6ec969c04bf93232a47c4c6d197a604c09a83d7.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
GIFT: Unlocking Full Potential of Labels in Distilled Dataset at Near-zero Cost
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission