Exploring Learning Complexity for Efficient Downstream Dataset Pruning
data efficiency
Yes
No ethics review needed.
see the Weaknesses
1.) The significance lies in its potential to reduce the computational burden of fine-tuning large pre-trained models while maintaining performance.
2.) The paper is well-structured, and easy to follow.
3.) The introduction of FlexRand adds another layer of adaptability to the pruning process, making it more robust across different data regimes, a valuable contribution to data pruning strategies.
The paper introduces Distorting-based Learning Complexity (DLC), a novel training-free hardness score for efficient downstream dataset pruning. DLC quantifies sample hardness by masking pre-trained weights and approximating loss integration via the Monte Carlo method. The authors also propose FlexRand, a flexible under-sampling strategy to adapt to different data regimes and avoid distribution shift.
1.) The paper suggests that DLC is not sensitive to the quality of pre-trained models, but this claim could be further experimented on different size level pre-trained models.
2.) The method requires storing multiple masked models, which could be a limitation in environments with constrained memory resources, potentially affecting the practicality of the approach.
3.) The paper could benefit from a more detailed discussion on scenarios where DLC might underperform or fail, providing a more comprehensive understanding of its limitations.
Yes
No ethics review needed.
When referring masking the pre-training weights, what specific operation is performed on the network parameters?
What's the meaning of dotted line in blue(10%), green(20%) and orange(30%) in Figure(d)?
The proposed scoring function, Distorting based Learning Complexity, is an efficient training-free score for dataset pruning. A under-sampling strategy with randomness, FlexRand, is designed to adapt to different data regimes and avoid distribution shift. Extensive experiments demonstrate the effectiveness of the proposed approach. DLC significantly reduces the pruning time by 35× in images pruning benchmark.
A novel training-free hardness score, Distorting-based Learning Complexity, is proposed to identify informative images and instructions from downstream dataset. Also, a flexible under-sampling method with randomness named FlexRand is proposed to alleviate the severe subset distribution shift. Extensive experiments demonstrate the effectiveness and efficiency of the proposed approach.
Some typo: Line 021, "a flexible under-sampling with randomness" -> "a flexible under-sampling strategy with randomness"
In Figure 4(a), the MMD value of Random is missing.
Yes
No ethics review needed.
1. How is the loss integration implemented? In the integration figures, the upper bound of loss is 1.0. Is it normalized to the range of (0, 1)?
2. How is the masking applied to the neural network? 
3. How is the splitting hyper-parameter $\gamma$ determined in the actual use? If multiple values need to be tested, the tuning time should also be counted towards the pruning time in Figure 1.
1. The design of using random masks to produce classifiers with different capabilities is interesting and practical. With the averaged feature serving as the prediction head, there is no more need to fine-tune the classifier on downstream tasks. 
2. Detailed experiments are conducted to illustrate the effetiveness of the proposed method. The method can be applied to both image and instruction datasets, both demonstrating performance improvement. 
3. The writing is generally fluent and easy to follow.
This work describes a novel dataset pruning method without the need of pre-training on the target dataset. Given models pre-trained on large scale datasets, this work proposes a Distorting-based Learning Complexity score to identify informative images and instructions. Sample hardness is estimated by randomly masked neural networks, representing networks with different capabilities. Then samples are randomly sampled from the easy and hard groups, respectively. The proposed method achieves effective dataset pruning with 35x less pruning time.
1. The authors claim that easy samples are more likely to be correctly classified by a weak classifier in the front part of the learning path. However, the overall Learning Complexity score is acquired by averaging classification loss of multiple randomly sampled networks. The definition of learning path seems not to be utilized in the method design. 
2. Can the utilization also be applied to some of previous methods? For example, the Herding method uses parameter influence as scores for each sample. Here the fine-tuned model can also be substituted by a pre-trained model with averaged features as the prediction head. Although the direct employment of pre-trained models is practical, it is not a unique design. And it will be interesting to see if applying the strategy to previous methods also leads to performance improvement. 
3. The strategy of dividing datasets into different groups and randomly sampling from each group is similar to the idea in Dataset Quantization [1]. Dataset Quantization first iteratively separate the data into multiple bins with coreset selection methods. Normally the early groups tend to cluster around the distribution center, while the later groups show more diversity. By sampling from each bin, the overall distribution will be kept similar to the original one. This paper has a similar claim that FlexRand avoids severe distribution shift. Please discuss the difference of the proposed strategy from Dataset Quantization and the advantages of it. 
4. Section 5 discusses the quality of pre-trained models. The authors claim that the method is not sensitive to the quality of pre-trained models. But weakly supervised models are not always worse than fully supervised models. Please also show the original performance comparison between these two groups of models. 
5. Minor:
    - The use of pretrain and pre-train need to be unified in the paper. 
    - The sample number is represented both by N (line 105) and |D| (line 129). Please unify the use. 

[1] Zhou, Daquan, et al. "Dataset quantization." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
We propose a novel and efficient hardness score for downstream dataset pruning.
@inproceedings{
anonymous2024exploring,
title={Exploring Learning Complexity for Efficient Downstream Dataset Pruning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=FN7n7JRjsk},
note={under review}
}
The ever-increasing fine-tuning cost of large-scale pre-trained models gives rise to the importance of dataset pruning, which aims to reduce dataset size while maintaining task performance.
However, existing dataset pruning methods require training on the entire dataset, which is impractical for large-scale pre-trained models.
In this paper, we propose a straightforward, novel, and training-free hardness score named Distorting-based Learning Complexity (DLC), to identify informative images and instructions from the downstream dataset efficiently.
Our method is motivated by the observation that easy samples learned faster can also be learned with fewer parameters.
Specifically, we define the Learning Complexity to quantify sample hardness and utilize a lightweight weights masking process for fast estimation, instead of the costly SGD optimization.
Based on DLC, we further design a flexible under-sampling with randomness (dubbed FlexRand), replacing the top-K strategy, to alleviate the severe subset distribution shift.
Extensive experiments with downstream image and instructions dataset pruning benchmarks demonstrate the effectiveness and efficiency of the proposed approach.
In the images pruning benchmark, DLC significantly reduces the pruning time by 35$\times$ while establishing state-of-the-art performance with FlexRand.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
data efficiency
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/804ae52a453b942db570f9ee419539a02d930952.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/71aa5439bfe69774afee09e796b6c5f07e550e17.zip
Exploring Learning Complexity for Efficient Downstream Dataset Pruning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission