Compressing Vision Foundation Models at ImageNet-level Costs
Foundation Model;Model Compressing;DINOv2;CLIP;SynCLR
Yes
No ethics review needed.
Suggestions:

Training small models with distillation for improved performance is a fairly standard practice, and there are numerous distillation loss functions proposed over the last decade. An important question worth looking into in the current "data-centric ML world" is ``What is the right dataset to use for distillation?``. While the paper is looking into this, focusing solely on ImageNet dataset makes the paper's contribution fairly limited, especially given that there are several large-scale public image datasets available now. 

You could avoid the redundant/confusing terminology used for the three loss terms in the paper.
Presentation: 
The paper was written well and was easy to follow.

Experiments: 
Authors evaluated their models on a wide range of CV tasks (classification, segmentation, depth estimation) using several downstream datasets. Experimental results show that models distilled only on ImageNet using the proposed approach match/outperform same size models trained on significantly larger datasets without distillation. Experiments also show that the feature-based distillation performs better than logit-based distillation approach DeiT.
While many Vision Foundation Model (VFM) checkpoints are available, their training datasets are inaccessible. This paper shows that distilling a vision foundation model using only the ImageNet dataset gives us a model that matches the performance of a similar size model trained with massive data, on several vision benchmark datasets from different domains. The paper uses the popular intermediate feature distillation strategy for KD. Experiments were conducted using several teacher-student combinations.
``The problem setting of the paper is not convincing`` 
While the original datasets used to train models such as DINOV2 are unavailable, there are numerous large scale image datasets (DataComp, LAION, etc.) that are available to the community these days. If the goal is to get a small model that performs well on a broad array of tasks, I do not understand why one should restrict themselves to distilling on ImageNet. They can use more diverse datasets, for example, diverse subsets of data taken from one of the large-scale public datasets. If the paper had compared different datasets in terms of their KD effectiveness or proposed methods to create better general purpose KD datasets, that would have been an interesting and useful contribution to the community.

The authors performed an experiment by using target task datasets for distillation and showed improved performance. This is an obvious/expected behavior, which also suggests that "we should go beyond ImageNet". If one has access to some data from target tasks, they can curate better distillation datasets as shown in a recent paper: "Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models", ICML 2024.



``Limited technical contribution``
Feature distillation is a standard approach in ML community. There are numerous papers that proposed various distillation objectives based on intermediate features. See "Knowledge distillation: A survey", International Journal of Computer Vision, 2021.
The presented approach uses standard l2 distillation loss on classification and patch tokens. It leverages masking strategy which is also a commonly-used data augmentation approach when training vision models.


``Presentation - Redundant/Confusing terminology``
The terminology of token-level, feature-level and patch-level is a bit confusing/redundant. In a vision transformer setting each image patch becomes a token, so it is confusing to refer to "classification token-based loss" as "token-level" and "patch token-based loss" as "feature-level". Both losses are token-based losses (either CLS token or patch token). Alternatively, both losses can also be called feature-based losses (CLS token features or patch token features). Both feature-level and patch-level losses described here are based on patch-tokens. So, the terminology of "feature-level vs patch-level" is confusing since both are actually patch-level losses, one with masking and one without masking. 

Table 1 - What does "Hint" in Table.1 refer to?
Yes
No ethics review needed.
The definition of "feature" is ambiguous. Currently, this paper just defines it as "intermediate feature _v_" (Line 171) without any additional information. This paper should specify which specific feature maps of vision transformers is used, like the output of the last transformer encoder layer of ViTs, the outputs of all 12 transformer encoder layers of ViTs, the attention maps, the Q/K/V within self-attention modules, etc.
1. This paper is well-motivated. With the development of foundation model trained with private data, it's difficult for the community to reproduce or compress the foundation model due to the inaccessible training data. Thus, how to use existing academic public datasets as a proxy to achieve comparable performance is an interesting question. 

2. The performance of the model is excellent, as it achieve comparable performance with the foundation model with much less training data. Besides, this paper conducts extensive experiments and ablation studies to demonstrate the generalization ability of Proteus. 

3. The empirically finding that conventional knowledge distillation with CE loss introduce dataset bias is interesting.
This paper aims to compress pre-trained vision foundation models, such as CLIP and DINO-V2, using only publicly available, small-scale datasets like ImageNet-1k, as the original training data is inaccessible. 
To achieve this, this paper proposed _Proteus_, which perform knowledge distillation (KD) from the pre-trained model at token-level, patch-level, and feature-level. With extensive experiments, this paper demonstrate that the distilled model achieves comparable performance with the foundation model across multiple tasks.
1. The novelty of this paper is limited. Although the results are promising, the loss at token, feature, and patch level has been widely explored in vision transformer distillation [1,2,3,4]. Besides, all of them can be considered as KD with intermediate features, which has been a common setting in KD. However, this paper does not cite and discuss any recent related works that perform KD with intermediate feature maps. 

2. Although this paper presents experimental results to support that empirically removing CE loss can mitigate dataset bias, it would be better to provide some theoretical analysis to understand this question better.

[1] Yang, Zhendong, et al. "ViTKD: Feature-based Knowledge Distillation for Vision Transformers." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.  
[2] Chen, Xianing, et al. "Dearkd: data-efficient early knowledge distillation for vision transformers." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.   
[3] Wang, Wenhui, et al. "Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers." Advances in Neural Information Processing Systems 33 (2020): 5776-5788.   
[4] Wang, Wenhui, et al. "Minilmv2: Multi-head self-attention relation distillation for compressing pretrained transformers." arXiv preprint arXiv:2012.15828 (2020).
Yes
No ethics review needed.
See weaknesses.
1. This paper presents a simple yet effective method that utilizes the hidden layer features of the teacher network to successfully transfer its generalization ability to the student network.

2. The article provides extensive experimental validation of its claims on the ViT architecture, with datasets that include classification, segmentation, and depth estimation. The student network architecture encompasses various sizes, including ViT-small, ViT-base and ViT-large.

3. This paper is well-written.
This paper proposes a simple yet effective method for distilling the generalization ability of visual foundation models using only the ImageNet dataset. Specifically, compared to traditional knowledge distillation, it omits hard logits distillation, cross-entropy loss, and KL-divergence loss, while introducing MSE loss to align the intermediate results of the student network with those of the teacher network. The alignment of intermediate results involves three levels: token level, feature level, and patch level. Experiments show that the proposed method effectively distills the generalization ability of the teacher network.
1. This paper is less novelty. Previous works have also utilized intermediate layers for distillation on ImageNet-1K and achieved good performance on other datasets, such as MiniViT[1], and [2]. What are the differences between this work and those approaches. What are the comparative results of these methods with your work?

2. Figure 3 is not referenced in the main text, and it is unclear what issue Figure 3 is intended to illustrate.

3. When validating the generalization ability of the student network, how many epochs were fine-tuned for fine-grained classification, segmentation, and depth estimation? Compared to baseline methods like DeiT and IBOT, which used the same amount of data, was the computational load higher?

4. Why does Figure 7 show that more data leads to lower results on the ImageNet linear evaluation? For instance, sampling 50% of the data seems to yield better results than sampling 100% of the data.

5. How do the results of distillation when integrating hard logits distillation, the Cross-Entropy loss, the KL-divergence loss, and the MSE loss compare to the results obtained using only the proposed method? Would there be a decline in performance?

[1] Zhang J, Peng H, Wu K, et al. Minivit: Compressing vision transformers with weight multiplexing[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 12145-12154.

[2] Zhang H, Zhou Y, Wang G H. Dense Vision Transformer Compression with Few Samples[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 15825-15834.
Yes
No ethics review needed.
Please refer to weaknesses.
1. Proteus offers a practical and accessible method for distilling foundation models, addressing the resource-intensive nature of training such models.
2. Proteus demonstrates appealing performance and efficacy in knowledge transfer, matching or outperforming models trained on significantly larger datasets.
3. The paper is well-structured and effectively communicates the proposed solution and its results.
The paper proposes Proteus, a method for distilling large vision foundation models into smaller equivalents on ImageNet-1K without access to the original training data. By focusing on token, patch, and feature-level training objectives, Proteus achieves impressive results at ImageNet-level costs, enabling broader access to training foundation models. When compared to other models including DINOv2-L/14, CLIP-L/14, OpenCLIP-L/14, and SynCLR-L/14, Proteus demonstrates competitive performance across 15 benchmarks, matching the Oracle method in some cases.
1. What is the performance of Proteus-L/14 distilled from the DINOv2-L/14 teacher? This informs the extent of performance degradation when distillation using ImageNet-1K.
2. Validation on varying data scales: The paper lacks validation on different data scales, such as subsets of ImageNet-1K or larger datasets like ImageNet-21K, which could provide more comprehensive insights.
3. Validation on other models and domains: The authors are suggested  to validate the effectiveness of  Proteus on diverse models and domains, such as large-scale pre-trained LLMs or diffusion models, to validate generalizability.
Compressing DINOv2, CLIP, SynCLR on ImageNet with comparable performance
@inproceedings{
anonymous2024compressing,
title={Compressing Vision Foundation Models at ImageNet-level Costs},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=LC6ZtQV6u2},
note={under review}
}
Vision foundation models are renowned for the generalization ability due to massive training data. Nevertheless, they demand tremendous training resources, and the training data is often inaccessible, e.g., CLIP, DINOv2, posing great challenges to developing derivatives that could facilitate the research. In this work, we offer a very simple and general solution, named Proteus, to distill foundation models into smaller equivalents on ImageNet-1K without access to the original training data. Specifically, we remove the designs from conventional knowledge distillation settings that result in dataset bias and present three levels of training objectives, i.e., token, patch, and feature, to maximize the efficacy of knowledge transfer. In this manner, Proteus is trained at ImageNet-level costs with surprising ability, facilitating the accessibility of training foundation models for the broader research community. Leveraging DINOv2-g/14 as the teacher, Proteus-L/14 matches the performance of the Oracle method DINOv2-L/14 (142M training data) across 15 benchmarks and outperforms other vision foundation models including CLIP-L/14 (400M), OpenCLIP-L/14 (400M/2B) and SynCLR-L/14 (600M).
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Foundation Model Model Compressing DINOv2 CLIP SynCLR
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/de0a696306f79fbee6f870eddd9eb092467fb27f.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Compressing Vision Foundation Models at ImageNet-level Costs
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission