Zero-Shot Distillation for Image Encoders: How to Make Effective Use of Synthetic Data
Niclas Popp Jan Hendrik Metzen Matthias Hein




 



Zero-Shot Distillation for Image Encoders






N. Popp et al.




¹University of Tübingen
 ²Bosch Center for Artificial Intelligence (BCAI), Robert Bosch GmbH

Zero-Shot Distillation for Image Encoders:
 How to Make Effective Use of Synthetic Data
    Niclas Popp¹^,², Jan Hendrik Metzen², Matthias Hein¹
    April 29, 2024
=======================================================================================




Multi-modal foundation models such as CLIP have showcased impressive zero-shot capabilities. However, their applicability in resource-constrained environments is limited due to their large number of parameters and high inference time. While existing approaches have scaled down the entire CLIP architecture, we focus on training smaller variants of the image encoder, which suffices for efficient zero-shot classification. The use of synthetic data has shown promise in distilling representations from larger teachers, resulting in strong few-shot and linear probe performance. However, we find that this approach surprisingly fails in true zero-shot settings when using contrastive losses. We identify the exploitation of spurious features as being responsible for poor generalization between synthetic and real data. However, by using the image feature-based ℒ_2 distillation loss, we mitigate these problems and train students that achieve zero-shot performance which on four domain-specific datasets is on-par with a ViT-B/32 teacher model trained on DataCompXL, while featuring up to 92% fewer parameters.







§ INTRODUCTION

Motivation.
Image classifiers built on top of large vision(-language) foundation models, such as CLIP <cit.> or DINOv2 <cit.>, have shown impressive zero-shot capabilities across various tasks. However, their extensive parameter count and high inference latency present significant challenges for deployment in resource-constrained edge devices used in driver-assistance systems, automated driving, mobile robotics, or video surveillance. Due to their reduced capacity, smaller models cannot be expected to match the performance of larger ones in arbitrary domains. Additionally, training large-scale foundation models typically involves several millions or billions of images, making it expensive and time-consuming. Together, this motivates the need for smaller domain-specific models, as well as data-efficient training procedures. In this work, we specifically focus on zero-shot image classification, for which only a small-scale image encoder is required. Class-specific text embeddings are fixed and can be precomputed off-device, while only image embeddings are computed on-device. Thus, our goal is to distill smaller drop-in replacements (students) of the CLIP image encoder (teacher) that achieve on-par performance on the specific target domains of interest. In particular, we want to specialize the image encoder student to novel domains for which we only know the relevant classes, but do not have access to actual images, the so called zero-shot distillation setting. 

For zero-shot distillation, domain-specific data can be obtained from “general-purpose” generative models, such as large-scale latent diffusion models <cit.>, by class-aware prompting. However, learning from synthetic images has proven  challenging <cit.>: using simple class-specific text prompts, like those used for zero-shot classification by CLIP <cit.>, yields low-diversity datasets and poor classifiers, as observed in previous studies <cit.>. Furthermore, when training on a combination of natural and such low-diversity synthetic images, the overall accuracy starts to decline as synthetic data outweighs real data <cit.>. More advanced methods for diversifying prompts with large-language models <cit.>, together with a large compute budget for synthetic image generation, allow achieving high accuracy in linear probe or few-shot scenarios, indicating strong representation learning capabilities. However, we observe in this work that even with diverse prompting, the performance in zero-shot distillation with actual zero-shot evaluation remains comparatively low. We identify that fine-tuning small CLIP image encoders with contrastive text-image losses leads to models that exploit spurious features in images and because of this exhibits poor generalization between synthetic and real images. Linear probe evaluation involves a final linear layer trained on real-data and is thus not affected by the same issue.

Besides identifying the issue of spurious feature learning, our main contribution is the (somewhat unexpected) observation that a simple change of the distillation loss can mitigate this problem. Specifically, we find that employing a ℒ_2 loss between student's and teacher's image features  substantially reduces the tendency of the models to exploit spurious features and enhances their generalization capabilities between synthetic and real data. We attribute this to the fact that this loss distills image encoders without any influence of the teacher's text encoder and the potential shortcut learning it might encourage. Through one epoch of pre-training on DataComp medium <cit.> and subsequent fine-tuning on diverse synthetic datasets generated using diffusion models and prompts from large language models, we achieve superior zero-shot classification performance on four target datasets compared to TinyCLIP <cit.>, the current state-of-the-art for distilled CLIP models. Furthermore, our approach achieves on-par performance with the teacher model, all without utilizing a single annotated image from the target domain during training. When using a image encoder that features only 11 million trainable parameters, we manage to achieve the zero-shot performance of a ViT-B/32 with 86 million parameters within a margin of 5 percent points and outperform a TinyCLIP model with 90 million trainable parameters on three of four test datasets.


Main contributions.
In this work, we show that small 1-to-1 replacements of the CLIP vision encoder can be efficiently and robustly trained in a zero-shot setting using feature distillation. Our main contributions are the following:

    
  * We introduce a unifying framework for zero-shot distillation of image encoders.
    
  * We identify failure cases of distillation when using contrastive losses for fine-tuning on small or synthetic datasets. We attribute this to models being prone to learning spurious features and overfitting to the training domain.
    
  * We propose feature distillation, which is not susceptible to these spurious features and generalizes better between synthetic and natural images.
    
  * Using our framework we manage to distill a ViT-B/32 CLIP vision encoder into student models with up to 93% fewer parameters that closely match the classification performance of the teacher and surpass existing baselines on the Oxford Pets <cit.>, Flowers-102 <cit.>, Stanford Cars <cit.> and Food-101 <cit.> datasets.




§ RELATED WORK

Knowledge Distillation of Vision-Language Models.
Knowledge Distillation <cit.> is a widely used technique for transferring knowledge from larger teacher models to smaller student models. In its vanilla form, the approach involves combining a standard training loss with a distillation loss that considers the output of both the student and teacher models, penalizing discrepancies between the two models. Knowledge distillation has been observed to not only benefit the test accuracy of the student on the target datasets but transfer other favorable properties of the teacher such as domain generalization <cit.>. While this approach has been well-established for single-modality tasks including vision <cit.> or language <cit.>, recent works have extended the concept to the multi-modal setting, specifically in the context of vision-language models. CLIP-KD <cit.> provides an extensive set of experiments comparing various different loss combinations. TinyCLIP <cit.> proposed an advanced initialization process using weight inheritance from the teacher to the student as well as a multi-stage progressive distillation culminating in models that are only 1/4 the size of a ViT-B/32 CLIP model. MobileCLIP <cit.> further refined the distillation process by incorporating image augmentation, synthetic captions, and dedicated architectural choices. In contrast to these existing methods, our approach focuses on finding only a one-to-one replacement of the vision encoder while the text encoder remains frozen. Apart from CLIP-specific techniques, unsupervised distillation based purely on images without labels has been identified as a data-efficient alternative to supervised training for vision encoders <cit.> and class-incremental generalization <cit.>. We build on this observation by combining unsupervised pre-training with targeted fine-tuning. Despite knowledge distillation being a widely adopted training technique, it has been observed that it does not always work as commonly understood. Even when the student features the same capacity as the the teacher, there can be significant discrepancies in their predictive distributions <cit.>.


Training and Distillation Using Synthetic Images.
Recent advancements in generative text-to-image models have sparked a growing interest in utilizing synthetic images for vision applications. Azizi et al. <cit.> demonstrated that images from fine-tuned text-to-image models can be combined with real images to enhance the accuracy of classifiers on ImageNet-1k <cit.>. For text-to-image generation, diffusion models are commonly employed, particularly for knowledge distillation <cit.>. However, it was observed that the performance deteriorates when the number of synthetic images surpasses that of real images. Yu et al. <cit.> attributed this decline to the lack of diversity in the used synthetic images. To mitigate this issue, they proposed a strategy to diversify the image generation process by incorporating prompts generated by large language models, thereby enhancing content and style variation. Another approach to diversification in the few-shot setting was presented by Da Costa et al. <cit.>, which involved augmentations and low-rank adaptation. By scaling up synthetic datasets, Tian et al.<cit.> and Hammoud et al. <cit.> demonstrated the feasibility of training vision-language foundation models solely using images from text-to-image models. However, achieving performance on par with or surpassing models trained on real data necessitates the utilization of a large number of synthetic images, on the order of 10^7 or 10^8. This not only prolongs the already long training process but also introduces additional computational overhead. Most importantly, the reported results are typically obtained by linear probing or after few-shot training and are not true zero-shot accuracies which we aim to optimize. To mitigate these challenges, we propose a hybrid approach that combines a large-scale dataset of natural images with a smaller set of domain-specific synthetic images. 




§ FRAMEWORK FOR ZERO-SHOT DISTILLATION


Zero-shot distillation refers to the process of transferring  knowledge from a teacher to a student model in a setting where one does not have access to images from the target domain. It is thereby a special case of data-free knowledge distillation which describes the setting where the training data for teacher and the student differ.  Zero-shot distillation specifically focuses on the ability of foundation models as teachers to perform well on unseen data due to their generalization properties. The objective is to transfer this performance to a smaller student model without utilizing any of the unseen data. Therefore, the primary goal is not to address the disparity between the datasets used to train the teacher and student, but rather to extract domain-specific knowledge from the teacher model without having access to the corresponding data. The term zero-shot distillation has been introduced previously <cit.>, yet only in the setting for single-modal classifiers that were trained using the cross-entropy loss. In our case, we consider CLIP which is a vision-language model instead of a simple image classifier. In this section, we present a structured framework for zero-shot distillation. Specifically, we discuss the data domains, the training pipeline, the generation of diversified synthetic training data as well as the selection of an appropriate loss function.



 §.§ Data Domain
 In the context of (pre-)training for a zero-shot setting, there are currently two core approaches. The first one involves relying on large-scale data such as common crawl datasets <cit.>. While this approach is feasible for large foundation models, it poses challenges for smaller models as these lack the same level of generalization capabilities due to their smaller capacity. The second approach involves training from scratch using either purely synthetic images <cit.> or a combination of real and synthetic images <cit.>. Yet by incorporating few-shot learning <cit.> on real images or linear probing <cit.> after training on synthetic data, the reported accuracies are no longer truly zero-shot. We optimize the actual zero-shot performance by adopting a two-stage approach: in the first stage, we pretrain on a large-scale general-purpose dataset consisting of natural images, and subsequently fine-tune using a smaller set of domain-specific synthetic images. This approach allows us to address the limitations of relying solely on generalization or training on synthetic data, and enables us to achieve strong zero-shot performance.





 §.§ Training Pipeline

In order to shorten training in comparison to training from scratch, Wu et al. <cit.> have introduced weight inheritance as an initialization scheme for distilling CLIP models. This method has a significant limitation as it can only be applied when the student model shares a similar architecture with the teacher model. Instead of using weight inheritance, we introduce a pre-training step <cit.> which is not targeted to a specific domain. Pre-training as for large foundation models like the original CLIP <cit.> typically requires substantial computational resources due to the use of billions of images. This contradicts the objective of resource-constrained training for small models. He et al. <cit.> observed that pre-training can be shortened significantly by using a feature-based loss. For our purpose of training 1-to-1 replacements of CLIP vision encoders, this step has further advantages: by aligning the embeddings of teacher and student, we can mitigate phenomena like the modality gap <cit.> where corresponding output vectors are located in different areas of the embedding space. Subsequently, we fine-tune the pre-trained models towards the target domain of interest with the same loss. The only difference between pre-training and fine-tuning is the training data. 














 §.§ Data Diversification of Synthetic Images
 
 In the context of zero-shot learning for image classification, synthetic data generation is based on the class names. However, it has been observed that using only the names to generate images using diffusion models leads to suboptimal performance <cit.>. This is primarily due to the lack of diversity in the generated images as well as class ambiguity <cit.>. One alternative is to utilize captions from existing real datasets for image generation. This approach is not entirely consistent with a zero-shot setting, as there may not be available captions for all classes. To address this challenge, recent approaches have turned to leveraging large language models (LLMs) to enhance diversity in the prompts. In addition to class names, LLMs are guided by additional inputs for diversification, such as information from a concept bank <cit.> or specific requirements related to contextual and style diversification <cit.>. By incorporating these additional sources of guidance, the generated synthetic data becomes more diverse and aligned with the desired objectives of the target setting. Using the approach from Yu et al. <cit.>, we focus on contextual dimensions to achieve diversification. These dimensions are attributes that describe the context of the image such as the background, camera angle, object position, presentation style, and superclasses, all of which are tuned specifically for the target dataset. In contrast to Yu et al. <cit.>, we do not prompt the LLM for each caption separately, but ask for different options for each contextual dimension. This reduces the risk of obtaining similar captions. The final prompt used for the text-to-image model is a comma-separated list of options for these contextual dimensions. Instead of using all possible combinations of options, which would result in a strongly growing number of images given more options, we perform combinatorial testing <cit.>.



 §.§ Loss Selection
 Knowledge distillation involves combining a training loss ℒ_training with a distillation loss ℒ_distillation <cit.>. The training loss ℒ_training takes into account the image and ground truth labels or captions, while the distillation loss ℒ_distillation is used to align the teacher and student models. Commonly, the overall loss is selected as ℒ_overall = ℒ_training + λ ℒ_distillation, where λ is a scaling parameter <cit.>. One might assume that using the CLIP loss for ℒ_training would be the most direct approach, as it was used to train the teacher model. However, He et al. observed that pre-training can be substantially sped up by solely employing a feature-based distillation loss without a supervised training loss. In contrast to the CLIP loss, which aims at aligning the text and image embedding of image-caption pairs, the student directly learns from the image features of the teacher without considering the text. More precisely, for every optimization step, we sample a batch of N images {t_i,i_i}_i=1,...,N as input. Denote the normalized image embedding corresponding to the i-th image by I_i^S for the student and I_i^T for the teacher. Based on this, the ℒ^2 feature distillation loss
ℒ_2^feature =  ∑_i=1^NI_i^S - I_i^T _2
is optimized. For fine-tuning, the CLIP loss function remains the commonly used approach. <cit.>. Yet, it was initially designed to pre-train both the text and image encoder on large-scale vision-language datasets where each image has a distinct caption, for most image classification datasets, each image only possesses a class label instead of a diverse prompts. In this case, we employ the zero-shot captions "a photo of {class name} which is a type of {superclass}", which were originally introduced for the zero-shot inference of the original CLIP model <cit.>, as T_i. "Superclass" refers to a general description of the object that can be encountered such as pets, food, cars or similarly. By using these class-specific prompts, several images in a batch may share the same caption which conflicts the goal of decreasing the similarity of images embeddings to the text embeddings of not matching captions in the CLIP loss. An alternative to the CLIP loss is given by the multi-positive contrastive loss introduced in StableRep <cit.>. To adapt the multi-positive (MP) loss to our setting, we replace the anchor sample by the embedding of a class-specific zero-shot prompt. Denote by Z_k the normalized embedding of the zero-shot prompt for class k. The contrastive distribution is given by
q_k =exp(⟨I_i,Z_k ⟩/τ)∑_j=1^Mexp(⟨I_i,Z_j ⟩/τ)
and the ground-truth categorical distribution is
p_k = 1_l(I_i)=k∑_j=1^M1_l(I_i)=j.
Given M classes, the MP loss is computed as
ℒ_MP = - ∑_k=1^M p_k log q_k.
We compare contrastive loss, MP loss, feature loss, as well as combinations thereof, on both synthetic and real data in our experiments. We observe that in the zero-shot settings, pure feature distillation is both the most efficient choice for pre-training and the most robust loss for fine-tuning.




§ EXPERIMENTS


In this section, we present the results of the models trained based on our framework and thereby explain how feature distillation enables zero-shot distillation with synthetic data. After a description of the setup we will compare our models to existing baselines which we outperform consistently. In the ablation studies, we uncover that using contrastive losses leads to students that exploit spurious features in the data, generalize poorly between synthetic and real data alongside being less robust to common corruptions of input images. Overall, this indicates that contrastive losses are a potential cause why efficient zero-shot distillation from synthetic has been an unresolved challenge.


 §.§ Setup
 
Datasets and Hyperparameters As introduced in Section <ref>, we perform feature-based pre-training on a large-scale dataset consisting of natural images for various domains. For this purpose, we select DataComp medium <cit.> and train for a single epoch. Originally, the dataset consists of 123 million images but at the time we conducted out experiments only 86% of the image URLs were still active. For fine-tuning, we target the Oxford Pets <cit.>, Oxford Flowers <cit.>, Food-101 <cit.> and Stanford Cars <cit.> to evaluate or models domain-specific datasets. These datasets are only used for testing while the actual training datasets used for training are synthetically generated based on the classes. Using the diversification strategy discussed in Section <ref>, we select a different set of five contextual dimensions and corresponding weights in the prompts to the diffusion model. More details on this selection are given in the supplementary material. For the pets, flowers and cars dataset we generate 15 options per contextual dimension while for the food dataset we use 30 as the target dataset is larger as well. This results in 265 and 1011 images per class, respectively. As the selection of options for the contextual dimensions and superclasses are relatively simple, we can use a smaller language model Llama-2 7B fine-tuned for chats <cit.> and still obtain sufficiently diverse prompts. For the generation of the images, we utilize a LCM LoRA <cit.> of Stable Diffusion XL <cit.> with a guidance scale of 0.5 and prompt weighting. As in the original CLIP paper <cit.> random square crops of the resized images are the only data augmentation used during training. 
For both pre-training and fine-tuning we use the same hyperparameters. We train using a batch size of 256 and a constant learning rate of 5× 10^-4 for the AdamW optimizer <cit.>. All other hyperparameters were kept consistent with the CLIP training methodology <cit.>. One epoch of pre-training corresponds to 4.3×10^5 optimization steps. For fine-tuning, we perform 96 optimization epochs for all models. Even on the largest synthetic dataset this equals less than 9% of the update steps done during pre-training.


Student and Teacher Architectures As teacher model, we employ a ViT-B/32 <cit.> CLIP vision encoder that has been trained on DataComp-XL, a dataset consisting of 12.8 billion image-text pairs from Common Crawl <cit.>. The corresponding text encoder follows the same architecture as described in the original CLIP paper, with 63 million parameters <cit.> and an embedding dimension of 512. For our student models, we utilize two different types of architectures: EfficientNets <cit.>, which are based on convolutional neural networks, and TinyViTs <cit.>, which are hybrid models combining convolutions and transformers. For our final results, we respectively select three models in the 5, 10, and 20 million parameter range from each architecture type. We only report intermediate results on the TinyViT with 11 million parameters. All models are systematically scaled down to improve inference speed and reduce the number of parameters while still maintaining a high capacity for representation learning. To align the output of the vision encoder with the embedding dimension of the teacher model, we apply a single linear projection head. This ensures consistent dimensionality of the embedding space between the teacher and student models. 




 §.§ Zero-Shot Performance and Comparison to Baselines
 
In this section, we report the zero-shot classification accuracy of our models based on the TinyViT-11M architecture and compare them to existing benchmarks. The results are shown in Table <ref>. The first reference for the performance is the teacher itself which is trained on DataComp-XL as well as the same model trained on DataComp-medium. The teacher model achieves zero-shot accuracies of over 80% on the pets, cars and food datasets as well as over 70% on the flowers dataset. Additionally, we report the performance of four TinyCLIP models that have been trained on LAION-400M <cit.> or YFCC-15M <cit.> datasets. These TinyCLIP models have undergone extensive training on large-scale datasets for multiple epochs, which differs from our approach. Specifically, the smallest TinyCLIP model has been exposed to six times as many images as our models, while the largest models have encountered over 120 times as many samples. The TinyCLIP models exhibit a comparable size to our models in terms of the number of parameters, even when considering the frozen text encoder which is not required for zero-shot classification. The largest TinyCLIP model has 40% fewer parameters than the ViT-B/32 CLIP model and achieves its performance up to a margin of 9%. The smallest TinyCLIP model features the same number of trainable parameters but has a gap of over 75% to  ViT-B/32 CLIP model on the cars dataset. At the time of conducting our experiments, we were unable to compare our results with MobileCLIP <cit.> and CLIP-KD <cit.> as these models are not publicly available.

We report three types of models from our framework: trained from scratch, pre-trained and finetuned. For reference, we include models that were fine-tuned on the real datasets as well. It is important to note that the reported accuracies are not zero-shot for these two cases. First, we observe that training from scratch or finetuning on synthetic data using the CLIP loss results in substantially worse performance. We assess this behavior in detail in the next section. Based on this observation, we base our framework solely on feature distillation. We find that the resulting models outperform even the largest TinyCLIP models on three of the four datasets despite having 88% less trainable parameters. Moreover, they achieve comparable performance to the teacher models with a margin of 5% on three of the four datasets. Similarly, our model outperforms the largest TinyCLIP model in these cases. The larger performance gap on the Food dataset can likely be attributed to its larger test set size, which is also evident in the TinyCLIP models. When comparing to ViT-B/32 trained on DataComp-medium, which has been trained on a comparable number of images, even our purely pre-trained models demonstrate significantly superior performance. Similarly, when pre-training a student using the CLIP loss on DataComp-medium for one epoch, the resulting zero-shot accuracies are far worse compared to the feature-based loss. This validates the data-efficiency of feature distillation for pre-training <cit.>. We report results for top-5 validation accuracy in the supplementary material. 





 §.§ Ablations

Based on the observation that models trained solely using feature distillation outperform those incorporating label-based losses when fine-tuning on synthetic data, we hypothesize that the utilization of labels during fine-tuning leads the model to learn spurious features, as well as domain-specific characteristics that distinguish between synthetic and natural images. Previously, it was observed that the using the CLIP loss hinders the class-incremental generalization of students distilled on real images <cit.>. We examine whether this observation transfers to the synthetic to real domain shift and potential spurious features. We highlight that the class-incremental setting is different from ours, as we use a fixed set of classes. To validate our hypothesis, we conducted three experiments on the pets dataset, deliberately introducing dedicated spurious features into synthetic and real images. Additionally, we evaluate students that we fine-tuned on either real or synthetic images on test sets from the respective other domain. Like that, we aim to provide further insights into the impact of label-based losses on the transferability of the models between synthetic and natural images. Subsequently, we employ larger and smaller students with two different architecture types to assess how the performance scales with the number of trainable parameters. Additionally, we report the performance of models trained from scratch.


Natural Images with Spurious Features. To investigate the impact of spurious features in the natural image domain, we add class-specific colored shapes to the images in the pets dataset. These shapes were added to each image in the training split, and examples can be seen in Figure <ref>. We then proceeded to fine-tune the pre-trained student models using the same hyperparameters employed during pre-training using these images. The test accuracies of the resulting models are shown in Table <ref>.  We observe a slight decrease in performance on the test set without spurious features when the pre-trained students were fine-tuned with contrastive losses. This suggests that the students did not acquire any additional class-specific features during fine-tuning. However, when evaluating these fine-tuned students on a test set of real images where the colored shapes were mixed between classes, we observe a significant degradation in performance. In contrast, the students trained with the ℒ_2 loss achieves accuracies comparable to the dataset without spurious features on both test sets. These findings highlight the robustness of the feature loss in mitigating the negative impact of spurious features in the natural image domain.

Synthetic Images with Spurious Features. To investigate whether the observed behavior on real images could be replicated using synthetic ones, we generated a synthetic dataset incorporating dedicated spurious features. Specifically, we sample images where pets are positioned against a solid-colored background, with each class assigned a distinct color. The results shown in Table <ref> indicate that the performance of students trained with contrastive losses deteriorates when confronted with the presence of these spurious features. Figure <ref> showcases instances of misclassifications. In contrast, the student trained with ℒ_2 loss exhibited a test accuracy of 84.4%, which is only 5% lower than the accuracy of the teacher despite the domain gap between real and synthetic images, as well as the presence of spurious features.






Generalization Between Synthetic and Real Images. In order to evaluate the ability of our models to generalize between synthetic and real images, we examine the performance of models that were fine-tuned on real training datasets when tested on independently generated synthetic datasets using the same methodology as the synthetic training sets. The results are presented in Table <ref>. The models fine-tuned with contrastive losses exhibited lower validation accuracy compared to the students trained with feature distillation. In contrast, for the models fine-tuned on synthetic data the reverse is true. Using the CLIP loss results in higher accuracy on the synthetic testset in comparison to feature distillation. This discrepancy suggests that the former models primarily learned domain-specific features of natural or synthetic images, thereby limiting their generalization capabilities between the two types. 






Robustness to Common Corruptions. 
In order to evaluate the robustness of the models' learned representations against image perturbations, we conducted a comprehensive benchmark study. We assessed the performance of the classifiers on 15 common corruptions <cit.> at a fixed severity level of three, focusing on the pets dataset. The corresponding results are presented in Table <ref>. Our observations revealed that the students trained using the ℒ_2 feature loss demonstrate higher robustness to corruptions, regardless of whether they were trained on real or synthetic data. The distinction to the models fine-tuned with contrastive losses is particularly prominent  when training on synthetic data, where the models fine-tuned with the CLIP loss even perform worse than the purely pre-trained model. These findings provide further evidence that contrastive losses lead to learning spurious and datatype-specific features, making the models less robust to disturbances caused by common corruptions. When evaluating on synthetic test data with common corruptions, we observe that the performance drops are lower and the characteristics of the synthetic data are less disturbed by corruptions.




Student Model Size. 
Alongside the results of a TinyViT-11M in Table <ref>, we report the zero-shot performance of five additional students after pre-training and fine-tuning using feature distillation in Figure <ref>. As expected, there is a general trend of improved performance with increasing model size. Yet, we observe that the effectiveness of zero-shot fine-tuning is more pronounced for smaller models compared to larger ones: after fine-tuning, the difference in performance between the largest and smallest models is around 10% to 15%, while after purely pre-training it is as high as 30%. These findings highlight that zero-shot distillation is  particularly effective for smaller models since it allows to adapt them to the target domain, without requiring real in-domain data.



Training From Scratch. We investigate the accuracy of models trained from scratch, both on real and synthetic images in <ref>. On real data, using a sum of the ℒ_2 and a contrastive loss indeed results in the best performance as observed in CLIP-KD <cit.>. The low accuracy of the feature distilled student on the flowers dataset can be attributed to the small training set with only 10 images per class. On synthetic training data, the ℒ_2 loss consistently performs best while the CLIP loss or the MP loss results are substantially worse. Combing the ℒ_2 with one of the contrastive losses does not yield any performance gains in contrast to purely feature-based distillation. This is in line with the behavior observed during fine-tuning in <ref>.









§ CONCLUSION

In this work, we introduced a framework for zero-shot distillation of small CLIP image encoders based on synthetic images. We make the surprising observation that contrastive losses are a potentially detrimental factor for generalization capabilities of models between synthetic and real data, due to the exploitation of spurious features. By employing a pure image feature-based distillation loss, we successfully mitigate this limitation. As a result, we are able to train models that surpass the current state-of-the-art for zero-shot CLIP distillation, while featuring fewer parameters and  not using labeled target domain images.



Limitations and Future Work. The presented results are based on a ViT-B/32 teacher; future work could explore the benefit of larger teachers for constant-sized students. Furthermore, the potential of our small-scale image encoders beyond zero-shot settings could be explored, for instance in architectures that use CLIP image encoders such as BLIP-2 <cit.> or LLava <cit.>. Moreover, the current framework is limited to classification tasks. To broaden its applicability, future research could extend the framework to encompass other computer vision tasks such as object detection or image segmentation. 
Acknowledgements. We would like to thank Nicole Finnie for helpful discussion on pre-training CLIP models. We also thank the European Laboratory for Learning and Intelligent Systems (ELLIS) for supporting
Niclas Popp.










splncs04









