Training-Free Unsupervised Prompt for Vision-Language Models
Sifan Long Linbin Wang Zhen Zhao Zichang Tan Yiming Wu Shengsheng Wang Jingdong Wang








Training-Free Unsupervised Prompt for Vision-Language Models
    Sifan Long1,2
Equal contribution. The work was done during an internship at Baidu.
    Linbin Wang1    Zhen Zhao3    Zichang Tan2 

    Yiming Wu4
    Shengsheng Wang1Corresponding authors.
    Jingdong Wang2 

1Jilin University
2Baidu VIS
3University of Sydney

4University of Hong Kong

{longsf22, wanglb23}@mails.jlu.edu.cn  zhen.zhao@sydney.edu.au
  yimingwu0@gmail.com

wss@jlu.edu.cn  {tanzichang, wangjingdong}@baidu.com



    
==================================================================================================================================================================================================================================================================================================================================================================================================================================================



Prompt learning has become the most effective paradigm for adapting large pre-trained vision-language models to downstream tasks.
Recently, unsupervised prompt tuning methods, such as UPL and POUF, directly leverage pseudo-labels as supervisory information to fine-tune additional adaptation modules on unlabeled data.
However, inaccurate pseudo-labels easily misguide the tuning process and result in poor representation capabilities.
In light of this, we propose Training-Free Unsupervised Prompts (TFUP), which maximally preserves the inherent representation capabilities and enhances them with a residual connection to similarity-based prediction probabilities in a training-free and labeling-free manner.
Specifically, we integrate both instance confidence and prototype scores to select representative samples, which are used to customize a reliable Feature Cache Model (FCM) for training-free inference.
Then, we design a Multi-level Similarity Measure (MSM) that considers both feature-level and semantic-level similarities to calculate the distance between each test image and the cached sample as the weight of the corresponding cached label to generate similarity-based prediction probabilities.
In this way, TFUP achieves surprising performance, even surpassing the training-base method on multiple classification datasets.
Based on our TFUP, we propose a training-based approach (TFUP-T) to further boost the adaptation performance. 
In addition to the standard cross-entropy loss, TFUP-T adopts an additional marginal distribution entropy loss to constrain the model from a global perspective.
Our TFUP-T achieves new state-of-the-art classification performance compared to unsupervised and few-shot adaptation approaches on multiple benchmarks.
In particular, TFUP-T improves the classification accuracy of POUF by 3.3% on the most challenging Domain-Net dataset [Code and logs: <https://github.com/wlb12345/TFUP>]. 













§ INTRODUCTION



Foundational large-scale vision-language models (VLMs), such as CLIP <cit.> and ALIGN <cit.>, have demonstrated impressive representation and generalization on various downstream tasks by using the contrastive learning objective on hundreds of millions of text and image pairs.
Due to a potential shift between the pre-training and the specific task <cit.>, fine-tuning is a common method to bridge this gap, which leverages labeled data on downstream tasks to fine-tune all parameters of the pre-trained model <cit.>.
However, it requires significant amounts of labeled data and is computationally expensive, even leading to over-fitting <cit.>.
To this end, prompt learning, as a parameter-efficient fine-tuning paradigm, recently attracted increasing attention from the research community.
As a representative work, CLIP <cit.> directly utilizes hand-crafted prompts to achieve impressive zero-shot classification performance in Fig. <ref> (a).
But the amount of manual prior knowledge required to design appropriate prompts for each specific task is intolerable.
Inspired by tuning studies in large language models (LLMs)  <cit.>, latter studies like CoOp <cit.> and CLIP-Adapter  <cit.>, train learnable prompts on labeled data to alleviate such reliance on hard-prompt designs.


Though few-shot prompt methods gain significant improvements, they still require artificial prior knowledge to label downstream data and rely on manual annotation quality, which may limit the scalability of the original model. 

To this end, recent unsupervised prompt tuning frameworks have been introduced to eliminate the need for data annotations and enhance the efficiency of adapting VLMs for various downstream tasks <cit.>. 
As shown in Fig. <ref> (b), these methods tend to fine-tune models or learnable prompts directly on unlabeled data. 
UPL <cit.> selects the top-K confidence samples per class to tune the whole model using pseudo labels generated by the pre-trained vision-language model. 
POUF <cit.> treats the representation of class-specific text prompts as class prototypes and further aligns these prototypes with target image features in the latent space.
However, these methods often undervalue the pre-trained CLIP generalization ability, which is crucial for achieving robust performance across diverse downstream datasets. 
By focusing on optimizing the CLIP performance on a specific set, they may overlook the importance of learning general features that can be applied to new, unseen data <cit.>. 
Besides, these methods rely heavily on the pseudo labels to tune additional adaptation modules, which will inevitably introduce the confirmation bias <cit.>. 
Inaccurate pseudo-labels can misguide the tuning process and result in poor generalization capabilities.
Consequently, our primary goal is to maximize the retention of the pre-trained VLMs' capabilities while adapting them to downstream tasks with minimal costs.

Motivated by above observed limitations, we propose Training-Free Unsupervised Prompt (TFUP), which maximally preserves the inherent representation capabilities and enhances them with a residual connection to similarity-based prediction probabilities in a training-free and labeling-free manner.
Our TFUP generates similarity-base prediction probabilities by customizing a Feature Cache Model (FCM) and designing a Multi-level Similarity Measure (MSM). 
As shown in Fig. <ref> (c), we first extract image features of unlabeled training images by CLIP's visual encoder and then calculate the cosine similarity between image features and text features to obtain the predicted probability of the current image.
In FCM, we select top-K samples as high-confidence samples for each category. Nonetheless, it is inevitable that these high-confidence images contain noisy information, such as complex backgrounds. In light of this, we further propose a prototype filter, which introduces an attention mechanism to filter out the representative samples from the constructed 
high-confidence sample set. Consequently, we can create a cache model using the features of the representative sample and the corresponding one-hot labels as key-value pairs.

Based on the constructed FCM, we calculate the distance between the test image and cached samples as the weights of the corresponding labels. Then, we combine different sample labels into a similarity-based prediction probability according to the weights.
In existing unsupervised tuning studies, the distance measure method only considers the similarity of the overall image information and easily introduces background noise. For example, two pictures with the same background but different foregrounds are judged to be of the same category. Differently, we design a new measure method called Multi-level Similarity Measure (MSM), which considers both feature-level and semantic-level similarities. 
Specifically, we not only calculate the cosine similarity between image features as feature-level similarity but also calculate the KL divergence between image prediction probabilities as semantic-level similarity. Ultimately, we leverage a hadamard product to combine feature and semantic similarities.
As shown in Fig. <ref>, through this non-parametric and non-training approach, our TFUP demonstrates extremely excellent efficiency and achieves promising performance, even surpassing the training-base unsupervised prompt learning methods <cit.> on the Domain-Net <cit.> and Office-Home <cit.>.



On top of our effective training-free strategy, we propose a training-base approach (TFUP-T) to further boost VLMs' unsupervised adaptation.
Following the standard Parameter-Efficient Fine-tuning (PEFT) methods <cit.>, we also append the image and text adapters to the image and text encoders, respectively.
In addition, we adopt a residual connection <cit.> to combine pre-trained features with the fine-tuned features to preserve the CLIP's generalization ability better.
By leveraging our TFUP to produce supervisory information on downstream unlabeled datasets, we can effectively tune the adaptors to achieve higher performance.
Different from existing pseudo-labeling strategies in unsupervised tuning, which mainly focus on instance-level predictions and may introduce accumulated errors <cit.>, we further introduce a marginal distribution entropy loss to constrain the model from a global perspective.
As shown in Fig. <ref>, our TFUP-T achieves state-of-the-art classification performance among multiple benchmarks. In particular, TFUP-T not only achieves an average accuracy improvement of 3.3% compared to the SOTA POUF of unsupervised methods, but also obtains improvement by 1.2% compared to KgCoOp of few-shot approaches on Domain-Net <cit.>.
Our contributions are summarized as follows,

    
  * We propose the first training-free approach (TFUP) for unsupervised prompt, which maximally preserves the inherent representation capabilities and enhances them with a residual connection to similarity-based prediction probabilities in a training-free and labeling-free manner.
    In particular, we generate similarity-base prediction probabilities by the Feature Cache Model and Multi-level Similarity Measure. 
    
  * Based on TFUP, we propose a training-based approach (TFUP-T) to further boost performance. Considering the lack of labeled downstream data, we simultaneously optimize individual and global predictions on unlabeled data via pseudo-label cross-entropy loss and marginal distribution entropy loss.
    
  * Through extensive empirical analysis, our TFUP outperforms the original CLIP on all classification datasets by a large margin. It achieves promising performance without any labeled data or training, even surpassing the training-base method on multiple classification datasets. In addition, the training-base TFUP-T obtains the new state-of-the-art compared with both unsupervised and few-shot prompt learning methods.








§ RELATED WORK




 §.§ Vision-Language Models

Large-scale vision-language models have exhibited remarkable representation capabilities on various downstream vision tasks <cit.>.
By employing the contrastive learning objective to train large-scale multi-level models on 400 million image-text pairs, CLIP <cit.> achieves impressive results in zero-shot visual recognition tasks. 
The success of CLIP inspired the development of several subsequent variants.
FLIP <cit.> randomly masks out and removes a large portion of image patches during training to increase training speed without sacrificing accuracy. 
DeCLIP <cit.> uses more expansive and scalable supervision to learn visual representations. 

To reduce the effect of noise in web-crawled data, SoftCLIP  <cit.> provides a soft cross-modal alignment by introducing a softened target, which is produced from the fine-grained intra-modal self-similarity.
While CLIP and its variant models achieve great results in downstream datasets, it is important to reactivate specific capabilities. 
In this work, we focus on parameter-efficient fine-tuning of large-scale vision-language models to adapt to downstream tasks.




 §.§ Prompt Learning in VLMs

Prompt learning has become the most popular paradigm in the natural language processing community for adapting LLMs to downstream tasks <cit.>.
Motivated by these studies, CoOp <cit.> is the first approach which applies prompt learning to VLMs adaptation in the computer vision community.
To further enhance generalization capabilities, CoCoOp <cit.> generates image-conditional context prompts for each image and incorporates them into text prompts for prompt tuning.
In addition, KgCoOp <cit.> reduces the forgetting of general knowledge by minimizing the difference between text embeddings generated by learned prompts and hand-crafted prompts.
Unlike the approaches mentioned above, which train learnable prompts on labeled data of specific tasks. 
The unsupervised prompt tuning framework proposes to fine-tune models or prompts directly on unlabeled data to efficiently adapt VLMs for downstream tasks.
UPL <cit.> selects the top-K confidence samples per class to train itself using pseudo labels generated by a pre-trained vision-language model.
POUF <cit.> treats the representation of class-specific text prompts as class prototypes aligned with target image features in the latent space.
However, inaccurate pseudo-labels easily misguide the tuning process and result in poor generalization capabilities. We propose Training-Free Unsupervised Prompt (TFUP), which maximally preserves the inherent representation capabilities and enhances them while adapting VLMs to downstream tasks in a training-free and labeling-free manner.







 §.§ Pseudo Labeling

Pseudo labeling is originally proposed for semi-supervised learning and gains popularity in other domains including NLP <cit.>, speech recognition <cit.>,
image classification <cit.>,
semantic segmentation <cit.>, object detection <cit.>, domain adaptation <cit.>, to name a few.
The main idea is to select the class with the maximum predicted probability as pseudo labels to fine-tune the model together with the true labels.
Pseudo-Label <cit.> is the first pseudo-label method proposed in semi-supervised learning, which selects the category with the maximum predicted probability and converts it into a hard label.
<cit.> uses a confidence-based strategy to further filter out unlabeled samples.
Recently, as a representative work in semi-supervision learning, 
FixMatch <cit.> continues to employ a confidence-based strategy, retaining pseudo-labels with high-confidence predictions.
Inspired by this, our proposed TFUP generates pseudo labels for downstream data based on the original and similarity-base prediction probability. 
Benefiting from the powerful generalization performance of VLMs and  high-confidence feature cache model, 
we can effectively select the more convincing pseudo-labels.
However, these pseudo-labels methods focus mainly on individual prediction can inevitably introduce prediction bias
 <cit.>. 
Inspired by recent studies in mutual-information maximization <cit.>,  we further introduce a marginal distribution entropy loss to constrain the model from a global perspective.







§ METHOD

Fig. <ref> presents an overview of our proposed TFUP. 
In this section, we first discuss a representative vision-language model, CLIP, which utilizes hand-crafted prompts in a zero-shot manner for downstream tasks in Sec. <ref>.
Subsequently, we introduce our proposed 
Training-free Unsupervised Prompt (TFUP)  which maximally preserves the representational
capabilities of pre-trained VLMs while adapting them to downstream tasks in a training-free and labeling-free manner in Sec. <ref>.
To further improve the performance, we then introduce an unsupervised prompt tuning method (TFUP-T) which simultaneously optimize individual and global predictions on unlabeled data via pseudo-label cross-entropy loss and marginal distribution entropy loss in Sec. <ref>. 



 §.§ Preliminaries of CLIP




  
Contrastive language-image pre-training CLIP is trained on 400 million image-text pairs with a language-image contrastive loss function. Specifically, the structure of CLIP consists of two components: visual encoder, denoted as F(·), for converting the input images into visual features and text encoder, denoted as G(·), for transforming input texts into text representations.
Then image and text features are projected into the same embedding space through joint training. 
After pre-training on a large dataset, it has excellent classification performance in zero-shot scenarios, demonstrating the vision-language model's excellent understanding of open-set concepts.
Consider an image classification task that 
is defined as classifying a given test image x_test into one of C categories.
Since the text description used in pre-training is different from the labels of the downstream recognition tasks,
CLIP places all category names into the
“”  token of a pre-defined textual template such as “”. We denote the text prompts obtained by converting the original labels as P_text.
Then, we obtain d-dimensional visual features f_test∈ℝ^1 × d and text features 𝐅_ text∈ℝ^C × d by

    f_test   =F( x_test), 
    𝐅_ text   =G( P_text),

where f_test and 𝐅_ text are L2-normalized visual and text features, respectively.
We can then obtain the classification logits∈ℝ^1 × C by calculating the cosine similarity of visual and textual features,

    logits=softmax(f_test𝐅_ text^⊤),

where logits denote the prediction probabilities for the C categories. We can easily identify the prediction ŷ = cargmax  (logits).







 §.§ Training-Free Unsupervised Prompt


We propose a training-free and labeling-free method called TFUP, which performs comparably or even better than 
training-base unsupervised methods
<cit.>.
To achieve this goal, we construct a new Feature Cache Model (FCM) which 
stores the features of the representative sample and the corresponding one-hot labels as key-value pairs.
Based on the constructed cache model, we calculate the similarity between test image and representative samples as the weights of the corresponding labels. In addition, we design a new Multi-level Similarity Measure (MSM) method which considers both feature-level and semantic-level similarity between images.



  
Cache model construction.
Given the pre-trained CLIP <cit.> model, we aim to leverage the unlabeled training set, denoted as M, for unsupervised classification. 
For each training image, denoted as x_i,
we utilize the CLIP's visual encoder to extract its d-dimensional visual feature, denoted as f_ i = F(x_i). 
Then, we can use Eq. (<ref>) to generate the prediction  probability, denoted as logits_i. Finally, we view CLIP outputs the maximum one as the prediction and the corresponding index as the pseudo label L_i= One Hot(cargmax  (logits_i)).
For all training samples, we denote their visual features and corresponding pseudo-labels as 𝐅_ train∈ℝ^M × d and 𝐋_ train∈ℝ^M × C.
To create the key-value cache, we view visual features 𝐅_ train as keys and the corresponding pseudo-labels 𝐋_ train as values.
Therefore, the unsupervised dataset M is transformed into a key-value cache database 𝐒_ train=⟨𝐅_ train, 𝐋_ train⟩.

For creating a standard Feature Cache Model (FCM), we first employ a confidence filter to filter out high-confidence samples and their corresponding pseudo-labels from the unsupervised dataset 𝐒_ train=⟨𝐅_ train, 𝐋_ train⟩. 
Specifically, we select top-K most  confident samples for each class, instead of keeping all samples with confidence scores higher than a pre-defined threshold, to preserve a balanced distribution of pseudo-labeled data. Then, we denote the high confidence visual features and corresponding pseudo-labels as 𝐅_ confi∈ℝ^K × d and 𝐋_ confi∈ℝ^K × C.
Consequently, we obtain the high confidence pseudo-label dataset 𝐒_ confi=⟨𝐅_ confi, 𝐋_ confi⟩.

However, these high-confidence images are not necessarily representative samples.
To further refine the confident pseudo-label data into representative samples, we propose a prototype filter which calculates the cosine similarity between
each confidence sample f_i∈𝐅_ confi and other confidence samples by

    S_i=∑_j=1^Kcos(f_i, f_j),

where S_i denotes the prototype score of the i-th image. 
Higher value of S_i means higher potential to be category prototypes. 
We select the N highest scoring samples for each category as prototype samples.
Then, we obtain the prototype visual features and corresponding labels as 𝐅_ proto∈ℝ^N × d and 𝐋_ proto∈ℝ^N × C.
Finally, we denote the prototype pseudo-label dataset as 𝐒_ proto=⟨𝐅_ proto, 𝐋_ proto⟩.



  
Training-free inference.
Based on the constructed feature cache model, we calculate the similarity
between test image and prototype samples as the weights of the
corresponding labels. We then combine different sample labels into a similarity-base prediction probability according to the weights.
The current similarity measure only considers the degree of similarity at the feature level.
Differently, we propose a Multi-level Similarity Measure (MSM) which considers both feature-level and semantic-level similarity to measure the similarity between test samples and prototype samples.
Specifically, for each test image, denoted as x_test,
we utilize the CLIP's visual encoder to extract its d-dimensional visual feature, denote as f_ test = F(x_ test).
Then, we calculate the multi-level similarity between test image and prototype samples as the weights of the corresponding labels.
The MSM consists of Feature Similarity Measure (FSM) and Semantic Similarity Measure (SSM).
For the FSM, we calculate the cosine similarity between
the test image features f_test∈ℝ^1 × d and the prototype features  𝐅_ proto∈ℝ^N × d by

    W_cont=f_test𝐅_ proto^⊤,

where W_cont represents the feature similarity score.
In addition, the feature-level similarity measure only considers the similarity of the overall information of the image and ignores the consistency of the sample categories.
To this end, we introduce SSM which calculates the KL-divergence between
the test image prediction probabilities, denoted as logits_test∈ℝ^1 × C, and the prototype prediction probabilities, denoted as logits_ proto∈ℝ^N × C, by

    logits_test   =softmax(f_test𝐅_ text^⊤), 
    logits_ proto   =softmax(𝐅_ proto𝐅_ text^⊤),
    W_sem   =1 - softmax(KL(logits_test, logits_ proto)),

where W_sem represents the semantic similarity score and 𝐅_ text denotes textual features.
KL computes the KL-divergence, i.e., KL(P,Q)=∑_i P_i logP_i/Q_i.
To emphasize the criticality of the two measures, we combine CSM and SSM by

    W_fsm= W_cont∘W_sem ,

where ∘ denotes hadamard product and
W_fsm represents the ultimate multi-level similarity score.
Then, we combine different prototype sample pseudo-labels 𝐋_ proto into a similarity-base prediction probability according to the multi-level similarity scores W_fsm by

    logits_ sim=W_fsm·𝐋_ proto,

where · denotes matmul product and logits_ sim denotes the predicted probability based on feature cache model. 
After that, we obtain the final prediction probability of the test image, denote as logits_ all, by adding the similarity-base prediction probability logits_ sim to the original prediction probability logits_test,

    logits_all = logits_test + logits_ sim,

where logits_all presents the ultimate prediction probability. 




 §.§ Training-Base Unsupervised Prompt Tuning
 




  
Training-base adapters
The performance of TFUP can be further boosted by parameter efficient fine-tuning. Following the standard Parameter-Efficient Fine-tuning (PEFT) methods <cit.>, we attach image and text adapters to the image and text encoders, denoted by T_x(·) and T_t(·), respectively. Each adapter consists of two layers of linear transformations.
Referring to ReZero <cit.>, we further employ two constant values α and β as residual ratio to adjust the proportion of downstream knowledge and model inherent knowledge to enhance the robustness of the model and prevent over-fitting.
Mathematically, the new knowledge captured via fine-tuning is added with the original features via residual connections,

    f_x^⋆   = αT_x(f_x) + (1-α) f_x, 
    𝐅_ text^⋆   = βT_t(𝐅_ text) + (1 - β) 𝐅_ text,

where f_x^⋆ and 𝐅_ text^⋆ denote the merged image and text features, respectively.
Then, we can use Eq. (<ref>) to generate the prediction probabilities of the test image, denote as Logits_test∈ℝ^1 × C. 








  
Training-base supervision  In the paradigm of unsupervised prompt tuning, it is crucial to seek appropriate supervision to train adapters on unlabeled data. Thanks to the effectiveness of our training-free strategy, we first generate pseudo-labels on unlabeled data via our TFUP. Thus we can utilize Eq. (<ref>) to generate the prediction probabilities of the unlabeled instances, denote as Logits_all∈ℝ^1 × C. We then view the maximum output of the CLIP model as the prediction and the corresponding index as the pseudo label L_all= One Hot(cargmax  (logits_all)). After that, we train the adapters via a standard cross-entropy loss,

    ℒ_ce = 1(max(Logits_test) ≥θ) CE(L_all, Logits_test),

where  1 denotes retaining only the high-confidence predictions higher than pre-defined threshold θ. 
However, all these pseudo-label-based methods <cit.> focus solely on the instance-level constraint, ignoring the significance of the global prediction statistic on unlabeled data. Though high-confidence filtering can effectively select the more convincing pseudo-labels,  constraining only on individual prediction can inevitably introduce prediction bias <cit.>, considering the different learning difficulties of distinct classes, especially with a huge category space. To this end, inspired by recent studies in mutual-information maximization <cit.>,  we further introduce a marginal distribution entropy loss to constrain the model from a global perspective,

    ℒ_md = logC- [-∑_ch_c logh_c],

where logC is the maximum value of distribution entropy and h_c denotes the marginal distribution of the prediction probabilities of the test images on the class index c, c∈{0,1,...,C-1} .







§ EXPERIMENTS

In this section, we evaluate the performance of the training-free TFUP and the training-base TFUP-T on four public datasets including 
Domain-Net <cit.>, 
Office-Home <cit.>,
Office-31 <cit.>,
and VisDA-2017 <cit.>. 
We compare our methods with recent prompt learning methods including Clip (Zero-Shot) 
<cit.>, Tent <cit.>, UPL <cit.>, POUF <cit.>, CoCoOp <cit.>, and KgCoOp <cit.>.
Additionally, we provide extensive experimental results and further ablation studies. 



 §.§ Datasets

1) Domain-Net <cit.>  is the largest and the most challenging domain adaptation benchmark which contains approximately 600,000 images from 6 different domains including Clipart (clp), Infograph (inf), Painting
(pnt), Quickdraw (qdr), Real (rel) and Sketch (skt).
2) Office-Home <cit.>  refers to a difficult domain adaptation dataset which includes 15500 images in office and home extracted from 4 different domains: Art (A), Clip (C), Product
(P) and RealWorld (R). 
3) Office-31 <cit.>  is a standard dataset in visual transfer learning which contains 4,652 images with 31 classes from three domains: Amazon (A), Webcam (W), and DSLR (D).
4) VisDA-2017 <cit.>  is a dataset from the 2017 Vision Domain Adaptation Challenge, covering 12 categories and more than 280,000 images, including 152,397 synthetic images and 55,388 real images.












 §.§ Baselines

To verify the effectiveness of our approach, we compare it with plenty of advanced methods: (1) Zero-shot CLIP <cit.>, which applies hand-crafted prompts; (2) Tent <cit.> refers to using entropy minimization to tune the model before making predictions on downstream datasets; (3) UPL <cit.> selects the top-K confidence samples per class to train the soft prompts using pseudo labels generated by a pre-trained vision-language model. For a fair comparison, we do not use model ensemble as done in <cit.>;
(4) POUF <cit.> treats the representation of class-specific text prompts as class prototypes aligned with target image features in the latent space;
(5) CoCoOp <cit.> proposes an image-condition prompt learning method to generate specific text prompts for each image input;
(6) KgCoOp <cit.> reduces the difference between text features generated by learnable prompts and hand-crafted prompts to enhance the generalization ability of learnable prompts.
The reported results for the baseline models are obtained using the officially open-source code.



 §.§ Implementation Details

Our implementation is based on the open-source repository of POUF <cit.>. For all experiments, we follow the same unlabeled downstream dataset, visual and text encoder, data augmentation, learning rate schedule, and batch size. 
Differently, we set the hyper-parameter α = 0.2 and β = 0.5 in Eq. (<ref>) and Eq. (<ref>), which are analyzed in our experiments. 
Following previous pseudo-labeling strategy, we set the predefined threshold θ in Eq. (<ref>) to 0.95 as a constant value for all experiments.
The final performance reported below is the average of three runs with different random seeds. All experiments are conducted based on RTX 3060.




 §.§ Experimental Results




  
Results on Domain-Net <cit.>
Tab. <ref> reports the results comparing TFUP and TFUP-T with unsupervised prompt tuning and few-shot prompt learning methods across 6 domains
on the Domain-Net <cit.> datasets.
It is obvious that our TFUP achieves the new state-of-the-art (SOTA) performance without any training compared with previous unsupervised prompt tuning methods. Specifically, TFUP outperforms original CLIP with prompt engineering by 3.2%. 
In addition to prompt tuning, TFUP-T not only achieves
an average accuracy improvement of 3.3% compared to
the current SOTA POUF of unsupervised methods, but also obtains improvement by 1.2% compared to the SOTA KgCoOp of  few-shot approaches.
The above experiment demonstrates the efficiency and effectiveness of TFUP, as well as the superiority of the training-base TFUP-T.



  
Results on Office-Home <cit.>
Tab. <ref> illustrated the results comparing TFUP and TFUP-T with unsupervised prompt tuning and few-shot prompt learning methods across 4 domains
on the Office-Home <cit.> datasets.
Our TFUP achieves new SOTA performance without any training compared with unsupervised tuning methods and outperforms original CLIP with prompt engineering by 2.2%. 
For prompt tuning, TFUP-T boosts the performance of the POUF by 2.0% and demonstrates clear advantages over few-shot methods.


  
Results on Office-31 <cit.>
Tab. <ref> provides the results comparing TFUP and TFUP-T with unsupervised prompt tuning and few-shot prompt learning methods across 3 domains
on the Office-31 <cit.> datasets.
Our TFUP has significant advantages without any training compared with Tent and UPL. Although the accuracy of training-free TFUP is slightly lower than the trainable POUF, TFUP-T achieves new SOTA on all domains.
Due to the smaller amount of unsupervised data in the Office-31 dataset compared to the other datasets, the performance achieved by our method is limited.




  
Results on VisDA-2017 <cit.>
Tab. <ref> summarizes the results compare TFUP and TFUP-T with unsupervised prompt tuning and few-shot prompt learning methods on the VisDA-2017 <cit.> datasets.
Obviously, the training-free TFUP demonstrates competitive performance compared to the other unsupervised methods.
By efficient fine-tuning, TFUP-T further improves the performance of model, confirming the effectiveness and versatility of our methods.




 §.§ Ablation analysis



  
Effectiveness of each component.
To evaluate the effectiveness of various components, we conduct ablation experiments on Office-Home and  Domain-Net datasets, as reported in Tab. <ref>. 
We can clearly see that each component significantly improves the performance of the model.
For training-free TFUP, FCM + FSM increases the average accuracy of Office-Home and Domain-Net datasets by 2.2% and 3.2%, respectively. 
By efficient fine-tuning, ℒ_ce further improves the performance of the model.
Additionally, the pseudo-labeling strategy mainly focuses on rectifying individual predictions. 
We introduce a global prediction constraint ℒ_md, which increases the average performance of CLIP
from 57.7% to 63.5% on the most challenging Domain-Net.
It demonstrates the effectiveness of each component, and the superiority of  our TFUP and TFUP-T.
 









  
Different sample filter strategies.
As presented in Tab. <ref>, we compare several common sample filter strategies to assess the effectiveness of our approach. 
i) Confidence Filter Strategy. Select top-K high confidence samples by pseudo labels.
ii) Prototype Filter Strategy. Select top-K representative samples by prototype score.
iii) Double Filter Strategy. Combine the confidence filter and prototype filter strategy to obtain the representative samples.
Compared with the overall unlabeled data, both of the filter strategy designs could improve the final results by large margins. 
It demonstrates that the key factor of the filter strategy to achieve significant performance is reducing the introduction additional noisy samples.
In addition, we find that combining confidence and prototype filter can both improve the results compared with using either of them. It indicates that only selecting the high confidence 
samples is not enough, and prototype scores are needed to screen representative samples.









  
Different similarity measure strategies.
As summarized in Tab. <ref>,
we provide in-depth analysis about the similarity measure strategies.
i) Feature Similarity Measure (FSM) leverages cosine similarity to measure the similarity of the overall information of the images.
ii) Semantic Similarity Measure (SSM) leverages KL-divergence to measure the distance of the image prediction probabilities.
iii) Multi-level Similarity Measure (MSM) considers both feature-level and semantic-level similarities between images.
We observe that FSM and SSM improve the average accuracy by 1.4% and 1.9% respectively, and the combination of them improves the results by 2.2%. 
It demonstrates that the effectiveness of considering both feature-level and semantic-level similarities, which not only measures the degree of similarity of the overall image information, but also ensures the semantic consistency of similar samples.




  
Sensitivity analysis.
As shown in Fig. <ref>, we evaluate the hyper-parameter sensitivity of  α and β of Eq. (<ref>) and Eq. (<ref>) across 4 domains on the Office-Home datasets.
α and β are  residual ratio to adjust the proportion of new knowledge and inherent knowledge.
Obviously, on most domains, the best residual rate of image features is α=0.2, and the best residual rate of text features is β=0.5.
The model performance will decrease whether the optimal parameters increase or decrease. 
It may be because too much new knowledge causes the model to overfit while too little new knowledge makes it difficult for the model to adapt to a specific task.
Therefore, we set α=0.2 and β=0.5 to obtain the best performance trade-off.







§ CONCLUSION

In this paper, we propose a novel approach named Training-Free Unsupervised Prompt (TFUP), which maximally preserves the inherent representation capabilities and enhances them with a residual connection to similarity-based prediction probabilities in a training-free and labeling-free manner.
We generate similarity-base prediction probabilities by the proposed Feature
Cache Model (FCM) and Multi-level Similarity Measure (MSM). 
In this way, TFUP outperforms original CLIP on all classification datasets by a large margin. 
It achieves promising performance without any labeled data or training, even surpassing the training-base prompt learning methods on multiple classification datasets. 
By efficient fine-tuning, TFUP-T
not only achieves the SOTA compare with unsupervised prompt learning approaches but also demonstrates clear advantages over few-shot prompt learning methods.
We hope that this paper, which maximally preserves the inherent representation capabilities and enhances them while adapting pre-train VLMs to specific downstream tasks in a training-free and labeling-free manner, will provide insights for future work on unsupervised prompt tuning.









10=-1pt

bias2019
Eric Arazo, Diego Ortego, Paul Albert, Noel E O'Connor, and Kevin McGuinness.
Pseudo-labeling and confirmation bias in deep semi-supervised learning.
In 2020 International Joint Conference on Neural Networks. IEEE, 2020.

bachlechner2020rezero
Thomas Bachlechner, Bodhisattwa Prasad Majumder, Huanru Henry Mao, Garrison W Cottrell, and Julian McAuley.
Rezero is all you need: Fast convergence at large depth.
arXiv preprint arXiv:2003.04887, 2020.

chen2020uniter
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.
Uniter: Universal image-text representation learning.
In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, pages 104–120. Springer, 2020.

choi2019pseudo
Jaehoon Choi, Minki Jeong, Taekyung Kim, and Changick Kim.
Pseudo-labeling curriculum for unsupervised domain adaptation.
arXiv preprint arXiv:1908.00262, 2019.

devlin2018bert
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
Bert: Pre-training of deep bidirectional transformers for language understanding.
arXiv preprint arXiv:1810.04805, 2018.

dou2022empirical
Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, et al.
An empirical study of training end-to-end vision-and-language transformers.
In CVPR, pages 18166–18176, 2022.

gao2021clip
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao.
Clip-adapter: Better vision-language models with feature adapters.
arXiv preprint arXiv:2110.04544, 2021.

gao2023softclip
Yuting Gao, Jinfeng Liu, Zihan Xu, Tong Wu, Wei Liu, Jie Yang, Ke Li, and Xing Sun.
Softclip: Softer cross-modal alignment makes clip stronger.
arXiv preprint arXiv:2303.17561, 2023.

he2019revisiting
Junxian He, Jiatao Gu, Jiajun Shen, and Marc'Aurelio Ranzato.
Revisiting self-training for neural sequence generation.
arXiv preprint arXiv:1909.13788, 2019.

he2016deep
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition.
In CVPR, pages 770–778, 2016.

houlsby2019parameter
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.
Parameter-efficient transfer learning for nlp.
In International Conference on Machine Learning, pages 2790–2799. PMLR, 2019.

hu2021lora
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.
Lora: Low-rank adaptation of large language models.
arXiv preprint arXiv:2106.09685, 2021.

huang2022unsupervised
Tony Huang, Jack Chu, and Fangyun Wei.
Unsupervised prompt learning for vision-language models.
arXiv preprint arXiv:2204.03649, 2022.

jia2021scaling
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.
Scaling up visual and vision-language representation learning with noisy text supervision.
In International Conference on Machine Learning, pages 4904–4916. PMLR, 2021.

kahn2020self
Jacob Kahn, Ann Lee, and Awni Hannun.
Self-training for end-to-end speech recognition.
In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7084–7088. IEEE, 2020.

krause2010discriminative
Andreas Krause, Pietro Perona, and Ryan Gomes.
Discriminative clustering by regularized information maximization.
Advances in neural information processing systems, 23, 2010.

lee2013pseudo
Dong-Hyun Lee et al.
Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.
In Workshop on challenges in representation learning, ICML, volume 3, page 896. Atlanta, 2013.

lester2021power
Brian Lester, Rami Al-Rfou, and Noah Constant.
The power of scale for parameter-efficient prompt tuning.
arXiv preprint arXiv:2104.08691, 2021.

li2021prefix
Xiang Lisa Li and Percy Liang.
Prefix-tuning: Optimizing continuous prompts for generation.
arXiv preprint arXiv:2101.00190, 2021.

li2023scaling
Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He.
Scaling language-image pre-training via masking.
In CVPR, pages 23390–23400, 2023.

li2021supervision
Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan.
Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm.
arXiv preprint arXiv:2110.05208, 2021.

liang2020we
Jian Liang, Dapeng Hu, and Jiashi Feng.
Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation.
In International Conference on Machine Learning, pages 6028–6039. PMLR, 2020.

liu2023graphprompt
Zemin Liu, Xingtong Yu, Yuan Fang, and Xinming Zhang.
Graphprompt: Unifying pre-training and downstream tasks for graph neural networks.
arXiv preprint arXiv:2302.08043, 2023.

mcclosky2006effective
David McClosky, Eugene Charniak, and Mark Johnson.
Effective self-training for parsing.
In Proceedings of the human language technology conference of the NAACL, main conference, pages 152–159, 2006.

mclachlan1975iterative
Geoffrey J McLachlan.
Iterative reclassification procedure for constructing an asymptotically optimal rule of allocation in discriminant analysis.
Journal of the American Statistical Association, 70(350):365–369, 1975.

parthasarathi2019lessons
Sree Hari Krishnan Parthasarathi and Nikko Strom.
Lessons from building acoustic models with a million hours of speech.
In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6670–6674. IEEE, 2019.

peng2019moment
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang.
Moment matching for multi-source domain adaptation.
In ICCV, pages 1406–1415, 2019.

peng2017visda
Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko.
Visda: The visual domain adaptation challenge.
arXiv preprint arXiv:1710.06924, 2017.

radford2021learning
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.
Learning transferable visual models from natural language supervision.
In International conference on machine learning, pages 8748–8763. PMLR, 2021.

rosenberg2005semi
Chuck Rosenberg, Martial Hebert, and Henry Schneiderman.
Semi-supervised self-training of object detection models.
Workshops on Applications of Computer Vision, 2005.

saenko2010adapting
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell.
Adapting visual category models to new domains.
In ECCV, pages 213–226. Springer, 2010.

shi2012information
Yuan Shi and Fei Sha.
Information-theoretical learning of discriminative clusters for unsupervised domain adaptation.
arXiv preprint arXiv:1206.6438, 2012.

sohn2020fixmatch
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li.
Fixmatch: Simplifying semi-supervised learning with consistency and confidence.
Advances in neural information processing systems, 33:596–608, 2020.

sohn2020simple
Kihyuk Sohn, Zizhao Zhang, Chun-Liang Li, Han Zhang, Chen-Yu Lee, and Tomas Pfister.
A simple semi-supervised learning framework for object detection.
arXiv preprint arXiv:2005.04757, 2020.

tanwisuth2023pouf
Korawat Tanwisuth, Shujian Zhang, Huangjie Zheng, Pengcheng He, and Mingyuan Zhou.
Pouf: Prompt-oriented unsupervised fine-tuning for large pre-trained models.
In International Conference on Machine Learning, pages 33816–33832. PMLR, 2023.

venkateswara2017deep
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan.
Deep hashing network for unsupervised domain adaptation.
In CVPR, pages 5018–5027, 2017.

wang2020tent
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell.
Tent: Fully test-time adaptation by entropy minimization.
arXiv preprint arXiv:2006.10726, 2020.

xu2021end
Mengde Xu, Zheng Zhang, Han Hu, Jianfeng Wang, Lijuan Wang, Fangyun Wei, Xiang Bai, and Zicheng Liu.
End-to-end semi-supervised object detection with soft teacher.
In ICCV, pages 3060–3069, 2021.

yao2023visual
Hantao Yao, Rui Zhang, and Changsheng Xu.
Visual-language prompt tuning with knowledge-guided context optimization.
In CVPR, pages 6757–6767, 2023.

zhang2021tip
Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li.
Tip-adapter: Training-free clip-adapter for better vision-language modeling.
arXiv preprint arXiv:2111.03930, 2021.

zhao2023instance
Zhen Zhao, Sifan Long, Jimin Pi, Jingdong Wang, and Luping Zhou.
Instance-specific and model-adaptive supervision for semi-supervised semantic segmentation.
In CVPR, pages 23705–23714, 2023.

zhao2023augmentation
Zhen Zhao, Lihe Yang, Sifan Long, Jimin Pi, Luping Zhou, and Jingdong Wang.
Augmentation matters: A simple-yet-effective approach to semi-supervised semantic segmentation.
In CVPR, pages 11350–11359, 2023.

zhao2023entropy
Zhen Zhao, Meng Zhao, Ye Liu, Di Yin, and Luping Zhou.
Entropy-based optimization on individual and global predictions for semi-supervised learning.
In Proceedings of the 31st ACM International Conference on Multimedia, pages 8346–8355, 2023.

zhao2022lassl
Zhen Zhao, Luping Zhou, Lei Wang, Yinghuan Shi, and Yang Gao.
Lassl: Label-guided self-training for semi-supervised learning.
In AAAI, volume 36, pages 9208–9216, 2022.

zhou2022conditional
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.
Conditional prompt learning for vision-language models.
In CVPR, pages 16816–16825, 2022.

zhou2022learning
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.
Learning to prompt for vision-language models.
International Journal of Computer Vision, 130(9):2337–2348, 2022.



