Revisiting Relevance Feedback for CLIP-based Interactive Image Retrieval
Ryoya Nara Yu-Chieh Lin Yuji Nozawa Youyang Ng Goh Itoh Osamu Torii Yusuke Matsui




 


















R. Nara et al.




The University of Tokyo Kioxia Corporation


Revisiting Relevance Feedback for 
 CLIP-based Interactive Image Retrieval
    Ryoya Nara1Yu-Chieh Linr2 Yuji Nozawa2 Youyang Ng2 Goh Itoh2 Osamu Torii2 Yusuke Matsui1

    April 29, 2024
=============================================================================================





Many image retrieval studies use metric learning to train an image encoder. However, metric learning cannot handle differences in users' preferences, and requires data to train an image encoder. To overcome these limitations, we revisit relevance feedback, a classic technique for interactive retrieval systems, and propose an interactive CLIP-based image retrieval system with relevance feedback. Our retrieval system first executes the retrieval, collects each user's unique preferences through binary feedback, and returns images the user prefers. Even when users have various preferences, our retrieval system learns each user's preference through the feedback and adapts to the preference.
Moreover, our retrieval system leverages CLIP's zero-shot transferability and achieves high accuracy without training. 
We empirically show that our retrieval system competes well with state-of-the-art metric learning in category-based image retrieval, despite not training image encoders specifically for each dataset. Furthermore, we set up two additional experimental settings where users have various preferences: one-label-based image retrieval and conditioned image retrieval. In both cases, our retrieval system effectively adapts to each user's preferences, resulting in improved accuracy compared to image retrieval without feedback. Overall, our work highlights the potential benefits of integrating CLIP with classic relevance feedback techniques to enhance image retrieval.








§ INTRODUCTION











With the rapid growth of available images in recent years, image retrieval systems have become ubiquitous, particularly in e-commerce and internet search applications. Image retrieval systems receive a query image from a user, retrieve similar images, and return them to the user.

Many content-based image retrieval studies utilize metric learning <cit.>.  However, in many cases, metric learning is insufficient because users may have various preferences for the returned images. For example, one may search for dog images by querying an image of a dog running in a park, and another may search for park images with the same query image. Simply mapping images into image features might not accommodate those varying preferences. Furthermore, metric learning requires abundant annotated data to train an image encoder.

To overcome these limitations of metric learning-based image retrieval methods, we revisit the classic relevance feedback technique by incorporating the Contrastive Language-Image Pre-training (CLIP) model <cit.>. We propose an image retrieval system that encodes images with a CLIP image encoder and employs relevance feedback to gather information about each user's preferences. The overall retrieval process is as follows:

    
  * Our retrieval system receives a query image from a user, retrieves images from a database, and returns them to the user.
    
  * The user reviews the returned images and provides binary feedback on each image, indicating whether the user prefers it. 
    
  * The system learns the user's preference and returns images the user prefers.


While our retrieval system does require users to provide feedback on the returned samples, it is through the feedback that our system learns each user's unique preference, even when they vary greatly. Whenever a user provides a query image, our system learns the user's preference in real-time and returns relevant images that align with the preference. Moreover, our retrieval system does not necessitate training. By leveraging CLIP's zero-shot transferability and incorporating the user's feedback, we can achieve high retrieval accuracy. Furthermore, because our system does not rely on injecting preferences through text, it can handle diverse and complex queries while avoiding the limitations of multimodal retrieval methods.

We empirically show that our retrieval system achieves high accuracy in various settings. We evaluate our retrieval system by automatically generating users' feedback from dataset labels to simulate users' feedback. In category-based image retrieval settings, our retrieval system achieves competitive accuracy with state-of-the-art metric learning methods. Additionally, we set up two experimental settings where users have various preferences: one-label-based image retrieval and conditioned image retrieval. In these two experimental settings, our retrieval system achieves higher accuracy than image retrieval without relevance feedback. 

In this work, we aim to cast a spotlight on relevance feedback again. Despite the significant advancements in deep learning techniques for image retrieval, relevance feedback has received less attention in recent years. However, our findings demonstrate that incorporating relevance feedback into modern deep learning-based image retrieval systems, such as those based on CLIP, can significantly improve their accuracy.

To sum up, our contributions are as follows:

    
  * We propose CLIP-based interactive image retrieval with relevance feedback. 
    
  * We propose an evaluation method of image retrieval systems with relevance feedback to simulate users' feedback.
   
  * With a realistic feedback size, our system achieves competitive results with supervised metric learning methods in category-based image retrieval, despite not training image encoders specifically for each dataset.
    
  * We set up two experimental settings where users have various preferences. In both settings, our retrieval system improves accuracy from image retrieval without user feedback.
    
  * With a realistic feedback size, our retrieval system achieves competitive results with state-of-the-art multimodal retrieval in conditioned image retrieval settings, despite not exploiting textual information.













§ RELATED WORK




 §.§ Metric Learning



Metric learning <cit.> is a major approach to training an image encoder for image retrieval systems. Metric learning trains an image encoder to map an image into an image feature so that semantically similar images are close together and dissimilar images are apart. Metric learning are utilized in image retrieval <cit.>, personal re-identification <cit.>, face recognition <cit.>, landmark retrieval <cit.>, and few-shot learning <cit.>. 



 §.§ Interactive Image Retrieval


Interactive image retrieval systems enable users to inject their preferences into the retrieval system and obtain samples they prefer. Before the advent of deep learning, relevance feedback <cit.> had been a popular technique for learning each user's preference. Recently, multimodal retrieval <cit.> has become mainstream of interactive image retrieval.


Relevance feedback <cit.> is a technique for interactive retrieval systems. Each user provides feedback for the returned samples, and the retrieval system receives the user's feedback and executes retrieval again to meet their preferences better. In addition to one-time binary feedback, Ahmed <cit.> used scalers for multi-level feedback, and Wu  <cit.> studied retrieval systems where users provide feedback multiple times.
Our work revisits relevance feedback. We focus on cases where users provide binary feedback just once.


Multimodal retrieval <cit.> enables users to inject their preferences through text. Multimodal retrieval addresses tasks such as composed image retrieval <cit.> and conditioned image retrieval <cit.>. For example, along with a query image of a long red dress, a user injects a text query of 
 “I want something short and yellow.” In response, the image retrieval system returns an image of a short yellow dress.




 §.§ Contrastive Language-Image Pre-training (CLIP) 


Contrastive language-image pre-training (CLIP) <cit.> is a pre-trained vision-language model trained on large-scale image-text pairs. CLIP consists of an image encoder that embeds images into a feature space and a text encoder that embeds strings into the same feature space. CLIP models achieve impressive results on various downstream tasks <cit.>, and many studies leverage CLIP as powerful feature extractors.




§ APPROACH



First, we explain the overall retrieval pipeline in <ref>. Next, we describe how we develop a retrieval algorithm that adapts to each user's preference in <ref>. Finally, we explain how to accurately assess whether our retrieval system aligns with user preferences in <ref>.



 §.§ Retrieval Pipeline



We execute retrieval twice for each query image q ∈ℐ. Here, ℐ is a space representing all possible images. We consider two databases: 𝒳_1, 𝒳_2 ⊂ℐ. First, our retrieval system executes retrieval for database images 𝒳_1. Next, it updates the retrieval algorithm and executes retrieval for database images 𝒳_2. In actual application, the database images in both retrievals are the same: 𝒳_1 = 𝒳_2, i.e., we have just one single database 𝒳 and execute retrieval for 𝒳 twice. We introduce the two databases here for a fair comparison, as we will explain in <ref>.  

In advance, we prepare database features. We use visual encoder ϕ: ℐ→ℝ^D and get database features in the first retrieval 𝒱_1 {ϕ(x) | x ∈𝒳_1 }⊂ℝ^D, and database features in the second retrieval 𝒱_2 {ϕ(x) | x ∈𝒳_2 }⊂ℝ^D. 

When a user provides a query image q, our retrieval system encodes q into D-dimensional feature as 𝐮ϕ (q)  ∈ℝ^D. Our retrieval system executes retrieval for 𝒱_1 to obtain similar features to 𝐮. Let us denote the top-M similar samples to 𝐮 from 𝒱_1 as 𝒲_1 ⊆𝒱_1. We write this K-NN (Nearest Neighbor) retrieval operation as a function form in <ref>:

    𝒲_1 =
        ψ(𝐮, M, 𝒱_1) 
        M-_𝐰∈𝒱_1𝐮^⊤𝐰/‖𝐮‖_2 ‖𝐰‖_2.

Note that |𝒲_1|=M. Our retrieval system returns 𝒲_1 to the user.


Next, the user provides binary feedback to each sample in 𝒲_1. We formulate the feedback as <ref>:

    ℱ = { (𝐰, b) |𝐰∈𝒲_1 }.

Here, b ∈{0, 1} is the user's feedback. If the user prefers a returned sample 𝐰, they provide b=1 to 𝐰. Otherwise, they provide b=0. We can regard ℱ as a labeled dataset where each sample 𝐰 has a binary label b.

With ℱ, we update the retrieval algorithm ψ to ψ̃. Next, our retrieval system executes retrieval for 𝒱_2 and retrieves top-K sample 𝒲_2 ⊂𝒱_2:

    𝒲_2 = ψ̃(𝐮, K, 𝒱_2).

𝒲_2 represents the final returned samples. We aim to update the retrieval algorithm so that each returned sample in 𝒲_2 is preferable for the user. Moreover, we aim to choose the visual encoder ϕ to achieve high retrieval performance in various user preferences and datasets.



 §.§ Proposed Method






<ref> illustrates the overview of our proposed method. We propose a CLIP-based retrieval system with relevance feedback. When updating the retrieval algorithm, our retrieval system predicts whether each sample 𝐰∈𝒲_2 is preferable for the user according to the information in ℱ.

We use CLIP visual encoder as our retrieval system's encoder ϕ. We aim to update the retrieval algorithm for various datasets and users' preferences, so we must extract appropriate information from images with an image encoder. Therefore, we utilize off-the-shelf CLIP because CLIP achieves surprising performance in many image recognition tasks without additional training.

After the first retrieval with the feedback ℱ, our retrieval system prepares a binary classifier f: ℝ^D →{0, 1} to predict the user's preference. There could be several options for f, but in our case, f is a simple 1-NN classifier over ℱ: 

    f(𝐚) = b_*   where   (𝐲_*, b_*) = ψ(𝐚, 1, ℱ),

Here, we use a slight abuse notation for ψ. This form of f enables us to update the retrieval algorithm online without requiring any additional training time.


In the second retrieval phase, our retrieval system utilizes f to execute the updated retrieval ψ̃ described in <ref>. First, our retrieval system executes K-NN retrieval as follows:

    𝒲̂_2 = ψ(𝐮, K̂, 𝒱_2).

Here, K̂ is a sufficiently large value. Our retrieval system then refines 𝒲̂_2 by asking f to obtain the returned samples' candidates 𝒲̃_2 as <ref>:

    𝒲̃_2 = {𝐰_2 ∈𝒲̂_2
        | f(𝐰_2) = 1 }.

Finally, our retrieval system picks up the top K elements.

    𝒲_2 =  𝒲̃_2[1:K].

We put <ref> all together to implement ψ̃ in <ref>.











 §.§ Evaluation

 


<ref> illustrates the overall evaluation framework.
As we explained, in the practical application of relevance feedback, the database features in the first and second retrieval are the same: 𝒳_1 = 𝒳_2 and 𝒱_1 = 𝒱_2. However, we could not correctly compare the performance of ψ and ψ̃ in this way, because our retrieval system knows the ground-truth label of whether the user prefers each of the top-M returned samples in the database of the first retrieval.

To evaluate the update of retrieval algorithm performance correctly, we revisit the classic evaluation method of relevance feedback, called the test and control method <cit.>. First, we split all the database images into two subsets: feedback database images and test database images. 
In the first retrieval, we use the feedback database images to obtain samples used for relevance feedback. In the second retrieval, we use the test database images to evaluate retrieval accuracy. That is, the feedback database images are 𝒳_1, and the test database images are 𝒳_2. Here, 𝒳_1 ∩𝒳_2 = ∅.

After splitting, we construct 𝒱_1 and 𝒱_2 as described in <ref>. To compare the retrieval performance of ψ and ψ̃, we execute retrieval for 𝒱_2 with ψ.

    𝒲_2' = ψ(𝐮, K, 𝒱_2).

𝒲_2' means samples returned from 𝒱_2 with the simple K-NN. Finally, we compare 𝒲_2 and 𝒲_2' and measure whether the returned samples are refined.

When we execute the splitting, we set the size of the feedback and test database to the same. Also, we adopt a stratified sampling approach, where the ratios of the labels in each subset are equal. We split the dataset in this way because the test and control method is unbiased only if the two database datasets have equivalent numbers and distributions of samples, as Hull <cit.> says. 




§ EXPERIMENT




Although our retrieval system requires users' feedback, we automatically generate pseudo-feedback for evaluation purposes. We use labeled datasets in each task to automatically generate a user's feedback for each returned sample. In this work, we assume that a user could provide binary feedback correctly. That is, we generate positive feedback for positive samples and negative feedback for negative ones. 

In our experiment, we consider evaluating accuracy of our retrieval algorithm with relevance feedback. First, we consider category-based image retrieval (<ref>), where a user prefers samples with the same label as the query image. Next, we consider two retrieval settings which considers diverse user preferences: one-label-based image retrieval (<ref>) and conditioned image retrieval (<ref>). We devise the two retrieval tasks from the same motivation as the GeneCIS benchmark <cit.>, where users have various intentions. 


Throughout all our experiments, we split the evaluation dataset into three groups: query images, feedback database images 𝒳_1, and test database images 𝒳_2. In all experimental settings, the ratio of these subset sizes is 1:2:2. We conduct retrieval tasks on ten different splittings and calculate the average and standard deviation of Recall@K. We execute all experiments in a single Tesla V100 GPU. Images are resized to 256 × 256 and then cropped to 224 × 224 at the center to input them into the model. We set K̂ = |𝒱_2|. Furthermore, if we could not obtain any returned samples' candidates 𝒲̃ for one query, we view this trial as a failure and calculate Recall@K as 0.



 §.§ Category-based Image Retrieval


Category-based image retrieval is the most common retrieval task. We calculate Recall@K based on whether each returned sample's label is identical to the query's. After the first retrieval, we generate binary feedback for each returned sample based on whether its label is the same as the query's. We use two datasets: CUB-200-2011 <cit.> and Cars-196 <cit.>. Each sample in the two datasets has a single label.

We compare Recall@K of our retrieval system with metric-learning-based image retrieval. We choose triplet loss <cit.> as simple metric learning and HIST <cit.> as state-of-the-art metric learning. 
As a metric-learning-specific procedure, we follow the common practice of metric learning <cit.> and split the whole labels into two groups in advance: the training labels and the evaluation labels. Image encoders for metric learning methods are trained with images of the training labels. That is all for the metric-learning-specific procedure, and we handle only the remaining images (with the evaluation labels) as an evaluation dataset below. We execute the dataset splitting and obtain query images, feedback database images 𝒳_1, and test database images 𝒳_2. We use the same database images and query images to evaluate each retrieval system.
When calculating Recall@K of metric-learning-based image retrieval, we encode test database images 𝒳_2 with the trained encoder and construct test database features. Next, we encode each query image q into a D-dimensional feature with the same encoder, execute K-NN retrieval, and calculate Recall@K.

We use ResNet-50 as a backbone of all image encoders to compare Recall@K equally. To obtain a visual encoder trained with triplet loss <cit.>, we execute training ourselves following the implementation of prior work <cit.>. For HIST <cit.>, the authors publicize their pre-trained models to each dataset, so we use them in our experiment. We fix the feedback size M to 50.





  §.§.§ Result





































<ref> shows the results. Our retrieval system utilizes relevance feedback and achieves competitive results with state-of-the-art metric learning methods, despite not training image encoders specifically for each dataset. Recall@1 of HIST is higher than that of our retrieval system in both datasets, but our retrieval system surpasses HIST in Recall@2,4,8 in both datasets. 

These results mean that simple user feedback for each query is enough to enable the CLIP image encoder to achieve competitive retrieval accuracy with metric learning methods. Although the feedback size M=50 is large for practical applications, this experiment reveals that we can build accurate retrieval systems by combining CLIP and realistic feedback size.




















































 §.§ One-label-based Image Retrieval


One-label-based image retrieval is an experimental setting simulating a user focusing on one of the query image's characteristics. Each user expects that the returned samples have the same characteristics on which the user focuses. For example, consider a user providing a query image of a boy with a hat standing before a door. The user hopes to obtain a hat image, not paying attention to other characteristics such as a door and a boy. In this case, the user provides positive feedback to images of a hat, and our retrieval aims to adapt to the user's preference and return images of a hat. Metric learning cannot handle this task because it does not take each user's preference as input.

To simulate this setting, we use three datasets in which each image has multiple labels: MIT-States <cit.>, Fashion200k <cit.>, and COCO 2017 Panoptic Segmentation <cit.>. For each query image, we focus on one of the query image's labels. We view the label as the characteristic each user focuses on, and regard samples with the focused label as positive when generating user feedback and calculating Recall@K. For example, when a query image has labels of “boy,” “hat,” and “door,” and we focus on the label “hat,” we simulate a user who provides the query image aiming to obtain images of a hat. In this case, we generate positive feedback to samples with the label “hat” and regard such samples as positive when calculating Recall@K.

For each query image q that has L labels {l_i}_i=1^L, we execute trials L times. In ith retrieval, we focus on the label l_i, generate user feedback, and calculate Recall@K as explained above.


We use test images in MIT-States <cit.> and Fashion200k <cit.> and evaluation data of COCO 2017 Panoptic Segmentation <cit.>. We view each dataset as an evaluation dataset and execute the dataset splitting. We regard attributes and nouns for each image of MIT-States and Fashion200k as labels, and annotated objects of each COCO 2017 Panoptic Segmentation image as labels. We fix the feedback size M to 50. As backbones of CLIP image encoders, we choose ViT-B/32 and ResNet-50. 





  §.§.§ Result






<ref> shows the results. Our retrieval system successfully improves Recall@K with relevance feedback in all datasets and encoder architectures by a significant margin. With the relevance feedback, Recall@1 improves by up to 9.5%. 
These results mean that our retrieval system successfully adapts to each user's preference in our simulation experiment. <ref> illustrates one example of our retrieval system's trial with CLIP ViT-B/32 in COCO dataset. In this case, we focus on a label “banana,” simulating a user who provides the query image in <ref> searching for banana images. With relevance feedback, our retrieval system can return images with the label “banana” more accurately.

Regarding the encoder architecture, ViT-B/32 surpasses ResNet-50 in Recall@K. In terms of Recall@1, ViT-B/32 ourperforms ResNet-50 by 1.8% in Fashion200k, and 2.1% in MIT-States. These results suggest that CLIP image encoders with large architectures work better in one-label-based image retrieval settings.




















































 §.§ Conditioned Image Retrieval


Conditioned image retrieval is an experimental setting simulating a user who searches for similar images to a query image but differs from it in some aspects. For example, a user provides a ripe apple image, but they want unripe apple images. In this case, the user provides positive feedback to images of an unripe apple, and our retrieval system aims to adapt to the user's preference and return unripe apple images. Metric learning cannot handle this task because it does not take such users' preferences as input.


To simulate this setting, we follow previous conditioned image retrieval studies <cit.> and use MIT-States <cit.>. Each image in MIT-States has one adjective label and one noun label. 
We use the same experimental settings as previous conditioned image retrieval studies. For each query image q with the adjective label l_adj and noun label l_noun, we choose one adjective label l_adj' that is different from l_adj. We view images that have both l_adj' and l_noun as those the user prefers, and regard samples with l_adj' and l_noun as positive when generating and calculating Recall@K. For example, when a query image has l_adj=“ripe” and l_noun=“apple,” and we choose l_adj'=“unripe,” we simulate a user who provides a ripe apple image aiming to obtain images of an unripe apple. In this case, we automatically generate positive feedback to samples with labels “unripe” and “apple,” and regard such samples as positive when calculating Recall@K.



For each q with l_adj and l_noun, we try all possible adjective labels ℒ_adj\{ l_adj}. Here, ℒ_adj is a set of all adjective labels. When we define { l_adj^i}_i=1^Nℒ_adj\{ l_adj} (N=|ℒ_adj\{ l_adj}|), we execute trials N times for q. In ith retrieval, we choose l_adj^i, generate binary feedback, and calculate Recall@K as explained above.



Additionally, we compare our retrieval system with multimodal retrieval. We choose GeneCIS <cit.> as the state-of-the-art multimodal retrieval addressing conditioned image retrieval on MIT-States. GeneCIS consists of visual encoder ϕ'_v:ℐ→ℝ^D and multimodal encoder ϕ'_mm: ℐ×𝒯→ℝ^D. Here, 𝒯 is a set of all text. 
We reimplement GeneCIS and calculate Recall@K in the same query images and test database images 𝒳_2 as those used to evaluate our retrieval system. We obtain test database features as 𝒱_2' {ϕ'_v(x) | x ∈𝒳_2}.
Consider the query image q has l_adj and l_noun, and we choose l_adj'(≠ l_adj). We generate the query feature 𝐮_mmϕ'_mm(q, l_adj'). Note that each adjective label is represented as text: l_adj' ∈𝒯. We execute retrieval and return samples 𝒲_mm as follows:

    𝒲_mm = ψ(𝐮_mm, K, 𝒱_2').




We view test images of MIT-States as an evaluation dataset and execute the dataset splitting. We fix M to 50. We choose ResNet-50x4 and ViT-B/16 as CLIP visual encoder backbones. The authors publicize the pre-trained models of GeneCIS <cit.>, and we use them for comparison. 






  §.§.§ Result






<ref> shows the results. We successfully improve Recall@K with relevance feedback from simple CLIP K-NN retrieval without relevance feedback by a large margin. Furthermore, when we provide a realistic amount of relevant feedback (M=50), our retrieval system achieves competitive accuracy with state-of-the-art multimodal retrieval. Recall@1 of ours and GeneCIS are almost equal in both encoder architectures. These results mean that our retrieval system adapts to each user's preference without textual information in this setting. 
We demonstrate an example trial of our retrieval system (CLIP ViT-B/16) in <ref>. In the first case, we choose an adjective label “spilled,” simulating a user who provides the peeled paint image searching for spilled paint images. With relevance feedback, our retrieval system can return images with the labels “spilled” and “paint” more accurately.




§ ADDITIONAL ANALYSIS




 §.§ Architectures of CLIP image encoders and the feedback size


In category-based image retrieval settings, we change the architecture of CLIP image encoder and evaluate how our retrieval system improves the retrieval accuracy. We choose ResNet-50, ViT-B/32, and ViT-L/14 as backbone architectures and calculate Recall@K in each setting. Also, we vary the feedback size (M) and assess the relationship between the feedback size and the retrieval accuracy. We vary M among 10, 25, and 50.




<ref> shows the results of comparison among various kinds of CLIP and M. Larger M leads to better Recall@K. We can infer that our retrieval system obtains more information from the larger feedback. Moreover, the larger architecture of CLIP image encoder achieves higher accuracy, implying that the larger CLIP image encoder extracts more appropriate information from images. 

In particular, CLIP ViT-L/14 performs exceptionally well for M=10, which is truly a realistic and user-friendly feedback size. Combined with the results described in <ref>, it is shown that we can achieve a practical interactive image retrieval system with CLIP ViT-L/14 and a smaller feedback size.



 §.§ Retrieval accuracy and the number of positive feedback


We examine the influence of positive samples in the first retrieval on the retrieval performance in the second retrieval. We choose category-based image retrieval settings, where the architecture of CLIP visual encoder is ViT-B/32 and M=50. We count the positive feedback in the first retrieval, which is represented as | {  (𝐰, b) ∈ℱ| b = 1 } |. At the same time, we calculate MAP@R of the returned samples in the second retrieval. MAP@R is calculated as follows <cit.>:

    MAP@R   = 1/R∑_i=1^RP(i).
    
            P(i)    = precision at i (if the ith returned sample is positive),
    
        0 (otherwise).

Here, R is the total number of positive samples in 𝒳_2. We execute the retrieval for all query images and collect pairs of the number of positive feedback in the first retrieval and MAP@R of the second retrieval.

<ref> shows the results for each dataset. As we can see, a positive correlation exists between the number of positive feedback and MAP@R. These results suggest that more positive feedback leads to higher retrieval performance. When we have positive samples in the user feedback, we can accurately predict whether each sample in the second retrieval is preferable.







 §.§ Retrieval runtime


We examine the retrieval runtime of our retrieval system. We choose category-based image retrieval settings. We choose ViT-B/32 as the backbone architecture of CLIP image encoder. We measure the runtime in each process: encoding, CLIP K-NN, and the updated retrieval. We can formulate each process as ϕ(q), ψ(𝐮, K, 𝒱_2), and ψ̃(𝐮, K, 𝒱_2) respectively. We set K̂ = | 𝒱_2|, the most time consuming settings. We use PyTorch libraries and execute all calculations in GPU memory. We use the same computational resources as <ref>.

<ref> shows the results. Our updated retrieval ψ̃ takes only 3 times overhead compared to the simple K-NN retrieval ψ. The encoding takes much longer than the retrieval, so the difference in the retrieval runtime can be ignored in our retrieval system. We discuss the retrieval runtime theoretically in the supplementary material.





 §.§ Limitation

One limitation of our work is that we utilize pre-existing datasets to create simplified experimental scenarios that mimic user feedback. However, real-world human users may possess a wider range of subjective preferences that cannot be fully replicated by current methods. To assess interactive image retrieval systems with greater accuracy, a novel dataset that captures the nuances of human users' preferences is imperative.



§ CONCLUSION

Our study proposed a CLIP-based interactive image retrieval system to overcome the shortcomings of metric learning. Our retrieval system receives binary feedback from each user, updates the retrieval algorithm, and returns images the user prefers. Our retrieval system adapts to any user preference and works well without training an image encoder. This paper revisited relevance feedback and integrated it with CLIP, suggesting a powerful baseline for interactive image retrieval. To the best of our knowledge, we are the first to explore the possibility of retrieval with modern vision-language models and relevance feedback. Our paper will throw a spotlight on relevance feedback again.






splncs04













Supplementary Material for 
 “Revisiting Relevance Feedback for 
 CLIP-based Interactive Image Retrieval”
    Ryoya Nara1Yu-Chieh Linr2 Yuji Nozawa2 Youyang Ng2 Goh Itoh2 Osamu Torii2 Yusuke Matsui1

    April 29, 2024
=========================================================================================================





§ IMPLEMENTATION DETAILS


To increase the reproducibility of our experiments and make them more understandable, we provide additional information and explanations about our experimental conditions that we omit due to space limitations.



 §.§ Category-based image retrieval


When training an encoder with metric learning, we follow the common practice of metric learning and treat the former half of the labels as the training labels and the latter half as the evaluation labels. <ref> illustrates the dataset usage in our category-based image retrieval experiment in the case of CUB-200-2011. CUB-200-2011 has 200 labels in total. Therefore, we use the first 100 labels for training and the second 100 labels for evaluation. We treat images of the second 100 labels as an evaluation dataset, and execute the dataset splitting to obtain three subsets: query images, feedback database images, and test database images. 


Since feedback database images are reserved for applying relevance feedback as in our proposed approach, when evaluating metric-learning-based image retrieval approaches, only query images and test database images have been used. 



 §.§ One-label-based image retrieval

For MIT-States and Fashion200k datasets, some images have been excluded in our evaluation because their captions, i.e., the set of labels, are not qualified. The details are as below. 

First, we exclude rare captions that are not shared by enough images. Specifically, if a caption occurs less than 5 times in the evaluation dataset, where 5 is the minimal number of samples for doing a 1:2:2 splitting introduced in Sec. 4, the few images with that caption are not included in our experiments. For example, there are only two images are with the "molten orange" caption in XXX dataset, so the two images are excluded in our experiments. 

Second, when using MIT-States, we exclude samples with the adjective label “adj,” which has no meaning as an adjective. This is the same implementation as the prior conditioned image retrieval studies.



 §.§ Conditioned image retrieval


To have MIT-States dataset for conditioned image retrieval evaluation, we first remove the images with unqualified captions as described in <ref>. Additionally, when a query image has a noun label l_noun and an adjective label l_adj, we choose a different adjective label l_adj' so that there is at least one sample that has l_adj' and l_noun. 



§ DATASET DETAILS







We describe the details of the evaluation datasets that we use in our experiments in <ref>. We also attach information about the number of queries in <ref>. We change the splitting way ten times and calculate the average and standard deviation of the number of queries because the number of queries depends on query images' labels.



§ EXAMPLES OF ONE-LABEL-BASED IMAGE RETRIEVAL



We provide additional examples of one-label-based image retrieval in <ref>. In the case of Fashion200k, our retrieval system successfully retrieves images with “blue” labels more accurately than image retrieval without relevance feedback.




§ THEORETICAL AND ADDITIONAL RUNTIME ANALYSIS






We provide a theoretical explanation of the retrieval runtime difference between K-NN (ψ) and the updated retrieval (ψ̃).
As in <ref>, simple K-NN calculates cosine similarities |𝒱_2| times. In contrast, the updated retrieval executes cosine similarities K̂M times in <ref>.
Along with the operation of <ref>, the updated retrieval calculates cosine similarities K̂M+|𝒱_2| times. Therefore, the updated retrieval is 1 + K̂M/|𝒱_2| times slower than the simple K-NN. In the experiment of <ref>, we set M=50 and K̂=|𝒱_2|, so the updated retrieval runtime could be 51 times longer than the simple K-NN retrieval runtime.
In our experiment, we execute retrieval in GPU memory. GPUs are good at simple parallel computation, and since the number of data we are dealing with in this paper can be fitted into GPU memory, there is not much difference in the retrieval runtime.

To explore retrieval runtime further, we try smaller K̂ and observe retrieval accuracy and speed. We set K̂=30,100,|𝒱_2| and compare retrieval runtime and Recall@1. <ref> shows the results. Larger K̂ leads to longer retrieval runtime, this can be expected from our theoretical explanation. Additionally, setting smaller K̂ slightly lower Recall@1.




§ FUTURE WORKS


This paper only focuses on one-time binary relevance feedback. We will achieve more complex interactive image retrieval tasks by exploring multiple-level and multi-time relevance feedback and combining them with modern deep learning. 

We also need to evaluate whether the returned samples match the user's preferences by having actual users provide feedback on the returned samples. In this study, we automatically generated feedback using labels in the dataset, assuming that users could correctly provide binary feedback. However, users may provide incorrect feedback with a certain probability. Experiments with users are necessary to evaluate whether our retrieval system can handle such cases.








CHAPTER: SUPPLEMENTARY MATERIAL FOR 
 “REVISITING RELEVANCE FEEDBACK FOR 
 CLIP-BASED INTERACTIVE IMAGE RETRIEVAL”




§ IMPLEMENTATION DETAILS


To increase the reproducibility of our experiments and make them more understandable, we provide additional information and explanations about our experimental conditions that we omit due to space limitations.



 §.§ Category-based image retrieval


For the implementation of metric-learning-based methods, when training an encoder with metric learning, we follow the common practice of metric learning and treat the former half of the labels as the training labels and the latter half as the evaluation labels. <ref> illustrates the dataset usage in our category-based image retrieval experiment in the case of CUB-200-2011. CUB-200-2011 has 200 labels in total. Therefore, we use the first 100 labels for training and the second 100 labels for evaluation. We treat images of the second 100 labels as an evaluation dataset, and execute the dataset splitting to obtain three subsets: query images, feedback database images, and test database images. 


Since feedback database images are reserved for applying relevance feedback as in our proposed approach, when evaluating metric-learning-based image retrieval approaches, only query images and test database images have been used. 



 §.§ One-label-based image retrieval

For MIT-States and Fashion200k datasets, some images have been excluded in our evaluation because their captions, i.e., the set of labels, are not qualified. The details are as below. 

First, we exclude rare captions that are not shared by enough images. Specifically, if a caption occurs less than 5 times in the evaluation dataset, where 5 is the minimal number of samples for doing a 1:2:2 splitting introduced in Sec. 4, the few images with that caption are not included in our experiments. For example, there are only two images with the "molten orange" caption in the dataset, so the two images are excluded in our experiments. 

Second, when using MIT-States, we exclude samples with the adjective label “adj.” This is the same implementation as the prior conditioned image retrieval studies.



 §.§ Conditioned image retrieval


To have MIT-States dataset for conditioned image retrieval evaluation, we first remove the images with unqualified captions as described in <ref>. Additionally, when a query image has a noun label l_noun and an adjective label l_adj, we choose a different adjective label l_adj' in the condition that there is at least one sample that has both l_adj' and l_noun. For example, no sample has both “unripe” and “mountain” in the dataset, so we do not choose l_adj'=“unripe” when we handle the query image which has l_noun=“mountain.”



§ DATASET DETAILS







We describe the details of the evaluation datasets that we use in our experiments in <ref>. We also attach information about the number of queries in <ref>. We change the splitting way ten times and calculate the average and standard deviation of the number of queries because the number of queries depends on the number of query images' labels. 

It is not a mistake that the number of queries in MIT-States differs between one-label-based image retrieval and conditioned image retrieval. As described in our main paper, the number of queries for one query image differs in these two settings. In one-label-based image retrieval, when a query image has L labels, we execute L queries for all labels of the query image. In conditioned image retrieval, when a query image has l_adj, we execute queries for all possible different adjective labels l_adj' ∈ℒ_adj\{ l_adj}.



§ EXAMPLES OF ONE-LABEL-BASED IMAGE RETRIEVAL



We provide additional examples of one-label-based image retrieval in <ref>. In the case of Fashion200k, we simulate a user who searches for images of a blue object by providing the query image to our retrieval system. Our retrieval system successfully learns the user's preference from the feedback, and returns images with “blue” labels more accurately than image retrieval without relevance feedback. In the case of MIT-States, we simulate a user who searches for tea images by providing the query image of streaming tea to our retrieval system. Our retrieval system successfully learns the user's preference from the feedback, and returns tea images more accurately.




§ THEORETICAL AND ADDITIONAL RUNTIME ANALYSIS






We provide a theoretical explanation of the retrieval runtime difference between K-NN (ψ(𝐮, K, 𝒱_2)) and the updated retrieval (ψ̃(𝐮, K, 𝒱_2))).
Simple K-NN calculates cosine similarities |𝒱_2| times. In contrast, the updated retrieval executes cosine similarities K̂M times.
The updated retrieval calculates cosine similarities K̂M+|𝒱_2| times. Therefore, the updated retrieval is 1 + K̂M/|𝒱_2| times slower than the simple K-NN. In our experiment, we set M=50 and K̂=|𝒱_2|, so the updated retrieval runtime could be 51 times longer than the simple K-NN retrieval runtime.
In our experiment, we execute retrieval in GPU memory. GPUs are good at simple parallel computation, and since the number of data we are dealing with in this paper can be fitted into GPU memory, there is not much difference in the retrieval runtime.

To explore retrieval runtime further, we try smaller K̂ and observe retrieval accuracy and speed. We set K̂=30,100,|𝒱_2| and compare retrieval runtime and Recall@1. <ref> shows the results. Larger K̂ leads to longer retrieval runtime, this can be expected from our theoretical explanation. Additionally, setting smaller K̂ slightly lower Recall@1.




§ FUTURE WORKS


This paper only focuses on one-time binary relevance feedback. In future works, we aim to consider more complex interactive image retrieval tasks by expanding our retrieval system to incorporate multi-level and multi-turn relevance feedback. 

Additionally, experiments with actual user feedback are necessary for a more accurate evaluation of our retrieval system. In this study, we prepare three experimental settings and automatically generate feedback to simulate users' feedback. However, actual users may have various preferences that cannot be fully replicated by current methods. By having actual users provide feedback on returned samples, we can evaluate the accuracy of our retrieval system in more practical situations.

