DVF: Advancing Robust and Accurate Fine-Grained Image Retrieval with Retrieval Guidelines
Xin Jiang Hao Tang Rui Yan Jinhui Tang Zechao Li









  Nanjing University of Science and Technology
  Nanjing
  China

xinjiang@njust.edu.cn



  Nanjing University of Science and Technology
  Nanjing
  China

tanghao0918@njust.edu.cn




  Nanjing University
  Nanjing
  China

ruiyan@nju.edu.cn




  Nanjing University of Science and Technology
  Nanjing
  China
jinhuitang@njust.edu.cn

Corresponding author.

  Nanjing University of Science and Technology
  Nanjing
  China
zechao.li@njust.edu.cn




















  Fine-grained image retrieval (FGIR) is to learn visual representations that distinguish visually similar objects while maintaining generalization.
  
  
  Existing methods propose to generate discriminative features, but rarely consider the particularity of the FGIR task itself.
  
  
  This paper presents a meticulous analysis leading to the proposal of practical guidelines to identify subcategory-specific discrepancies and generate discriminative features to design effective FGIR models.
  
  These guidelines include emphasizing the object (G1), highlighting subcategory-specific discrepancies (G2), and employing effective training strategy (G3).
  
  Following G1 and G2, we design a novel Dual Visual Filtering mechanism for the plain visual transformer, denoted as DVF, to capture subcategory-specific discrepancies.
  
  Specifically, the dual visual filtering mechanism comprises an object-oriented module and a semantic-oriented module. These components serve to magnify objects and identify discriminative regions, respectively.
  
  
  Following G3, we implement a discriminative model training strategy to improve the discriminability and generalization ability of DVF.
  
  Extensive analysis and ablation studies confirm the efficacy of our proposed guidelines.
  
  Without bells and whistles, the proposed DVF achieves state-of-the-art performance on three widely-used fine-grained datasets in closed-set and open-set settings.




















































DVF: Advancing Robust and Accurate Fine-Grained Image Retrieval with Retrieval Guidelines
    Zechao Li
    April 29, 2024
=========================================================================================





§ INTRODUCTION
 

Fine-grained image retrieval (FGIR) aims to retrieve images with the same subcategory as the query images from a database within the metacategory (e.g., birds, and cars) <cit.>.

Retrieving visually similar images, however, faces challenges arising from subtle inter-class differences caused by similar objects, as well as significant intra-class variations due to factors such as location, scale, and deformation.

Additionally, existing works <cit.> follow a closed-set training setting, where all the subcategories in the training set are known.

However, the evaluation includes a closed-set setting with known test set categories and an open-set setting with unknown test set subcategories.


Consequently, the essence of FGIR tasks lies in learning discriminative and generalizable embeddings to identify visually similar objects.








Recently, FGIR methods can be categorized into two main groups: encoding-based <cit.> and localization-based <cit.>.

The encoding-based methods primarily optimize image-level features, which often include background and non-discriminative information.

Localization-based methods typically formulate an effective object feature extraction strategy in the image encoder to capture subtle differences among subcategories, often rooted in the distinctive properties of object parts.

However, these methods still suffer from small-sized objects in the input image, making it difficult to identify discriminative regions.

Moreover, the scarcity of fine-grained image data jeopardizes the model's discriminative capacity and generalization ability.


Based on the preceding analysis, we present a set of guidelines for designing high-performance fine-grained image retrieval models:

    
  * G1: emphasize the object. 
    The FGIR models should utilize object-emphasized images as input to alleviate the challenges posed by small-sized objects that make identifying objects and discerning subcategory-specific discrepancies difficult <cit.>.
    
    Without object-emphasized images as input, the accuracy of previous methods <cit.> is fundamentally limited.
    
    
  * G2: highlight subcategory-specific discrepancies. Given the considerable intra-class differences and subtle inter-class variations inherent in the FGIR task, highlighting subcategory-specific discrepancies becomes paramount.
    
    An illustration of this is seen in methods <cit.> that do not highlight subcategory-specific discrepancies, thus restricting their discriminative capability.
    
    
  * G3: employ effective training strategy. As shown in previous studies <cit.>, limited fine-grained image data inevitably limits retrieval performance.
    
    Some works <cit.> affirm that effective training strategies can alleviate this limitation.
    
    However, others still neglect this aspect, hindering the design of high-performance retrieval models.



Following the guidelines above, we propose a straightforward yet effective approach: the plain visual transformer equipped with a Dual Visual Filtering mechanism, referred to as DVF.


Fig. <ref> illustrates the proposed DVF framework, consisting of an Object-oriented Visual Filtering module (OVF) and a Semantic-oriented Visual Filtering module (SVF), to generate discriminative features.

Motivated by the visual foundation model's ability to accurately detect arbitrary objects without requiring additional fine-tuning, OVF utilizes a visual foundation model and a post-processing strategy to zoom in on the object in the input image (G1).

In this way, OVF can adjust image content to aid subcategory prediction and locate regions of subcategory-specific discrepancies.

In SVF, we propose a token importance generator to calculate token-level importance, which is subsequently used as a weighting factor in class attention to enhance the selection of discriminative tokens and eliminate noisy features (G2).

This improves the model's ability to discern subtle discrepancies between subcategories.

Following G3, we propose a Discriminative Model Training strategy that combines data augmentation with loss constraints, significantly improving the discriminability of DVF across both closed-set and open-set scenarios.

In summary, the primary contributions of this work are as follows:

    
  * By considering the particularity of fine-grained image retrieval, we formulate a set of practical guidelines for designing high-performance fine-grained image retrieval models.
    
  * Following our proposed guidelines, we develop a straightforward yet potent retrieval model called DVF, which incorporates a dual visual filtering mechanism to capture subcategory-specific discrepancies.
    
    Additionally, we employ a discriminative training strategy to enhance the model's discriminability and generalization ability.
    
  * The experimental results on three fine-grained image retrieval benchmarks demonstrate the superior performance of DVF exhibits in closed-set and open-set scenarios.
    
    Moreover, visualization results demonstrate DVF's capability to capture discriminative image regions accurately.



§ RELATED WORKS


Existing methods for fine-grained image retrieval (FGIR) can be categorized as encoding-based or localization-based.

The encoding-based methods <cit.> aim to learn an embedding space in which samples of a similar subcategory are attracted and samples of different subcategories are repelled.

The methods can be decomposed into roughly two components: the image encoder maps images into an embedding space, and the metric method ensures that samples from the same subcategories are grouped closely, while samples from different subcategories are separated.

While these studies have achieved significant achievements, they primarily concentrate on optimizing image-level features that include numerous noisy and non-discriminatory details.

Therefore, the localization-based methods <cit.> are proposed, which focus on training a subnetwork for locating discriminative regions or devising an effective strategy for extracting attractive object features to facilitate the retrieval task.

Unlike these approaches, our methodology considers the specific characteristics of the FGIR tasks, offering guidance for designing high-performance retrieval models.





















§ METHOD











 §.§ Image Encoder

For the input image I∈ℝ^3 × H × W, the image encoder will first split it into N=N_h× N_w non-overlapping patches of size P × P. Here, N_h = H/P and N_w =W/P.

Subsequently, these patches are transformed into embedding tokens E = E^1, E^2, …, E^N∈ℝ^N × D using learnable linear projection, where D denotes the dimension of each token.

Finally, the embedding tokens E are concatenated with a class token E^class∈ℝ^D, and the position information of the tokens is retained through combined a learnable position embedding E_pos to form the initial input token sequence as E_0 = E^class, E^1, E^2, …, E^N + E_pos.

The image encoder comprises L layers, each incorporating multi-head self-attention (MHSA) and a multi-layer perception (MLP) block. Therefore, considering the input E_i-1 of the i-th layer, the output can be expressed as follows:



    𝐄_i^'   = MHSA(LN(𝐄_i-1)) + 𝐄_i-1, 
    𝐄_i   = MLP(LN(𝐄_i^')) + 𝐄_i^',



where i = { 1, 2, …, L }, and LN(·) denotes the layer normalization operation. The class token E^class_L from the L-th layer serves as the retrieval embedding, we simplify it to E^R.



 §.§ Dual Visual Filtering Mechanism

Subtle yet discriminative discrepancies are widely acknowledged as crucial for FGIR <cit.>.

However, in real-life scenarios, subcategory objects often occupy only a fraction of the image, with the background dominating a larger portion. This presents challenges for the model in both object recognition and identifying discriminative regions within objects.

Additionally, traditional visual Transformers (ViTs) <cit.> are originally designed to represent general visual concepts, rather than effectively capturing subtle discrepancies for subordinate categories within a given category in fine-grained visual concepts.

To address these challenges, following G1 and G2, we design a dual visual filtering mechanism consisting of two components: an object-oriented visual filtering module and a semantic-oriented visual filtering module.



  §.§.§ Object-oriented Visual Filtering Module
 
Following G1, we propose the Object-oriented Visual Filtering module (OVF) to zoom in on the object in the image.

Previous methods <cit.> typically train a sub-network to locate objects in a weakly supervised manner, consuming training resources and often leading to inaccurate positioning due to the limitations of weakly supervised training.

In contrast, DVF investigates the utilization of visual foundation models <cit.> for accurate object localization in a training-free manner.

Specifically, OVF leverages the visual foundation model Grounding-DINO <cit.> along with a post-processing strategy, to locate and magnify objects.

This allows the model to capture more discriminative details, thereby enhancing the quality of the representation.

Besides, the proposed OVF is model-agnostic, making it applicable to other FGIR methods.




Specifically, Grounding-DINO utilizes an image and its associated text prompt (metacategory name) as input, as shown in Fig. <ref>, to generate coordinate results and confidence scores for the corresponding metacategory object.

We can use the coordinate results to crop the input image to obtain an image with emphasized objects.


However, we observed that to acquire input images with emphasized objects, it is necessary to 1) confirm the presence of the object in the image, 2) ensure the object remains intact, and 3) prevent deformation after image resizing.








To achieve this, we introduce a post-processing strategy, which involves self-checking and filtering adaptation, the process of post-processing is summarized in Algorithm <ref>.



To confirm the presence of the object in the image, we perform self-checking before utilizing the detection results.

 If the detection score surpasses the threshold α, we adopt the detection result; otherwise, the original input image serves as the output of the OVF. In this paper, we set α to 0.5.


To ensure the object remains intact, we enlarge the detection results by a factor of 1.1, ensuring the inclusion of edge areas possibly overlooked during the detection process.



In addition, to prevent deformation after image resizing, we constrain the aspect ratio of the image to 3:4 by adding margins to the shortest side. 



The experimental results depicted in Fig. <ref>(g) validate the effectiveness of the post-processing strategy.





  §.§.§ Semantic-oriented Visual Filtering Module
 
In plain visual transformers <cit.>, the transformer layer treats all image regions divided into patches equally, leading to the merging of a substantial amount of redundant patch information.

This limits the retrieval accuracy of ViT-based models in fine-grained image retrieval tasks.

Following G2, we propose a Semantic-oriented Visual Filtering module (SVF). 

This module merges attention scores from the class token with token importance generated by a token importance generator to select discriminative tokens and eliminate noisy tokens.



The attention score of the class token indicates the semantic significance of each region for subcategory prediction.

Therefore, for the attention scores A = A^1, A^2, …, A^m, …, A^M from the class token in the penultimate layer, where A^m∈ℝ^N denotes the attention score of the m-th attention head.

We aggregate the attention scores of all heads to obtain the semantic scores for embedding tokens as Â = ∑_i=1^MA^i.


Then, the semantic score Â can be used as the evaluation metric to select discriminative tokens as in previous work <cit.>.

However, there arises the issue of inaccurate subcategory prediction, leading to the semantic score's inability to precisely reflect the semantic importance of the token. This problem is particularly pronounced in the open-set scenario, where the subcategory in the test set remains unknown.

To eliminate the issue, the SVF introduces a token importance generator Ω to transform the input embedding tokens into a new space, thereby specifying the importance of the embedding tokens.

Concretely, we project the embedding tokens into the token importance Z∈ℝ^N. This can be expressed as follows:

    Z = σ(Ω(E_i)) i = { 1,2, …, N } ,

where σ(·) is a sigmoid activation function, Ω is a linear projection layer, and E_i is the embedding token.


We choose the semantic score Â of the token as the basic evaluation metric. Through token importance weighting as per Eq. <ref>, we can more accurately estimate the semantic score O.


    O = Â⊕ (Â⊗Z),

where ⊕ and ⊗ are element-wise summation and element-wise multiplication, respectively.

Subsequently, we select the top-k significant embedding tokens with high activation values, denoted as 𝐢𝐝𝐬 =  (O) ∈ℤ^K, where (·) sorts the importance scores O in descending order and returns the index 𝐢𝐝𝐬 of the top k embedding tokens.

The selected k embedding tokens concatenate them with the class token to form the input sequence of the L-th transformer layer, expressed as:

    𝐄_L-1 = 𝐄^class_L-1, 𝐄^𝐢𝐝𝐬(1)_L-1, 𝐄^𝐢𝐝𝐬(2)_L-1, ⋯, 𝐄^𝐢𝐝𝐬(k)_L-1.

By replacing the original input sequence of the last transformer layer with 𝐄_L-1, the proposed SVF encourages the image encoder to highlight the subtle discrepancies between different subcategories while ignoring regions such as background or less meaningful foreground regions, thereby promoting fine-grained understanding.








 §.§ Discriminative Model Training Strategy


With the collaborative efforts of the dual visual filtering mechanism, the proposed DVF demonstrates exceptional retrieval performance in FGIR tasks.

However, it focuses on model design and does not effectively address the challenge of limited fine-grained image data, as noted in G3.

Therefore, following G3, we delve into the formulation of discriminative model training strategy (DMT).

Data augmentation is a training strategy that has proven effective in the field of classification <cit.> to alleviate the problem of limited training samples.

Inspired by this, we incorporate data augmentation into the retrieval tasks as a component of DMT.

It enhances data diversity throughout model training and diminishes the model's reliance on specific features from the training set.

Specifically, we consider the following transformations:

  
  * Color-jitter: Add perturbation to colors.
  
  * Grayscale: Give more focus on shapes.
  
  * Gaussian Blur: Slightly change the details in the image.


For each image, we consistently apply color-jitter and randomly apply Grayscale and Gaussian Blur.

In Table <ref>, we present an ablation study of our various data augmentation components in the open-set setting.

Meanwhile, contrastive loss <cit.> is presented as a loss constraint into DMT, which improves the generalization and balance of the model in the training process.

Formally, the contrastive loss with batch size B is expressed as:

    ℒ_con = 1/B^2∑_i=1^B∑_j:y_i=y_j^B(1 - Sim(E_i^R, E_j^R) + 
    ∑_j:y_i≠ y_j^Bmax( (Sim(E_i^R, E_j^R) - β), 0)  ,

where y_i is a subcategory label, Sim(E_i^R, E_j^R) represents the dot product between E_i^R and E_j^R, and the β is a constant margin.








 §.§ Overall Function

We optimize a training objective as below:

    ℒ = ℒ_pnca + ℒ_con,

where ℒ_pnca denotes the ProxyNCA loss <cit.>:

    ℒ_pnca = -log( exp(-d(‖E_i^R‖, ‖c_i‖))/∑_c∈Cexp(-d(‖E_i^R‖, ‖c‖))),

where d(‖E_i^R‖, ‖c_i‖) represents the distance between ‖E_i^R‖ and ‖c_i‖, c_i denotes the class proxy corresponding to retrieval embedding E_i^R, C is the class proxy set, and ‖·‖ denotes the L^2-Norm.






§ EXPERIMENTS







 §.§ Experiment Setup



  §.§.§ Datasets
 
CUB-200-2011 <cit.> comprises 11, 788 bird images from 200 bird species. In the closed-set setting, the dataset is divided into training and testing subsets comprising 5,994 and 5,794 images, respectively, out of a total of 11,788 images. For the open-set setting, we employ the first 100 subcategories (5,864 images) for training, and the remaining subcategories (5,924 images) are used for testing.

Stanford Cars <cit.> consists of 16,185 images depicting 196
car variants. Similarly, these images were split into 8,144 training images and  8,041 test images in the closed-set setting. For the open-set setting, we utilize the first 98 subcategories (comprising 8,054 images) for training and the remaining 98 subcategories (comprising 8,154 images) for testing.

NABirds <cit.> contains 48,562 images showcasing North American birds across 555 subcategories. For the closed-set setting, the training set contains 23,929 images, while the remaining 24,633 images are used for testing. For the open-set setting, we set up a more challenging training/test set split, with 20,984 images from the first 255 subcategories used for training and 27,578 images from the remaining subcategories used for testing.



  §.§.§ Evaluation protocols
 To evaluate retrieval performance, we adopt Recall@K with cosine distance in previous work <cit.>, which calculates the recall scores of all query images in the test set.

For each query image, the top K similar images are returned. A recall score of 1 is assigned if at least one positive image among the top K images; otherwise, it is 0.



  §.§.§ Implementation details

In our experiments, we employ the ViT-B-16 <cit.> pre-trained on ImageNet21K <cit.> as our image encoder.

All input images are resized to 256 × 256, and crop them into 224 × 224. In the training stage, we utilize the Adam optimizer and employ cosine annealing as the optimization scheduler. The initialize the learning rate as 3e-2 for all datasets. The number of training epochs is set to 10 for both CUB-200-2011 and Stanford Cars, while the NABirds are trained for 5 epochs, and the batch size is set to 32. 



The results in Table <ref> are provided by the original paper results, while the results in Tables <ref> and <ref> were reproduced using the source code of the original paper.



 §.§ Comparison with State-of-the-Art Methods



  §.§.§ Closed-set Setting

We compare our proposed DVF with previous competitive methods in a closed-set setting. The comparison results for the CUB-200-2011 and Stanford Cars datasets are presented in Table <ref>, and the results for the NABirds dataset can be found in Table <ref>.

From these tables, it can be observed that our proposed method outperforms other state-of-the-art methods on CUB-200-2011 and NABirds, and achieves competitive performance on Stanford Cars.

Specifically, in comparison with Hyp-ViT <cit.>, the current state-of-the-art on CUB-200-2011, our DVF demonstrates a 3.4% improvement in Recall@1 and a 4.3% improvement over our base framework ViT <cit.>.

The experimental results on the Stanford Cars indicate that our method outperforms the most of existing methods but falls slightly behind HIST <cit.>.

We argue the possible reason may be attributed to the relatively regular and simpler shape of the cars.

In the larger and more challenging NABirds dataset, we observe a significant superiority of the ViT structure over the CNN structure.

Our DVF outperforms the best CNN method by 9.0% on the Recall@1 and improves the performance of the previous leading method ViT <cit.> by 8.1%.






  §.§.§ Open-set Setting

The open-set setting poses greater challenges compared to the closed-set setting due to the unknown subcategories in the test set.

The experimental results for the CUB-200-2011 and Stanford Cars datasets are shown in Table <ref>, while the results for the NABirds dataset are provided in Table <ref>.

The results reveal a consistent trend in both open and closed-set settings: our method outperforms other state-of-the-art approaches on CUB-200-2011 and NABirds, while delivering competitive results on the Stanford Cars dataset.

To be specific, in comparison with Hyp-ViT <cit.> on CUB-200-2011, our DVF exhibits a 4.8% improvement in Recall@1 and a 6.2% enhancement over our base framework ViT <cit.>.

Experimental results on the Stanford Cars dataset show that our method outperforms most existing methods but lags behind FRPT <cit.> by a slight margin.

The results of experiments on NABirds are shown in Table <ref>.

Even in more challenging settings, DVF consistently demonstrates excellent performance, whereas other methods, particularly those based on CNNs, experience a significant drop in performance.

In numerical terms, our DVF surpasses the top-performing CNN method by 24.8% in Recall@1 and enhances the performance of the previously leading method Hyp-ViT <cit.> by 9.0%.

The significant performance improvement can be attributed to the combined effects of the ViT structure and our proposed guidelines in Section <ref>.






 §.§ Ablation Studies and Analysis



  §.§.§ Efficacy of various components

The proposed DVF comprises three essential components: OVF and SVF from the dual visual filtering mechanism, and Discriminative model Training strategy (DMT).

We conducted ablation experiments on these components, and the results are reported in Table <ref>, with the Baseline representing the pure ViT.

The introduction of SVF and OVF improved the Recall@1 performance on the CUB-200-2011 by 1.7% and 3.6% respectively. Besides, the combined use of OVF and SVF results in further performance improvement.

This observation indicates that both OVF and SVF can help DVF generate discriminative features, which are complementary, thereby improving the performance.

Moreover, incorporating DMT into the model variation improved its performance on the CUB-200-2011.



  §.§.§ Number of selected embedding tokens in SVF

The performance comparison of different values of k, representing the number of selected embedding tokens in SVF, is presented in Fig. <ref>.

The performance gradually increases with the number of embedding tokens but decreases when the number exceeds 12.

The decline in performance may be attributed to the fact that as the number k rises, the semantic visual filtering may emphasize the locations of objects or parts rather than focusing on subcategory-specific discrepancies, making it ineffective for FGIR.

Consequently, we have chosen to set k to 12 for all datasets.








  §.§.§ Generalizability of OVF

The proposed OVF is training-free and model-agnostic.

Therefore, we conducted comparative experiments to explore the generalizability of OVF.

The results in Table <ref> show that the performance of both the CNN-based method and the ViT-based method is significantly improved after integrating OVF.

This demonstrates the generality of OVF, as it can aid various methods to emphasize the object, allowing them to concentrate more on subcategory-specific discrepancies and consequently improving performance on FGIR.


























  §.§.§ Importance of the Token Importance Generator

Here, we investigate the influence of the token importance generator on retrieval performance. 

The retrieval results, presented in Table <ref>, demonstrate that the performance declines in the absence of the token importance generator.

The possible reason for the performance degradation is that, without the token importance generator, SVF can generate semantic scores solely based on class token; however, its ability to capture discriminative regions is limited.

To more intuitively reflect the significance of the token importance generator, we conducted a visual experiment to demonstrate its effectiveness, as depicted in Fig. <ref>.

The first column represents the input image, while the second and third columns illustrate the selected tokens without/with token importance, respectively.

Visualization results show that the token importance generator enables DVF to focus more on discriminative areas such as the beak, tail, and color mutation areas. 











  §.§.§ Influence of DMT









To demonstrate the effectiveness of the components in the proposed DMT. We conducted ablation experiments on DMT components, and the results can be found in Table <ref>.

It is observed that both data augmentation and contrastive loss contribute to performance improvement, proving the effectiveness of DMT components in training discriminative models.

Furthermore, the DMT achieves the best results, demonstrating that data augmentation and contrastive loss are complementary to each other.








  §.§.§ Visualization

Upon visualizing the top 10 results obtained from CUB-200-2011 in Fig. <ref>, we observe that DVF excels in retrieving images belonging to the same subcategory across various subcategories, even amidst diverse variations and backgrounds.

There is also a failure case that requires careful observation of the subtle differences between the query image and the returned image.

Additionally, we conduct visualization experiments to illustrate the effectiveness of DVF, as shown in Fig. <ref>.

The original input images are presented in the first column, followed by the second column illustrating the class activation maps computed using GradCAM for the baseline model, which takes the original input images as input.

The third column describes the input image with OVF, while the fourth column shows the class activation map of DVF taking it as input.

The visualization results demonstrate that DVF effectively avoids focusing on background regions and enhances the baseline model’s ability to focus on a more comprehensive region, especially in capturing more detailed discrepancies specific to subcategories, such as head, wings, and tail.



§ CONCLUSION

In this paper, we propose a set of guidelines for designing fine-grained image retrieval (FGIR) models by analyzing the unique characteristics of FGIR tasks and the shortcomings of previous methods.

Following these guidelines, we propose an effective FGIR model and a discriminative training strategy.

Specifically, we propose a visual transformer with an object-oriented visual filtering module, and a semantic-oriented visual filtering module to generate discriminative representation.

Furthermore, we introduce a discriminant model training strategy that combines data augmentation with contrastive loss to enhance the discriminative and generalization capabilities of the model.

The effectiveness of the proposed guidelines is verified through extensive ablation studies.

The experiment results demonstrate that our model, following these guidelines, achieves state-of-the-art performance on three challenging fine-grained image retrieval benchmarks.




ACM-Reference-Format




