Visual Explanation for Deep Metric Learning
Sijie Zhu Taojiannan Yang Chen Chen











































Journal of  Class Files, Vol. 14, No. 8, August 2015
Shell et al.: Bare Demo of IEEEtran.cls for IEEE Journals


























Visual Explanation for Deep Metric Learning
    Sijie Zhu,
        Taojiannan Yang,
        and Chen Chen, Member, IEEE
S. Zhu, T. Yang, and C. Chen are with the Center for Research in Computer Vision, University of Central Florida, Orlando, FL 32816 USA. E-mails: sizhu@knights.ucf.edu, taoyang1122@knights.ucf.edu, chen.chen@crcv.ucf.edu
This work is partially supported by the National Science Foundation under Grant No. 1910844.


    December 30, 2023
======================================================================================================================================================================================================================================================================================================================================================================================================






This work explores the visual explanation for deep metric learning and its applications. As an important problem for learning representation, metric learning has attracted much attention recently, while the interpretation of the metric learning model is not as well-studied as classification. To this end, we propose an intuitive idea to show where contributes the most to the overall similarity of two input images by decomposing the final activation. Instead of only providing the overall activation map of each image, we propose to generate point-to-point activation intensity between two images so that the relationship between different regions is uncovered. We show that the proposed framework can be directly applied to a wide range of metric learning applications and provides valuable information for model understanding. Both theoretical and empirical analyses are provided to demonstrate the superiority of the proposed overall activation map over existing methods. 
Furthermore, our experiments validate the effectiveness of the proposed point-specific activation map on two applications, i.e. cross-view pattern discovery and interactive retrieval. Code is available at <https://github.com/Jeff-Zilence/Explain_Metric_Learning>




Deep metric learning, visual explanation, convolutional neural networks, activation decomposition





















§ INTRODUCTION

















Learning the similarity metrics between arbitrary images is a fundamental problem for a variety of tasks, such as image retrieval <cit.>, verification <cit.>, localization <cit.>, video tracking <cit.>, etc. Recently the deep Siamese network <cit.> based framework has become a standard architecture for metric learning and achieves exciting results on a wide range of applications <cit.>. However, there are surprisingly few works conducting visual analysis to explain why the learned similarity of a given image pair is high or low. Specifically, which part contributes the most to the similarity is a straightforward question and the answer can reveal important hidden information about the model as well as the data. 


Previous visual explanation works mainly focus on the interpretation of deep neural network for classification <cit.>. Guided back propagation (guided BP) <cit.> has been used for explanation by generating the gradient from prediction to input, which shows how much the output will change with a little change in each dimension of the input. Another representative visual explanation approach, class activation map (CAM) <cit.>, generates the heatmap of discriminative regions corresponding to a specific class based on the linearity of global average pooling (GAP) and fully connected (FC) layer. However, the original method only works on this specific architecture configuration (i.e. GAP+FC) and needs re-training for visualizing other applications. Based on the gradient of the last convolutional layer instead of the input, Grad-CAM <cit.> is proposed to generate activation maps for all convolutional neural network (CNN) architectures. Besides, other existing methods explore network ablation <cit.>, the winner-take-all strategy <cit.>, inversion <cit.>, and perturbation <cit.> for visual explanation.



Since verification applications like person re-identification (re-id) <cit.> usually train metric learning models along with classification, recent work <cit.> leverages the classification activation map to improve the performance, but the activation map of metric learning is still not well explored. For two given images, a variant of Grad-CAM has been used for visualization of image retrieval <cit.> by computing the gradient from the cosine similarity of the embedding features to the last convolutional layers of both images. However, Grad-CAM only provides the overall highlighted regions of two input images, the relationship between each activated region of two images is yet to be uncovered. Since the similarity is calculated from two images and possibly based on several similar patterns between them, the relationship between these patterns is critical for understanding the model.





In this paper, we propose an activation decomposition framework for visual explanation of deep metric learning and explore the relationship between each activated region by point-to-point activation response between two images. As shown in Fig. <ref>, the overall activation maps (the second column) of two input images are generated by decomposing their similarity (score) along each image. In this example, the query image (A) has a high/strong activation on both the eyes and mouth areas, but the overall map of the retrieved image (B) only highlights the eyes. Therefore, it is actually hard to understand how the model works only based on the overall maps. 
For image B, the mouth region (green point) has a low activation which means the activation between the mouth and the whole image A is low compared to the overall activation (similarity). 
However, by further decomposing this activation (green point) along image A, that is to compute the activation between the mouth region of image B and each position in image A, the resulting point-specific activation map (green box) reveals that the mouth region of image B still has a high response on the mouth region of image A. This point-specific activation map can be generated for each pixel, which encodes the point-to-point activation intensity for representing the relationship between regions in both images, e.g. eye-to-nose or mouth-to-mouth. Compared with the overall activation map, the point-specific activation map provides much more refined information about the model which is crucial for explanation. 

The main contributions of this paper are summarized as follows.

   
  * We propose a novel explanation framework for deep metric learning architectures based on activation decomposition. It can be served as a white-box explanation tool to better understand the metric learning models without modifying the model architectures, and is applicable to
   
   a host of metric learning applications, e.g. face recognition, person re-identification, image retrieval, etc.
   
  * The proposed point-specific activation map uncovers the point-to-point activation intensity which, to the best of our knowledge, has not been explored by existing methods. Our experiments further show the importance of the point-specific activation map on two practical applications, i.e. cross-view pattern discovery and interactive retrieval. 
   
  * We provide both theoretical and empirical analysis to show the superiority of the proposed overall activation map for deep metric learning over the popular Grad-CAM algorithm.

    
    
    




The remainder of this paper is organized as follows. Section <ref> provides a brief review on existing visual interpretation methods for classification and metric learning. Section <ref> introduces the idea of activation decomposition for visual interpretation on a simple network architecture. 
Section <ref> further extends the framework to more complex architectures for metric learning as a unified solution. Section <ref> presents a detailed comparison between the proposed method and Grad-CAM. 

Section <ref> validates the effectiveness of the overall activation map of our method, and Section <ref> demonstrates the unique advantages of the point-specific activation map on two practical applications.
Finally, Section <ref> concludes the paper. 




§ RELATED WORK






 §.§ Interpretation for Classification


As a default setting, most existing interpretation methods <cit.> are designed in the classification context where the output score is generated from one input image. Among the approaches that do not require architecture change, one intuitive idea <cit.> is to look into the model and see where contributes the most to the final prediction. CAM <cit.> and its variant Grad-CAM <cit.> are among the most widely used approaches, but recent works <cit.> simply consider CAM as a heuristic linear combination of convolutional feature maps which is limited to the original architecture, global average pooling (GAP) and one FC layer <cit.>. Specifically, to generate CAM for standard VGG <cit.> with global max pooling (GMP) and three FC layers, <cit.> have to replace this architecture with GAP+FC and re-train the model.

In this paper, we show that CAM is actually a special case of activation decomposition on the specific architecture, and the activation decomposition idea applies to more complex architectures without re-training, even beyond the classification problem.


Grad-CAM <cit.> is considered as a generalization of CAM for arbitrary CNN architectures, but the original paper only provides the proof on CAM's architecture. We show that Grad-CAM is not equivalent to activation decomposition for some architectures (Section <ref>). 

Another direction of explanation is to check the black box model by modifying the input and observing the response in the prediction. A representative method <cit.> aims to optimize the blurred region and see which region has the strongest response on the output signal when it is blurred. Perturbation optimization is applicable for any black box model, but the optimization is computationally expensive.





 §.§ Interpretation for Metric Learning

As ignored by most existing methods, the explanation for metric learning is important for a wide range of applications <cit.>, e.g. weakly supervised localization. In this work, we further shows its potential significance on diagnosis of metric learning losses and new applications including cross-view pattern discovery and interactive retrieval for person re-identification <cit.> and face recognition. Gradient-based methods, e.g. guided BP <cit.> can be adopted to metric learning, but recent works <cit.> claim that gradient can be irrelevant to the model and guided-BP fails on the sanity check <cit.>. Among methods that pass the sanity check, Grad-CAM has been used for visualization of image retrieval in <cit.>, but no quantitative result is provided in the paper. Instead of adding more gradient heuristics like Grad-CAM++ <cit.>, we propose to explain the result of Grad-CAM under the decomposition framework. As shown in Section <ref>, by removing the heuristic step in Grad-CAM, the meaning of the generated activation map can be clearly explained using the proposed framework. 

Recent work <cit.> also explores visualization techniques on metric learning, but it only provides qualitative results on several specific architectures. Another variant <cit.> replaces the classification score with triplet loss in Grad-CAM framework and deploys weights transfer from the training data in the inference phase. The method requires three samples to form the triplet and its usage in the inference phase is dependent upon the specifically designed transfer procedure, thus is not a general method given only the pretrained model. Moreover, all the aforementioned methods only focus on the overall activation map which is not satisfactory for understanding the model as discussed in Section <ref>. Apart from white-box explanation, black-box optimization method, such as perturbation<cit.>, needs reformulation for metric learning and can be more computationally expensive if the point-to-point activation map is desired. 



§ ACTIVATION DECOMPOSITION ON A SIMPLE ARCHITECTURE




To better introduce our method, we first review the formulation of CAM and illustrate our idea based on a simple architecture (CNN+GAP in Fig. <ref>) for metric learning. Note that the proposed method is not limited to this architecture, the general formulation will be presented in Section <ref>. As shown in Eq. <ref>, CAM is actually a spatial decomposition of the prediction score of each class, and the original method only applies for CNN with GAP and one FC layer without bias. The decomposition clearly shows how much each part of the input image contributes to the overall prediction of one class and provides valuable information about how the decision is made inside the classification model. The idea is based on the linearity of GAP:

    S_c = ∑_kω_k,c(1/Z∑_i,jA_i,j,k) = 1/Z∑_i,j(∑_kω_k,cA_i,j,k).

S_c denotes the overall score (before softmax) of class c and ω_k,c is the FC layer parameter for the k-th channel of class c. A_i,j,k denotes the feature map of the last convolutional layer at position (i,j), and Z is the normalization term of GAP. Here, Z equals to m× n, which is the spatial size of feature map A∈^m× n × p. In fact, the result ∑_kω_k,cA_i,j,k can be considered as a decomposition of S_c along (i,j). For a two-stream Siamese-style architecture (see Fig. <ref>) with GAP and the cosine similarity metric (S) for metric learning applications (e.g. image retrieval), we propose to do decomposition along (i,j,x,y) so that the relationship between different parts of two images are uncovered:

    S     =  (E^q· E^r)/|E^q||E^r|
        =  ∑_kGAP(A^q_k)GAP(A^r_k) /|E^q||E^r|
        =1/Z∑_k(∑_i,jA^q_i,j,k∑_x,yA^r_x,y,k) 
        =1/Z∑_i,j,x,y(∑_kA^q_i,j,kA^r_x,y,k).

Here, E^q,E^r denote the embedding feature of query and reference image. |E| denotes L2 norm of E and · is the inner product. The normalization term Z equals to m^qn^qm^rn^r|E^q||E^r|, if the spatial size of A^q,A^r are m^q× n^q, m^r× n^r. We use (i,j) and (x,y) for different streams because some applications such as cross-view image retrieval <cit.> may have different image sizes for two streams. A denotes the feature map of the last convolutional layer. The superscripts q and r respectively denote the query and retrieved images in this paper. For each query point (i,j) in the query image, the corresponding point-specific activation map in the retrieved image is given by ∑_kA^q_i,j,kA^r_x,y,k which is the contribution of features at (i,j,x,y) to the overall cosine similarity. Like CAM and Grad-CAM, bilinear interpolation is implemented to generate the contribution of each pixel pair and the full resolution map. The overall activation maps of two images are generated by a simple summation along (i,j) or (x,y), e.g. ∑_i,j(∑_kA^q_i,j,kA^r_x,y,k). Although we only show the positive activation in the map, the negative value is also available for counterfactual explanation like <cit.>.

Recent works <cit.> have highlighted the superiority of L2 normalization, we therefore adopt the cosine similarity S as the default metric. With L2 normalization, the squared Euclidean distance D equals to 2-2S so that S and D are equivalent as metrics. Although there are still a number of methods <cit.> utilizing the Euclidean distance without L2 normalization, <cit.> shows that the cosine similarity performs well as the evaluation metric. We further empirically show that the cosine similarity works well for explanation in this case (Section <ref>).




§ GENERALIZATION FOR COMPLEX ARCHITECTURES


Recent metric learning approaches usually leverage more complex architectures to improve performance, e.g. adding several FC layers after the flattened feature or global pooling. Although different metric learning applications have different head architectures (e.g. GAP+FC) on CNN, the basic components are highly similar. 
Given the introduction of activation decomposition on a simple architecture in the previous section, we present an unified extension to make our method applicable to most existing state-of-the-art CNN architectures for various applications, including image retrieval <cit.>, face recognition <cit.>, person re-identification <cit.>, and image geo-localization <cit.>. 
Specifically, in Section <ref>, we address linear components by considering them together as one linear transformation. Then Section <ref> focuses on transforming nonlinear components to linear ones in the inference phase.



 §.§ Linear Component





For the last convolutional layer feature A∈^m× n× p (m, n and p denote the width, height, and number of channels), its GAP (global average pooling) is equivalent to the flattened feature Â∈^mnp multiplied by a transformation matrix T_GAP∈^p × mnp:  

    GAP(A) = 1/mn∑_i,jA_i,j =  T_GAPÂ.

Here (i,j) denotes the spatial coordinates. By reshaping T_GAP to p× m × n× p as T^*_GAP, the matrix is given by:

    T_GAP^*(k',i,j,k) = {1/mn       k'=k 
    
    0        k'≠ k 
    .

The matrix T_GAP is simply calculated by reshaping T^*_GAP to p× mnp. 
Therefore, the GAP can be considered as a special case of flattened layer multiplied by a transformation matrix.
Without loss of generality, we consider a two-stream framework with flattened layer followed by one FC layer with weights Ŵ∈^l × mnp and biases B∈^l (l denotes the length of the feature embedding vector). Since the visual explanation is generated in the inference phase when typical components such as FC layer and batch normalization (BN) are linear, all the linear components together are formulated as one linear transformation g(Â) = ŴÂ + B = ∑_i,j W_i,jA_i,j+B in the FC layer. Here W_i,j∈^l× p and A_i,j∈^p denote the weights matrix and feature vector corresponding to position (i,j). Although B is ignored in CAM, we keep it as a residual term in the decomposition. Then Eq. <ref> is re-formulated as:

    SZ 
            =g^q(A^q)· g^r(A^r) 
        = (∑_i,jW^q_i,jA^q_i,j+ B^q)·(∑_x,yW_x,y^rA^r_x,y+B^r)
       = ∑_i,j,x,y(W^q_i,jA^q_i,j)· (W^r_x,yA_x,y^r)_point-to-point activation +  ∑_i,j(W^q_i,jA^q_i,j· B^r) 
          + ∑_x,y(W^r_x,yA^r_x,y· B^q)+B^q· B^r.


Here Z denotes the normalization term for cosine similarity (L2 norm of embedding features), and · is the inner product. As can be seen from Eq. <ref>, the decomposition of S has four terms, and the last three terms contain the bias B. The first term clearly shows the activation response for location pair (i,j, x,y) and the point-specific map for query position (i,j) is given by I(x,y) = (W^q_i,jA^q_i,j)· (W^r_x,yA_x,y^r). The second and third terms correspond to the activation between one image and the bias of the other image. Although they can be considered as negligible bias terms, they actually contribute to the overall activation map. For the overall activation map of the query image, the second term can be included, because W^q_i,jA^q_i,j· B^r varies at different (i,j) positions while the third and last terms stay unchanged. Similarly, the first and third terms are considered when calculating the overall map for the retrieved image. We investigate both settings about the bias term (i.e. w or w/o) on the overall activation map and they are referred to as “Decomposition": (W^q_i,jA^q_i,j)· (∑_x,yW^r_x,yA^r_x,y), and “Decomposition+Bias" : (W^q_i,jA^q_i,j)· (∑_x,yW^r_x,yA^r_x,y+B^r) in Section <ref>. The last term is pure bias which is the same for every input image. Therefore, we ignore this term if not mentioned. 













 §.§ Non-linear Component


Non-linear transformation increases the difficulty of activation decomposition, our idea is to 

find its approximate linear transformation which can be directly integrated into the formulation of Section <ref>. For a non-linear transformation f(x) with its current input data x ∈ [x_0-δ, x_0+δ] (δ is a small number), the first two terms of the Taylor expansion of f(x) can be considered as its linear transformation in the inference phase:

    f(x) = f(x_0) + f'(x_0)/1!(x-x_0)+... 
    ≈ f(x_0) + f'(x_0)(x-x_0).

Therefore, our method can generalize to more complex architectures 

as long as the network is differentiable, since the formulation of gradient f'(x_0) can be computed by chain rule.

In this paper, we shed light on the practical guidelines for deploying our method on image retrieval, face recognition, person re-identification, and cross-view geo-localization frameworks. For these popular applications, the most widely used non-linear components – global maximum pooling (GMP) and rectified linear unit (ReLU) – can be transformed as linear operations in the inference phase by multiplying a mask[The mask is computed once for each input image.].

In the inference phase, GMP can be considered as a combination of a maximum mask M and GAP as shown in Fig. <ref>. 
The transformation matrix of GMP is given by

    T^*_GMP = mn(T_GAP^*⊙ M).

M∈ R^m× n × p is the maximum matrix of A where only the maximum position of each channel has nonzero value as 1. T_GMP is computed by reshaping T^*_GMP to p× mnp.
The result of Hadamard product (M⊙ A) is considered as the new feature map which can be directly applied to Eq. <ref>.
Similarly, by adding a mask for ReLU, the FC layer with ReLU can be included in W and B in the inference phase. 







§ COMPARISON WITH GRAD-CAM


In this section, we provide a comprehensive comparison between the proposed method and Grad-CAM with theoretical derivation. Apparently, the proposed method can generate point-specific activation map which uncovers more fine-grained information than Grad-CAM, as Grad-CAM only generates  overall activation map. We will further show the relationship between our overall activation map and Grad-CAM. As another way for generating overall activation map on more complex architectures, Grad-CAM follows a heuristic design without a clear meaning for the generated map. When we calculate the activation map based on Grad-CAM, what do we get? In <cit.>, it has shown that Grad-CAM is equivalent to CAM on GAP based architecture, while it is not true on other architectures. For a classification architecture with flattened feature and one FC layer, the prediction score for class c is formulated as S_c=∑_i,j,kW_i,j,k,cA_i,j,k, where W is the reshaped weights of the FC layer following the shape of the last convolutional feature (A). From our activation decomposition perspective, the contribution of each position (i,j) is clearly given by ∑_kW_i,j,k,cA_i,j,k, while the Grad-CAM map for class c at (i,j) position is given by:

    GradCAM_i,j,c    = ∑_k A_i,j,kGAP(∂ S_c/∂ A_k)
        =∑_k A_i,j,kGAP(W_k).


In this case, the result of Grad-CAM is different from activation decomposition because of the GAP operation in Eq. <ref>. For architectures using GMP, they are also different, because only the maximal value of each channel contributes to the overall prediction score with the activation decomposition framework, while Grad-CAM puts the same weight for features at all positions. 
Concretely, for a classification architecture with GMP and one FC layer (W∈^p× l and no bias), the prediction score for class c is given by S_c=∑_k max_i,j(A_:,:,k)W_k,c=∑_i,j,kA_i,j,k M_i,j,kW_k,c, where M is the maximum matrix in Eq. <ref>. The decomposition of position (i,j) is ∑_kA_i,j,kM_i,j,kW_k,c and only the maximum position of each channel has a nonzero value. However, for Grad-CAM, the activation map is given by:

    GradCAM_i,j,c=GAP(∂ S_c/∂ A)A_i,j =1/mn∑_kA_i,j,kW_k,c,

where all (i,j) positions have a nonzero weight resulting in a more scattered activation map.



Although <cit.> empirically shows that the heuristic GAP step can help improve the performance of object localization, this step makes the meaning of the generated map unclear. By removing this step, which means combining the gradient and feature directly as A_i,j,k∂ S_c/∂ A_i,j,k, the modified version would generate the same result as activation decomposition for flattened and GMP based architectures. As for the metric learning architecture discussed in Section <ref>, the Grad-CAM map of query image is written as (the detailed derivation is presented in Appendix <ref>):

    GradCAM_i,j =   1/Z(∂ (E^q/|E^q|)/∂ E^qGAP(W^q) A^q_i,j) 
       ·(∑_x,yW^r_x,yA^r_x,y+B^r)_E^r.

E^q and E^r denote the embedding vectors of the query and retrieved images before L2 normalization and Z is the normalization term. The gradient term ∂ (E^q/|E^q|)/∂ E^q, which comes from the L2 normalization, would put less weights on the dominant channels so that the generated activation map becomes more scattered as shown in Fig. <ref> (please also refer to Appendix <ref> for proof). It can be removed by calculating gradient from E^q· E^r without L2 normalization, denoted as “Grad-CAM (no norm)". If we remove the gradient term as well as the GAP term, the result is actually equivalent to the overall map of our “Decomposition+Bias" given by the first two terms of Eq. <ref> as (W^q_i,jA^q_i,j)· (∑_x,yW^r_x,yA^r_x,y+B^r).




§ EVALUATING OVERALL ACTIVATION MAP



To validate the advantage of the proposed method, we provide results of human evaluation as well as several practical applications. We first verify the effectiveness of the proposed overall activation map with weakly supervised localization and human evaluation (Sections <ref> and <ref>). We also show how the activation map brings potential insights on the generalization ability of different metric learning models (Section <ref>). 













 §.§ Weakly Supervised Localization



Following previous works <cit.> on visual explanation, we conduct weakly supervised localization experiment in the context of metric learning. 
Among the popular datasets for image retrieval, CUB <cit.> is a challenging dataset with over 10,000 images from 200 bird species, and the bounding box annotation is available for localization evaluation. We first train a metric learning model on CUB using the state-of-the-art image retrieval method <cit.>. Then the weakly supervised localization is conducted to show the differences between the variants of the proposed visual explanation method. 

We first generate the activation maps of different methods, then the mask is generated by segmenting the heatmap with a threshold. Following previous works <cit.>, the bounding box that covers the largest connected component in the mask is generated as the prediction. A predicted bounding box is considered correct if the IoU (Intersection over Union) between the bounding box and ground-truth is greater than 0.5. 


Apparently, different thresholds would lead to different predictions, we thus apply a grid search on the threshold to find the best performance of each competing method for a fair comparison. Table <ref> lists the localization accuracy on CUB validation set under different thresholds.

For convenience, we report the best result of each method in Table <ref>, including the result of CAM (classification based method) adopted from the original paper <cit.>.


In these two tables, “Grad-CAM (no norm)" means Grad-CAM with gradient computed from the product of two embedding vectors without normalization and “Decomposition+Bias" denotes the first two terms of Eq. <ref>. Since the architecture of <cit.> is based on GMP, “Grad-CAM (no norm)" is not equivalent to “Decomposition+Bias" as shown in  Section <ref>. The proposed framework outperforms Grad-CAM for metric learning as well as CAM based on classification. As justified in Section <ref>, computing the gradients before normalization does improve the performance of Grad-CAM by a large margin. The qualitative result in Fig. <ref> also illustrates the scattering effect of Grad-CAM with normalization as demonstrated in our theoretical analysis (Appendix <ref>). The information in the bias term does not help the weakly supervised localization, as witnessed by the performance of “Decomposition+Bias". 






 §.§ Human Evaluation



Following the work of Grad-CAM <cit.>, we also conduct human evaluation on the widely used platform (AMT: Amazon Mechanical Turk) to further evaluate the quality of overall activation maps generated by different methods. 
Based on Table <ref>, two comparison pairs are formed as “Decomposition" vs. “Grad-CAM" and “Decomposition" vs. “Grad-CAM (no norm)". For each comparison pair, the AMT raters are asked to carefully check the visual explanation generated from two different methods as shown in Fig. <ref>. Each visual explanation contains an image pair with birds from the same species and the top activated regions (2% pixels) are highlighted based on the overall activation map of one competing method. We denote these two methods as robot A and robot B in the questionnaire. The raters will decide which robot they think is more reasonable in a five-point Likert scale as shown in Fig. <ref>. The five scales denote method A is “clearly more", “slightly more", “equally", “slightly less", “clearly less" reasonable than method B.

720 evaluations of comparison pairs are collected from at least 40 different AMT workers for two comparison pairs (half for each) and the result is consistent with the outcome of Section <ref> as shown in Fig. <ref>. Over 60% evaluations support that the proposed method is clearly more reasonable than the vanilla Grad-CAM. Removing the normalization for the gradient computing does improve the quality of Grad-CAM's activation map significantly, but the proposed method still holds more supportive evaluations (41.1% vs. 25.2%) in the AMT test.





 §.§ Model Diagnosis


A large number of loss functions have been proposed for metric learning <cit.>, while only the performance and embedding distribution are evaluated. We show that the overall activation map can also help evaluate the generalization ability of different metric learning methods. 










We follow the setting of <cit.> and train two metric learning models with Multiple Similarity (MS) loss <cit.> and Triplet loss <cit.>, respectively. As shown in Table <ref>, although they have almost the same accuracy on the training set, there is a big gap between their generalization abilities on the validation set. Before checking the result on the validation set, is there any clue in the training set for such gap? Despite the similar training accuracies, the predictions of two models are actually based on different regions from the activation maps generated by our activation decomposition framework (see Fig. <ref>). The activation maps also provide valuable insights on the model generalization ability to the validation/test set. It is evident from Fig. <ref>, the model with MS loss is more focused on the salient object area than the one with Triplet loss, which leads to better validation performance as confirmed in Table <ref>. The quantitative results of localization in Table <ref> also support the fact that “Triplet" model is more likely to focus on the background region rather than the object as we witness the localization accuracy drop for Triplet loss. Although “Decomposition+Bias" can be more sensitive to different loss functions, implying that the bias term does provide valuable information on the overall map, both settings of the proposed method have the same trend on accuracy. Therefore, our activation decomposition framework can shed light on the generalization ability of loss functions for metric learning, thereby providing an useful tool for model diagnosis. 




§ EVALUATING POINT-SPECIFIC ACTIVATION MAP







The proposed activation decomposition framework not only generates the overall activation map, but also can produce the point-specific activation map for fine-grained visual explanation of metric learning. In this section, we introduce two applications to demonstrate the importance of the proposed point-specific activation map.





 §.§ Application I: Cross-view Pattern Discovery



When metric learning is applied to cross-view applications, e.g. image retrieval <cit.>, the model is capable of learning similar patterns in different views which may provide geometric information of the two views, e.g. the camera pose or orientation information. We conduct orientation estimation experiment to show the advantage of point-specific activation map compared with the overall activation map on providing geometric information based on cross-view patterns. In our experiment, we take street-to-aerial image geo-localization <cit.> as an example.

The objective of cross-view image geo-localization is to find the best matched aerial-view image (with GPS information) in a reference dataset for a query street-view image. We conduct the experiment on CVUSA <cit.>, which is the most popular benchmark for this problem containing 35,532 training pairs and 8,884 test pairs. Each pair consists of a query street image and the corresponding orientation-aligned aerial image at the same GPS location. For example, as shown in Fig. <ref>, the left two images (street-view panorama and aerial-view images) are the original aligned image pair and the yellow line denotes the 0^∘ which corresponds to the South direction (180^∘ corresponds to the North direction). The location of panorama always lies at the center of the corresponding aerial image. We use [0,360]^∘ to denote different angles as marked on those two images.



We train a simple Siamese-VGG <cit.> network following the modified triplet loss in <cit.>. During the training, if the image pairs are not aligned (e.g. randomly rotate the aerial view images), the activation map can be used for orientation estimation. Specifically, we train the Siamese-VGG with randomly rotated aerial images so that the model is approximately rotation-invariant and so is the overall activation map for aerial image. In the example of Fig. <ref>, the overall activation maps for the original aligned pair are first generated to show the highlighted regions contributing to the similarity score, and both views focus on the road areas in this example. The most activated regions are highly relevant in both views and most of them are similar patterns. When we randomly rotate the aerial image (30^∘ in this example and is denoted as the ground truth rotation angle in Fig. <ref>), the aerial view activation map still focuses on the road area which is relevant to the street view. 



In this paper, we simply use the pixel with the maximum activation value for orientation estimation, because the most activated areas (highlighted in cyan boxes in Fig. <ref>) from two views are likely to be relevant. For the street-view image, the selected pixel lies in the angle of 90.2^∘ (the cyan line) based on the angle marks on the left aligned image. For the rotated aerial view, the selected pixel lies in the angle of 302.1^∘ (the cyan line). Since the overall map contains multiple activated regions, the selected (maximum activation) pixels in both views actually do not correspond to the same object, which reveals one disadvantage of using only the overall activation map. In this example, the estimated angle is 302.1^∘-90.2^∘ = 211.9^∘, which is not correct. However, for the selected pixel (the one with the maximum activation value) in the street-view image, if we generate its corresponding point-specific activation map on the aerial-view image (as shown on the very right of Fig. <ref>), the new selected pixel (maximum activation) on this point-specific activation map lies in the angle of 122.1^∘ (the red line). Then, the estimated angle is calculated as 122.1^∘-90.2^∘=31.9^∘, which is very close to the ground truth (30^∘). This demonstrates the advantage of the point-specific activation decomposition for finding more fine-grained information in this application. 


We present two more visual examples of orientation estimation in Fig. <ref>.








We also present the quantitative results in Fig. <ref>. The angle error is computed by . We then add or subtract 360^∘ to the error to fit it in the range of [-180,180]^∘ if the absolute value of the error is greater than 180^∘. For example, when the error is 359^∘, we subtract it by 360^∘ and get -1^∘ as the error. This is a reasonable setting adopted from the previous work <cit.>. As evident from Fig. <ref>, the point-specific activation map significantly outperforms the overall map for cross-view orientation estimation. Point-specific activation based orientation estimation has over 16% samples (a total of 8,884 test sample pairs in the CVUSA dataset) with angle error less than ± 3.5^∘ (the red bar at 0^∘ in Fig. <ref>), while overall activation map based method has less than 12%. The result further validates the superiority of the point-specific map compared with the overall map.
Moreover, we also provide a demo[Please check the demo (the “Geo-localization" section) at <https://github.com/Jeff-Zilence/Explain_Metric_Learning>] to show how the point-specific map changes according to the query pixel. Failure samples of both methods tend to have an angle error of 180 degree, because the activation maps of both views are likely to focus on roads which are symmetric in aerial images. 









 §.§ Application II: Interactive Retrieval








Verification applications like face recognition and person re-id usually retrieve the most similar images to the query image. However, we may be more interested in a specific RoI (region of interest) than the overall image for certain circumstances, for example finding people with a similar bag or pair of shoes (i.e. RoI) in surveillance applications.

Since our framework provides the point-to-point activation map, it can be a reasonable tool for measuring partial similarity to serve this purpose. 

With the partial similarity, we are able to interactively retrieve images with only parts/regions similar to the query image instead of the overall most similar images. 

As shown in Fig. <ref>, the point-specific activation generated by the proposed method works well on retrieving people with similar clothes and faces with similar beard.

No explicit supervision is adopted except the pre-trained retrieval model.

Specifically, we follow the pipeline of recent approaches on face recognition <cit.> and person re-id <cit.>. For re-id, the model is trained and evaluated on Market-
1501 <cit.> where some validation images only contain a small part of a person. 
Although the model is trained with Euclidean distance without L2 normalization, cosine similarity still works well as the evaluation metric. For face recognition, we take the trained model on CASIA-WebFace <cit.> and evaluate it on FIW <cit.> where the face identification and kinship relationship are available.



The interactive search is conducted by simply matching the equivalent partial feature in Eq. <ref> (W^q_i,jA^q_i,j) with the reference embedding features. We first compute W^q_i,jA^q_i,j as the feature of each position in the last convolutional layer, and a bilinear interpolation is adopted to generate the feature for every pixel of the original image. For a point of interest (i,j), we compute  the cosine similarity between the calculated feature on (i,j) and the embedding feature of the reference images as the point-specific similarity (note that the equivalent feature is normalized with the l_2 norm of the original embedding as |E^q|). In this case, the embedding features of the reference dataset do not have to be recomputed. This similarity can be also considered as 
the summation of the values in the point-specific map corresponding to (i,j). In the case of Eq. <ref> (CNN+GAP), the similarity can be simplified as ∑_x,y(∑_kA^q_i,j,k A^r_x,y,k)/|E^q||E^r| (E^q and E^r denote the original embedding features of the query and retrieved images). If the ROI is very large, we can merge multiple equivalent features in the ROI to generate better result. Since all the objects have similar size in the face and re-id datasets, we simply adopt the equivalent feature of the center pixel in the ROI.















 
 



Apparently, the images retrieved by the original feature are likely to be from the same identity as the query image, because the models are trained for identification task. However, the proposed interactive retrieval is able to retrieve different images based on the same query image with different RoIs as shown in Fig. <ref>. To further quantitatively evaluate the advantage of the proposed point-specific map, we conduct human evaluation on the overall retrieval (based on the original feature) vs. the interactive retrieval (based on the point-specific map). 

Similar to Section <ref>, the AMT workers are asked to decide which retrieval result is better given a specific RoI (see Fig. <ref>). We conduct experiments on both face verification and person re-id tasks. 360 evaluations of comparison pairs in total are collected from at least 40 different AMT workers. As shown in Fig. <ref>, the interactive retrieval holds considerably more supportive evaluations than the overall retrieval (57.5% vs. 28.0% on face, and 47.5% vs. 31.2% on person re-id), which demonstrates the effectiveness of the proposed point-specific activation map. 

However, we do find some failure cases as shown in the last row of Fig. <ref> with the blue bounding box. There is no shoes in the failure image, but why is this image retrieved with a top rank (i.e. 6th rank)? We provide an in-depth analysis from the perspectives of both overall and point-specific activation maps in Fig. <ref>.
In the overall activation map, three regions of the query image have a high activation on the retrieved image. 

From the point-specific activation maps, there is a high activation on the purse in the retrieved image corresponding to the shoes in the query image. This might be because of their similar red color and the arm of the retrieved person may appear like a leg from a specific viewpoint. This example also validates the importance of the point-specific activation map generated by our framework for explanation.


Comparison with Cropping/Masking. One may argue that another possible way to achieve interactive retrieval is to crop or mask out a specific RoI. Then we can generate the feature with the specific part of image and use this feature as query to retrieve images. However, generally speaking, this operation will cause the feature to be very different from features of the original images, thus would not generate satisfactory retrieval results. And we validate this by providing the qualitative comparison of retrieval results in Figs. <ref> and <ref>.




For face verification, flattened features followed by FC layers are usually used, because the input image size is relatively small and flattened features reserve all the information in the last convolutional layer. However, this architecture does not allow change in the input size, so one may need to resize the image to the original size after cropping a specific RoI. Another way is to assign 0 for all pixels outside the RoI. The comparison in Fig. <ref> shows that cropping with resizing or masking are not able to generate satisfactory results.

For person re-identification, GAP is usually used so that we do not need to resize the cropped image. The results in Fig. <ref> also show the superiority of the proposed method over cropping/masking. 




§ CONCLUSION


We propose a simple yet effective framework for visual explanation of deep metric learning based on the idea of activation decomposition. The framework is applicable to a host of applications, e.g. image retrieval, face recognition, person re-id, geo-localization, etc. Experiments show the importance of visual explanation for metric learning as well as the superiority of both the overall and point-specific activation maps generated by the proposed method. Furthermore, we introduce two applications, i.e. cross-view pattern discovery and interactive retrieval, which reveal the importance of the point-specific activation map for explanation. Our work also points to interesting directions in exploring the point-specific activation maps for fine-grained information discovery and analysis. 










§ GRAD-CAM FOR METRIC LEARNING


For metric learning architecture discussed in Section <ref> of the main paper, the similarity is formulated as S=E^q· E^r/|E^q||E^r|, where E^q∈^l and E^r∈^l are the embedding vectors of the query and retrieved images. |x| denotes the L2 norm and a· b is the inner product of a and b. The Grad-CAM map of the query image is given by:

    GradCAM_i,j = GAP(∂ S/∂ A^q) A^q_i,j

With the gradient chain rule, the gradient is written as:

    ∂ S/∂ A^q   
        = ∂(E^r)^TE^q/|E^q||E^r|/∂ A^q
        = (E^r/|E^r|)^T∂ (E^q/|E^q|)/∂ E^q∂ E^q/∂ A^q
        =(E^r/|E^r|)^T∂ (E^q/|E^q|)/∂ E^qW^q

By expanding E^r as ∑_x,yW^r_x,yA^r_x,y+B^r (Section <ref>)
and merging Eq. <ref> with Eq. <ref>, the Grad-CAM map is reformulated as:

    GradCAM_i,j    = GAP((E^r/|E^r|)^T∂ (E^q/|E^q|)/∂ E^qW^q)A^q_i,j
       = 1/Z(∑_i^*,j^*(E^r)^T∂ (E^q/|E^q|)/∂ E^qW_i^*,j^*^q)A^q_i,j
       =1/Z(E^r)^T∂ (E^q/|E^q|)/∂ E^q(∑_i^*,j^*W_i^*,j^*^q)A^q_i,j
       =1/Z(∑_x,yW^r_x,yA^r_x,y+B^r)
         ·(∂ (E^q/|E^q|)/∂ E^qGAP(W^q)A^q_i,j)
       = 1/Z(∂ (E^q/|E^q|)/∂ E^qGAP(W^q)A^q_i,j)·(∑_x,yW^r_x,yA^r_x,y+B^r)

Here the Z is the normalization term for simplicity. ∂ E/|E|/∂ E is the l× l Jacobian matrix given by:

    ∂ (E/|E|)/∂ E = (∂ (E_i/|E|)/∂ E_j)_i,j= {1/|E|(1-E_i^2/|E|^2)          i=j
    
        -E_iE_j/|E|^3       i≠ j
        .

1/|E| is the normalization term. The  ∂ E/|E|/∂ E term can be removed, if we compute the gradient from E^q· E^r instead of E^q· E^r/|E^q||E^r|.  For the dominant channel i, the weight (1-E^2_i/|E|^2) is small resulting in a more scattered activation map.



§ QUALITATIVE RESULTS

In Fig. <ref>, we show an example of human evaluation interface for person re-identification. We also provide extra qualitative results for cross-view pattern discovery, interactive retrieval (both face verification and person re-identification) in Figs. <ref>, <ref>, <ref>.













































IEEEtran





















Michael Shell
Biography text here.



John Doe
Biography text here.






Jane Doe
Biography text here.



















