Directional Statistics-based Deep Metric Learning for Image Classification and Retrieval
Xuefei Zhe Shifeng Chen Hong Yan

































































































Deep distance metric learning (DDML), which is proposed to learn image similarity metrics in an end-to-end manner based on the convolution neural network, has achieved encouraging results in many computer vision tasks. 

L2-normalization in the embedding space has been used to improve the performance of several DDML methods. 

However, the commonly used Euclidean distance is no longer an accurate metric for L2-normalized embedding space, i.e., a hyper-sphere. 

Another challenge of current DDML methods is that their loss functions are usually based on rigid data formats, such as the triplet tuple.

Thus, an extra process is needed to prepare data in specific formats.

In addition, their losses are obtained from a limited number of samples, which leads to a lack of the global view of the embedding space.

In this paper, we replace the Euclidean distance with the cosine similarity to better utilize the L2-normalization, which is able to attenuate the curse of dimensionality. 

More specifically, a novel loss function based on the von Mises-Fisher distribution is proposed to learn a compact hyper-spherical embedding space.

Moreover, a new efficient learning algorithm is developed to better capture the global structure of the embedding space.  

Experiments for both classification and retrieval tasks on several standard datasets show that our method achieves state-of-the-art performance with a simpler training procedure.  

Furthermore, we demonstrate that, even with a small number of convolutional layers, our model can still obtain significantly better classification performance than the widely used softmax loss.




Deep distance metric learning, Directional statistics, Image retrieval, Image similarity learning 




Directional Statistics-based Deep Metric Learning for Image Classification and Retrieval
    Xuefei Zhe, 
        Shifeng Chen, 
        and Hong Yan, Fellow, IEEE

Xuefei Zhe and Hong Yan are with the Department of Electronic Engineering, City University of Hong Kong, Hong Kong.

Shifeng Chen is with Shenzhen Institutes of Advanced Technology, CAS, China.
Corresponding author: Xuefei Zhe (e-mail: xfzhe2-c@my.cityu.edu.hk).
    ^⋆ E-mail: mailto:Jordan.Collier@csiro.auJordan.Collier@csiro.au

Accepted 2018 February 23. Received 2018 February 23; in original form 2017 June 16
==================================================================================================================================================================================================================================================================================================================================================





























§ INTRODUCTION




By combining deep learning with classical distance metric learning, deep metric learning achieves exciting results on many visual tasks. 

For example, by introducing the triplet loss to the deep learning framework, deep metric learning is found to be effective in face verification <cit.>, person re-identification <cit.>, 3D object retrieval <cit.> and image retrieval <cit.>. 

This new combination is also known as deep distance metric learning (DDML) or deep metric learning (DML). 

Among many DDML methods, the triplet embedding is the most widely used one.

For instance, deep metric learning with triplet shows competitive results on fine-grained visual categorization (FGVC) tasks <cit.>. 

FaceNet <cit.> uses the triplet loss with L2-normalization to handle face related tasks better. 

The work in <cit.> achieves the state-of-the-art performance on the re-identification problem. 

Besides triplet based methods, many other deep metric learning approaches have been proposed in recent years, e.g., the quintuplet loss <cit.> and lifted structure loss <cit.>.

Most existing deep metric learning methods are designed based on the Euclidean distance. 

Several recent studies <cit.> use L2-norm to normalize the embedding space. 

Though L2 normalization as an effective method is widely used to deal with the curse of dimensionality, little attention is paid to the fact that the normalization process projects a p-dim Euclidean space to a high dimension sphere, 𝕊^p-1. 

In such a manifold, the Euclidean distance is no longer an accurate measurement. 

Therefore, it is reasonable to exploit geometric properties of the manifold, the hyper sphere here, for applying machine learning models to data with the unit norm <cit.>. 


Although there is little attention paid to the conflict between the Euclidean metric and the spherical embedding space in DDML, the studies on the data with the unit norm have achieved promising results in several disciplines <cit.>, such as image clustering, text mining and gene expression analysis. 

This kind of data is also known as directional data for which the “direction” of data contains richer information than the “magnitude”. 

It has already been shown by many studies that direction can represent data better than magnitude <cit.>.


Besides the metric problem, it is widely complained that training with deep metric learning models is usually more complicated compared with the softmax loss.

Most of existing DDML methods rely on specific mini-batch formats, such as triplets <cit.> and n-pair tuples <cit.>. 

Preparing these formats is very time-consuming.

Besides, the pair selection strategies have a significant influence on the final performance.

Other methods such as the algorithm in <cit.> do not require preparing data in any formats. However, extra steps are needed to solve their local facility function.


To address the above issues, we first replace the Euclidean distance in deep metric learn with the cosine similarity which is more suitable for L2-normalized embedding space. 

Then, by introducing the directional distribution, a novel deep metric learning model is proposed. 

More specifically, the von Mises-Fisher distribution, which can be treated as the Gaussian distribution for spherical data, is used to defined a new loss function named von Mises-Fishes loss (vMF loss) for our model. 

Besides the loss function, an alternative learning algorithm is proposed to efficiently train our model. 

Extensive experiments on both classification and retrieval tasks show that our method achieves the state-of-the-art performance with an simple training process. 

More over, our method obtains a better performance with shallow convolutional neural networks, which indicates our method has a wide potential use for many mobile applications.  

The main contributions of this paper are summarized as follows:



  * To our knowledge, it is the first time that directional statistics is introduced to deep distance metric learning.


  * A novel loss function based on the von Mises-Fisher distribution is proposed for deep metric learning to learn an embedded probability space on a hyper sphere.


  * An alternative learning algorithm is proposed to train our model efficiently.




The rest of this paper is organized as follows. 

We firstly review some deep metric learning methods and briefly introduce directional statistics in machine learning in Sec. <ref>.

Some preliminary knowledge about directional statistics is given in Sec. <ref>.

In Sec. <ref>, we present our deep metric learning model, followed by the learning algorithm. 

A toy example is used to show the embedding spaces at the end of this section.

Experiments on both classification and retrieve tasks are conducted in Sec. <ref>.

In Sec. <ref>, we show the performance of vMF on convolutional neural networks with different depths and clustering performances with different clustering methods.

The conclusion and future works of this paper are drawn in the final section.





§ RELATED WORK




This section briefly discusses two active research areas highly related to our approach. 

After the review of the latest deep metric learning research, the directional statistics in machine learning will be briefly discussed.



 §.§ Deep Metric Learning


Deep metric learning aims to learn a non-linear projection function which can transform an image from the pixel level to a discriminate space where samples from the same class will be gathered together, and samples from different classes will be pushed apart.

Recent studies of deep metric learning advance performance of many visual tasks, such as fine-grained categorization <cit.>, image retrieval <cit.>, deep hashing  <cit.>, face verification <cit.>, and person re-identification <cit.>. 

In following parts, several latest deep metric learning methods are briefly reviewed.



  §.§.§ Triplet loss


The main idea of the triplet loss is that distances between dissimilar pairs should be larger than distances between similar pairs with a margin m.

Given an anchor image denoted with X_a, a similar image X_p and a dissimilar image X_n are selected to form a triplet, the triplet loss function can be defined as:


    ℒ=  [(d(f_θ(X_a),f_θ(X_p))-d(f_θ(X_a),f_θ(X_n)) + m ]_+,


where d is a distance function and f is a mapping function with parameters θ. 

The [·]_+ operator denotes the hinge function which equals to  max(·,0).

The performance of triplet-based deep metric learning highly relies on the quality of triplet pairs. 

Many methods have been proposed to deal with this, e.g., mining hard negative samples <cit.> or containing more pairs within a mini-batch <cit.>. 

As a consequence, the training process becomes more complicated.

Moreover, the slow convergence problem of the triplet loss is widely complained. <cit.> 




  §.§.§ Lifted structured loss


The lifted structured embedding <cit.> considers that each similar pairs should compare the distance with all the negative pairs.

The loss function is given as a log-sum-exp formulation:


    ℒ = -1/2|𝒫|∑_(i,j)∈𝒫   [ log(  ∑_(i,k)∈𝒩exp{α - D_i,j} + . .   
       . . ∑_(j,l)∈𝒩exp{α -D_j,l}) + D_i,j] ^2_+ ,


where D_i,j = || f(X_i)-f(X_j)||_2 is the Euclidean distance in the embedding space of samples X_i and X_j,
 𝒫 is the set of positive (similar) pairs and 𝒩 is the set of negative (dissimilar) pairs, and α is a constraint margin.



  §.§.§ N-pair loss


N-pair <cit.> considers to make full use of all pairs in a mini-batch. 

Given a (N+1)-pairs tuple: {(X_i,X_i^+)}_i=1^N, where samples from N classes are selected and X^i_+ is a similar sample to X_i, the loss function can be formulated as follows:


    ℒ =    1/N∑_i=1^N log( 1+ ∑_i≠ jexp( f(X_i)^T f(X_j^+) . . - 
       .  . f(X_i)^Tf(X_i^+)  ) )  + η/m∑_i=1^m||f(X_i)||_2,


where η is a L2 regularization on the embedding vectors.



  §.§.§ Magnet


The Magnet <cit.> suggests to punish overlaps between different clusters. The mini-batch structure is based on clusters. 

One cluster is first randomly selected, then M-1 closest clusters are chosen based on the distances on the embedding space. 

D images per selected cluster will be randomly selected (M× D in total) to form a mini-batch. 

The loss function can be presented as 


    ℒ=1/MD∑^M_m=1∑^D_d=1{logexp{-1/2σ^2r_d^m -μ̂_m^2_2}/∑_μ̂:C(μ̂)≠ C(r_d^m)exp{-1/2σ^2r_d^m -μ̂^2_2}},


where r_d^m = f(X_d) is a point in the embedding space belong to m-th cluster.

The cluster center, μ̂_m, is estimated based on each batch data as follows:

    μ̂_m=1/D∑_d=1^D r_d^m


Though the convergence is claimed to be faster than triplet based methods, it is still very time consuming for generating mini-batch by retrieving images from adjacent clusters. 




 §.§ Directional Statistics in Machine Learning


Besides visual tasks, the directional statistics methods in machine learning have been successfully introduced to many disciplines, e.g., text mining <cit.>, gene expression analysis <cit.>, and bio-medical data analysis <cit.>.

More recently, an SNE method based on von Mises-Fisher distribution is proposed to deal with the high dimensional spherical data visualization. 

More detail can be found in a review paper <cit.>.




§ PRELIMINARY




In this section, we will provide a brief introduction to the directional statistics.



 §.§ Directional Data

The directional data refer to data with unit norm ||x||=1, corresponding to points on a hyper sphere in ℝ^p. 

Usually we can transform any kinds of data to directional data by L2-normalization.




 §.§ Von Mises-Fisher Distribution


The von Mises-Fisher (vMF) distribution is a probability distribution in directional statistics for spherical data. 

The probability density function for a unit vector 𝐫 in 𝕊^p-1 is given as follows:


    𝑓_p(𝐫;μ,κ) = 𝑍_p(κ)exp(κμ^T𝐫),

where κ≥0, ||μ||=1 and the normalization parameter 𝑍_p(κ) is defined as follows:

    𝑍_p(κ) = κ^p/2-1/(2π)^p/2𝐼_p/2-1(κ),



where 𝐼_v is the modified Bessel function of the first kind with order v. 

Similar to the Gaussian distribution, the vMF distribution is captured by two parameters: the mean direction μ and the concentration parameter κ. Here κ characterizes the tightness of the distribution around the mean direction μ. 

The larger value of κ is, the more strongly the distribution is concentrated to the mean direction. 

Given N sample points r_i, the mean direction can be estimated as follows,


    μ = ∑_i=0^N 𝐫_i/||∑_i=0^N 𝐫_i||.


A simple approximation estimate to κ is

    κ̂ = R̅(p-R̅^2)/1-R̅^2,

where 

    R̅= ∑_i=0^N r_i/N.

Figure <ref> shows points generated from different vMF distributions[This figure is adopted from the Wikipedia <https://en.wikipedia.org/wiki/Von_Mises-Fisher_distribution>.]. 

More information can be found in Directional Statistics <cit.>.





 §.§ Hyper-surface Area of a Unit Hyper Sphere


It is essential to understand the geometric properties of feature spaces for deep metric learning. 

However, for the feature space of the directional statistic, the property of the hyper-surface area of a unit hyper sphere is seldomly addressed. 

For a unit sphere in ℝ^p, the area of hyper-surface is given as follows <cit.>,


    S_p = 2π^p/2/Γ(1/2𝑝)=
    2^(p+1)/2π^(p-1)/2/(p-2)!!     if  p  is odd,
    2^p/2π^p/2/(1/2p-2)!     if  p  is even,


where Γ refers to the Gamma function, p! is the factorial and p!! is the double factorial. 

We plot the area of hyper sphere in Figure <ref>. 

The area of a unit hyper sphere firstly increases to a maximum at dimension p=7 then decreases when p increases. 


A distinct advantage of L2-normalization is that it can attenuate the curse of dimensionality. 

The volume of the Euclidean space increases exponentially as the dimension increase.

However, the input data are mapped to the surface of the surface of a hyper-sphere.

The area of the sphere increases as the sphere increases as the dimension increases initially, but then decreases as the dimensionality increases further.

Thus, a more compact feature space is created through the mapping.






§ METHOD




In this section, we provide the problem definition of our deep metric learning model. 

Then we propose a loss function based on the von Mises-Fish (vMF) distribution along with an alternative learning algorithm. 

The overview of our model is presented in Figure <ref>.





 §.§ Problem Definition


Given a training set with N pairs of labeled data {𝐱_n,y_n}_n=1^N that belong to C classes, our model aims to learn a non-linear mapping function 𝑓(·;Θ), which is a convolutional neural network here. 

The learned non-linear mapping function projects an image 𝐱 to a point, 𝐫 and 𝐫=1, on a p-dimension hyper sphere space dominated by C learned vMF distributions {κ_i,μ_i}_i=1^C. 

In this probability space, a point 𝐫 is assigned to class c with the following normalized probability:


    P(c|𝐫,{κ_i,μ_i}_i=1^C) = 𝑍_p(κ_c) exp(κ_c μ_c^T𝐫)/∑_i=1^C𝑍_p(κ_i) exp(κ_i μ_i^T𝐫),


where 𝑍_p(κ_i) is defined in Equation (<ref>) and ||𝐫||=1. 

The learned mapping function should project image data 𝐱_n to a point 𝐫_n in the new pace, which has the higher probability assigned to the right class y_n=c than assigned to the other classes.





 §.§ Von Mises-Fisher Loss


The above goal can be summarized as maximizing the following probability:

    𝑃(Y|X;Θ,𝐔,K)    = ∏_n=1^N P(y_n|𝐫_n,{κ_i,μ_i}_i=1^C)
        = ∏_n=1^N 𝑍_p(κ_y_n) exp(κ_y_nμ_y_n^T𝐫_𝐧)/∑_i=1^C𝑍_p(κ_i) exp(κ_i μ_i^T𝐫_𝐧),


where X and Y represent the training data and labels, Θ represents all the parameters in the mapping function 𝑓 and 𝐔={μ_i}_i=1^C, K={κ_i}_i=1^C, 𝐫_n = 𝑓(𝐱_n;Θ) and ||𝐫_n||=1, n=1,2,…,N.

By taking the negative log-likelihood of Equation (<ref>), the objective function is obtained as follows:



    Θ,𝐔min      𝒥= -log𝑃(Y|X;Θ,𝐔,K)
             =-∑_n=1^N log𝑍_p(κ_y_n) exp(κ_y_nμ_y_n^T𝐫_𝐧)/∑_i=1^M𝑍_p(κ_i) exp(κ_i μ_i^T𝐫_𝐧),
       subject to        ||𝐫_n||=1,   n=1,2,3,…,N 
              ||μ_i||=1,   i = 1,2,3,…,C,



where 𝐫_n=𝑓(𝐱_n;θ), r_n,μ_i∈𝕊^p-1, and the concentration parameters {κ_i}_i=1^C is treated as hyper parameters to capture the divergence of each class. 

Here we simply set the same κ for every class as a global scaling factor. Then Equation (<ref>) can be simplified to,


    ℒ = -log exp(κμ_y_n^T𝐫_𝐧)/∑_i=1^Cexp(κμ_i^T𝐫_𝐧).



We name this loss function as von Mises-Fisher Loss (vMF Loss) and it will be used for all experiments in this paper. 

Because Equations (<ref>) and (<ref>) are differentiable, they can be used directly to train the neural network with the back propagation method.





 §.§ Learning


Our vMF loss has two types of parameters: the mean directions of vMF distributions 𝐔 and the parameters of the mapping function Θ. 

It is difficult to optimize them simultaneously. 

So we apply an alternative training algorithm to learning these two types of parameters. 

In summary, we fix the mean directions of vMF distributions 𝐔 when we train CNN by the mini-batch based stochastic gradient descent for some iterations. 

Then we use the learned CNN to forward pass all training data to obtain the representation vectors {𝐫_n} and update the mean directions by the following expression,


    μ_i = ∑_n=1^N_i𝐫_n^(i)/||∑_n=1^N_i𝐫_n^(i)||,


where N_i is the number of samples that belongs to class i and 𝐫_n^(i)=𝑓(𝐱_n;Θ). 

The overall learning algorithm is described in Algorithm<ref> and illustrated in Figure <ref>.




The training CNN step in the proposed alternative learning algorithm does not rely on any rigid data formats. 

Even we do not need to guarantee the class number in a mini-batch as in <cit.>. 

The training procedure is as simple as training with the softmax loss.

More important is that the global structure of the embedding space is represented by the estimated class mean directions.

For each sample, the loss computed with all mean directions is an approximation computed with the whole training set, which indicates that our method can capture the global information of the embedding space.

Previous works usually are insensible of the global landscape of the embedding space.

Taking the triplet loss as an example, the loss (see Equation (<ref>)) of an anchor image is only computed with one similar sample and one dissimilar sample.

The anchor image is kept in the dark about all other training points in the embedding space except the selected two.

That leads to a unstable training process, especially at the beginning, and the model is easy to be trapped in bad local optimal <cit.>.



 §.§ Inference


After training, we can predict the labels of samples by measuring the cosine similarity between sample features and the learned mean directions {u_i}. 

Each Sample will be assigned with a class labels, of which the mean direction has the largest cosine similarity with the sample features. 

An interesting feature of this scheme is that, this prediction process also can enjoy the common nearest neighbor classifier based on the Euclidean distance. 

For two direction points 𝐫_n and μ_i, their distance is

    d(𝐫_n,μ_i)=||𝐫_n - μ_i||_2^2 = ||𝐫_n||_2^2 + ||μ_i||_2^2-2𝐫_n^T μ_i.

Considering ||𝐫_n||_2^2 = ||μ_i||_2^2=1, their Euclidean distance reduces to d(𝐫_n,μ_i)=2-2cos<𝐫_n,μ_i>, which means that their Euclidean distance is inversely proportional to the cosine similarity.

The largest cosine similarity pair can be found by looking for the pair with the shortest Euclidean distance.




 §.§ Visualization of the Learned Space


We used the MNIST <cit.> dataset as a toy example to give a simple comparison of different embedding spaces with different loss functions. 

A simple multi-layer perceptron (MLP) <cit.> is used here. 

The models are first trained with the training set then the learned model projects the validation set to the embedding spaces.

The embedding spaces of softmax, triplet loss <cit.> and center loss <cit.> are also presented here for comparison.

From Figure <ref>, it can be seen that each class takes over an arc on a unit circle. 

Figures <ref> and <ref> provide the visualization results of different κ in a 3D unit sphere. 

Both figures show that our model can learn a discriminant space for classification.  








§ EXPERIMENTS




In this section, experiments on both classification and retrieval are conducted to verify the performance of our model for different tasks.

For the classification task, we test our model on three fine-grained data sets: Flower-102 <cit.>, Oxford IIT Pet <cit.> and Stanford Dog <cit.>

For the retrieval task, the performance are evaluated base on three standard data sets: CUB-200-2011 <cit.>, Cars196 <cit.> and Stanford Online Products <cit.>. 




 §.§ Implementation Details


We implement our model with MXNet<cit.>, an open source deep learning software. 

We initialize parameters of CNNs before the final fully connected layer with models pre-trained on the ImageNet <cit.>. 

The parameters in last fully-connected layer are initialize with Xavier initialization <cit.>.

Experiments are run on a single NVIDIA GTX-1080 GPU.






 §.§ Classification on Fine-grained data


Classifing object images in subordinate classes is known as fine-grained visual categorization (FGVC). 

For instance, in the general image classification task, a classifier only needs to recognize that it is a dog from a given picture of a dog.

But for the FGVC task, the classifier need to distinguish the breed which a dog belongs to, such as a Beagle or a Basset Hound. Many types of FGVC datasets have been built, including identification different species of animals and plants, classifying galaxies <cit.> and categorizing different air crafts <cit.>. 

One of main challenges for FGVC comes from the following two paradoxical properties. (1) FGVC has a high inter class similarity along with a large intra-class similarity. 

The difference caused by different sub classes may be smaller than that caused by various viewpoints or different shapes from the same sub-classes. 

(2) FGVC tasks usually have a large number of classes but a small number of samples for each class. Taking Oxford Flowers data set <cit.> as an example, it contains 102 categories of flowers and only allows to use ten images for training and 10 for validation for each class. 

Compared with CIFAR-10 <cit.> (6000 images per class) and ImageNet <cit.> (around 1200 images for each class), the number of samples for each class in FGVC is challenging to learn a reliable model. 

Several deep metric learning approaches are proposed as promising solutions for FGVC tasks.


We evaluate our method on three FGVC datasets.  

To conduct a fair comparison, we follow the experiment setting in the Magnet <cit.>.
The Inception with batch normalization <cit.> is used as our CNN part. 

The SGD with momentum is used to fine-tuning models. 

Rich data augmentation methods are used during training. 

The results of the Magnet and the triplet are directly cited from the original paper. 

The results of softmax and n-pairs are obtained by training with the same condition as the vMF loss. 

We set the dimension p = 128 and κ = 15 for three experiments. 

Neither the bounding box nor part information is used in our experiments.
 


  §.§.§ Flower-102

The flower-102 data set contains 102 categories of flowers. 

The number of images of each class varies from 40 to 258 and there are 8189 images in total. 

For each category, there are 10 images for training, 10 images for validation, and the rest images for testing. 

We use all training and validation images as the training set. 

The concentration parameters {μ}_i=1^102 are updated every three epochs (around 100 iterations for 64 images per mini-batch ).  
The accuracy results are presented in Table <ref>.





  §.§.§ Oxford-IIIT Pet


The Oxford-IIIT Pet <cit.> data set provides image data from 37 different breeds of dogs and cats, among which 25 categories are dogs and 12 classes are cats. 

For each  breed, there are around 200 images. 

Half number of images of each class are used for training and the rest are used for testing. 

Compared with previous flower-102 dataset, this pet data set has a larger variance in shapes, colors and scales. 

We update the mean direction of each class for every 100 iterations.






  §.§.§ Stanford Dogs


The Stanford Dogs data set <cit.> contains 120 categories of dog breads, and around 150 images per class and 20,580 in total.

Considering that this data set is a subset from ImageNet <cit.>, we follow the suggestion in the Magnet <cit.> that a model that only trained two epochs on the ImageNet is used as initialization to avoid over-fitting.


The mean directions are updated after every epoch.

The results are reported in Table <ref>.






  §.§.§ Conclusion


It can be found from above classification results that our vMF model achieves state-of-the-art performance on the tree fine-grained datasets. 

Only the Magnet and our method obtain better performance than the softmax for all three datasets.

On Flower-102, our model achieves 95.61%, which is 1.61% higher than the result of the Magnet.

On Oxford-IIIT pet, our method has very close performance with the Magnet and is 0.7% higher than the Magnet.

Our model exceeds the Magnet with 0.9% on the Stanford Dogs.



 §.§ Retrieval Task



In this part, we focus on comparing retrieval performance with several state-of-the-art deep metric learning methods including (1) triplet learning with semi-hard negative mining strategy <cit.>, (2) lifted structure embedding <cit.>, (3) N-pairs metric loss <cit.> and (4) local facility clustering <cit.>.

Because the Magnet only demonstrates the classification results in the original paper <cit.>, we do not include the Magnet in this part.

The evaluation is conducted following the experiment protocol in <cit.>, which is k nearest neighbor retrieval results on data whose classes are not appeared in the training sets.

The retrieval quality is measured by Recall@K <cit.> (R@k), that is, the proportion of query images for which relevant items appear in the top K neighbors. 

As suggested in <cit.>, the embedding size does not significantly influence performance. 

The embedding size is fixed at d=64 for all experiments, which is the same as in <cit.>. 

Comparison results are based on three data sets: CUB200-2011 <cit.>, Cars196 <cit.> and Stanford Online Products <cit.>.

All models are trained on data of first half classes then tested on images from the left half classes. 

Images are resized to 256×256 then cropped at 227×227. 

Random cropping and random mirroring are used for training data augmentation and single center crop are used for testing images.

The stochastic gradient descent (SGD) is used to training our vMF model for the retrieval task. 

The learning rate is reduced from 0.001 to 0.00001 with the batch size of 96.

Due to the much larger number of classes in Stanford Online Product than in Cars196 and CUB-200-2011, the parameter κ is experimentally set to 15 for Stanford Online Product and 40 for the other two datasets. 

The mean directions μ are updated after every epoch.

Because the same experiment setting is applied as <cit.>, the results of other deep metric learning methods are directly cited from <cit.>. 




  §.§.§ Cars196


Cars196 <cit.> is a large car dataset that includes 16,185 images from 196 classes of cars. 

Images from the first 98 classes are used for training and the rest are used for testing. 

The results at 20,000 iterations are presented in Table <ref>. 

Our model outperforms other state-of-the-art methods.

The vMF achieves 62.34% at R@1, which is 4% higher than previous best one.

The successful and failed retrieval examples are presented in Figures <ref> and  <ref>, respectively.

It can be observed that our model is sensitive to the view points to the cars.
















  §.§.§ CUB-200-2011


CUB-200-2011 <cit.> collects 11,718 images from 200 bird species.

From the dataset, 5,864 images of the first 100 categories are used for training and the rest 5,924 images are used for testing. 

The quantitative result at 10,000 iterations is shown in Table <ref>.

It clearly shows that the vMF achieves the state-of-the-art performance.

Our method is 1.3% higher than the Clustering method for R@1.

The successful and failed retrieval results are shown in Figures <ref> and <ref>, respectively.

It can be found that large variants of different poses for the same class are one of the main challenges for CUB-200-2011.



















  §.§.§ Stanford Online Products


The Stanford online Products dataset <cit.> includes 120,053 images of 22,634 online products collected from eBay.com. 

Each product is treated as one class.

The average number of images for each product is around 5.3.

The images of the first 11,318 products are used as training samples in this experiment.

The rest data are used for testing.

The R@K metric results are presented in Table <ref>. 

Our vMF method surpasses other compared latest deep metric learning methods. 

Some successful and failed retrieval examples are presented in Figures <ref> and <ref>, respectively.

Though the viewpoints and product poses change dramatically, our method still can return the correct images from the same class.

Most of false retrieval results come from the products from the same category but belong to different products with few differences.
















§ DISCUSSION





 §.§ Classification Performance on Different Depths


It has been a trend that improving the performance of CNN by making it deeper and more complicated.

However, it is very challenge to apply large networks to many real applications, such as mobile phones and robotics.

As an advantages of our model, vMF can achieves better results with “shallow” networks compared with wildly used softmax.

To demonstrate this property, we evaluate our model with different depths of ResNet <cit.> on the CIFAR-100 dataset <cit.>. 

The CIFAR-100 provides 60,000 images from 100 classes (600 image per class). 

The train and test sets contain 50,000 and 10,000 images respectively.

We firstly resize images to 240×240, then the random cropping and random mirroring are used as data augment. 

The mean directions are updated after every epoch.

We report the quantitative results in Table <ref> and plot results in Figure <ref> for easy comparison.

It can be found that our method surpasses softmax with a significantly margin for all tested depths.

VMF is 15.8% higher than softmax for ResNet-18 and 10.6% higher for ResNet-101.

Moreover, vMF obtains 79.25% accuracy for ResNet-18, which is 7.8% higher than softmax with ResNet-101.

It clearly shows that our method has a higher depth efficiency.









 §.§ Clustering Performance


The clustering task is another important application for the deep distance metric learning. 

In practice, the clustering performance not only depends on the learned features, but also is influenced by how well the clustering methods works with the features.

In this parts, we show that, with the help of von Mises-Fisher distribution-based clustering methods, our method can outperform other state-of-the-art DDML for the clustering task.


In previous works, <cit.>, all clustering performance is conducted with the affinity propagation clustering <cit.> with bisection method[<http://www.psi.toronto.edu/affinitypropagation/apclusterK.m>] (apk).

However, it is worthwhile to choose the clustering methods that can better exploit the geometric properties of the embedding space.

Besides the apk method, we presents the clustering results with other three clustering methods based on the von Mises-Fisher distribution[<https://github.com/clara-labs/spherecluster>]  <cit.>, including: (1) spherical k-means, (2) mixture of vMF-soft, (3) mixture of vMF-hard. 

The features of testing data are first extracted by models trained in Sec <ref> then used for different clustering methods to obtain the cluster labels.  

The clustering performance is measured by normalized mutual information (NMI) <cit.>. 

We refer the clustering performance reported in <cit.> as a comparison, which is the previous best one.

The results are presented in Table <ref>. 



It can be found that, using the apk method, our model achieves 0.5915 and 0.5889 for the Cars196 and CUB-200-2011, respectively.

These results are close to the state-of-the-art performance achieved by <cit.>.

More over, Working with all three von Mises-Fisher distribution-based clustering methods, our model can outperforms the clustering method <cit.> with apk.

Using movMF-hard clustering method, our method surpasses the clustering <cit.> with 3.39% and 5.10% for the Cars196 and CUB-200-2011, respectively. 









 §.§ Impacts of κ


In this part, we experimentally investigate how the hyper parameter κ influences our model. 

The Cars196 <cit.> is used as an exemplar dataset. 

To simultaneously monitor classification and retrieval performances, 20 images per class of first 98 classes are randomly selected as testing set for classification. 

The rest data in the first 98 classes are used for training. 

All images in rest 98 classes are used to test retrieval performance.

The mean accuracy and Recall@1 (R@1) are presented as measurements for classification and retrieval, respectively.

The results are plotted in Figure <ref>.

It can be observed the value of κ has little influence for classification performance.

However, the selection of κ significantly impacts the retrieval performance. 

The retrieval performance first increases with the value of κ, and reaches a peak around 30∼40.

Then recall@1 goes down when κ continues increasing.



To better understand how κ influences the vMF model, we calculate the distribution of train data in the learned embedding space based on the Average κ̂ and the Average Cosine value between class mean directions.

The κ̂ here is the true concentration parameter for each class of training data in the embedding space defined in Equation <ref>.  

The larger value of κ̂, the higher the concentration of the distribution around the class mean direction. 

A smaller average κ̂ suggests that the training samples are more evenly distributed among the hyper sphere.

The average cosine value between mean directions of training data is used to represent the class gap.

The smaller average cosine value indicates that the class mean directions are far away from each others.

The results are presented in Figure <ref>.

It shows that the average cosine between class mean directions is proportional to the hyper parameter κ.

It indicates that the class gap is smaller when κ is greater.

However, too large or too small value of κ makes the training data being tightly concentrated around their mean directions.

The smallest average κ̂ is reached by κ around 30∼40.

A comparison of the curves of the average κ̂ and R@1 shows that R@1 is roughly inversely proportional to the average κ̂.

Theoretical discussion of the relationship between the distribution of training data in the embedding space and the retrieval performance is beyond the scope of this paper. 

We give following hypothesis and leave the analysis to future work.

Different from classification task, the classes in testing data for the retrieval task are not included in the training set.

To achieve better retrieval performance, the DDML models should have better global generalization ability.

In order to achieve better global generalization ability, training points should not tightly concentrate around the mean directions.

So a small value of κ̂ usually has a better retrieval performance.










§ CONCLUSION



In this work, we introduce directional statistics to deep metric learning. 

By considering the feature space as a directional statistical probability space, we propose a new deep metric learning approach.

Specifically, a novel loss function named von Mises-Fisher loss is proposed based on the von Mises-Fisher distribution. 

Then an alternative learning algorithm is applied to train a neural network efficiently with our vMF loss. 

Extensive experiments shows that our vMF can surpass other state-of-the-art methods on both classification and retrieval tasks.

For the future work, we will provide a more analytic method to decide the hyper-parameters in our model. 

As the proposed method can effectively handle the fine-grained categorization problem, applying our model to handle one-shot or zero-shot learning problem will be the top concern for our future work.



§ ACKNOWLEDGMENT


This work is supported by the Hong Kong Research Grants Council (Project C1007-15G), City University of Hong Kong (Project 9610034) and Shenzhen Science and Technology Innovation Committee (Project JCYJ20150401145529049). 




























IEEEtran














[
    < g r a p h i c s >
]Xuefei Zhe received the B.Sc. degree in Information Engineering from Nanjing University, China in 2014. He is currently working toward the Ph.D. degree in the Department of Electronic Engineering, City University of Hong Kong. His research interests include computer vision and deep learning.





[
    < g r a p h i c s >
]Shifeng Chen received the B.E. degree from the University of Science and Technology of China, Hefei, in 2002, the M.Phil. degree from City University of Hong Kong, Hong Kong, in 2005, and the Ph.D. Degree from the Chinese University of Hong Kong, Hong Kong, in 2008. He is now an Associate Professor in the Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, China. His research interests include computer vision and machine learning.


[
    < g r a p h i c s >
]Hong Yan received his Ph.D. degree from Yale University. He was professor of imaging science at the University of Sydney and currently is professor of computer engineering at City University of Hong Kong. His research interests include image processing, pattern recognition and bioinformatics. He has authored or co-authored over 300 journal and conference papers in these areas. He was elected an IAPR fellow for contributions to document image analysis and an IEEE fellow for contributions to image recognition techniques and applications. He received the 2016 Norbert Wiener Award from IEEE SMC Society for contributions to image and biomolecular pattern recognition techniques.


















