A Multi-Domain Feature Learning Method for Visual Place Recognition
Peng Yin Lingyun Xu Xueqian Li Chen Yin Yingli Li Rangaprasad Arun Srivatsan Lu Li Jianmin Ji Yuqing He





A Multi-Domain Feature Learning Method for Visual Place Recognition
    Peng Yin^1,*, Lingyun Xu^1, Xueqian Li^3, Chen Yin^4, Yingli Li^1,
 Rangaprasad Arun Srivatsan^3, Lu Li^3, Jianmin Ji^2,*, Yuqing He^1

This paper was supported by the National Natural Science Foundation of China (No. 61573386, No. 91748130, U1608253) and Guangdong Province Science and Technology Plan projects (No. 2017B010110011).

P. Yin, L. Xu, Y. Li and Y. He are with the State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, University of Chinese Academy of Sciences, Beijing. 
(yinpeng, xulingyun, liyingli, heyuqing@sia.cn)
J. Ji is with the School of Computer Science and Technology, University of Science and Technology of China, Hefei Anhui.
(jianmin@ustc.edu.cn)
X. Li, R.A. Srivatsan and L. Li are with the Biorobotics Lab, Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA.
(xueqianl, arangapr, lilu12@andrew.cmu.edu)
Y. Chen is with the School of Computer Science, University of Beijing University of Posts and Telecommunications, Beijing.
(chenyin@bupt.edu.cn)

(Corresponding author: Peng Yin, Jianmin Ji)


    today
===================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================






Visual Place Recognition (VPR) is an important component in both computer vision and robotics applications, thanks to its ability to determine whether a place has been visited and where specifically.
A major challenge in VPR is to handle changes of environmental conditions including weather, season and illumination.
Most VPR methods try to improve the place recognition performance by ignoring the environmental factors, leading to decreased accuracy decreases when environmental conditions change significantly, such as day versus night.
To this end, we propose an end-to-end conditional visual place recognition method. 
Specifically, we introduce the multi-domain feature learning method (MDFL) to capture multiple attribute-descriptions for a given place, and then use a feature detaching module to separate the environmental condition-related features from those that are not. The only label required within this feature learning pipeline is the environmental condition.
Evaluation of the proposed method is conducted on the multi-season NORDLAND dataset, and the multi-weather  GTAV dataset. 
Experimental results show that our method improves the feature robustness against variant environmental conditions.







§ INTRODUCTION


In the last decade, the robotics community has achieved numerous breakthroughs in vision-based simultaneous localization and mapping (SLAM) <cit.> that have enhanced the navigation abilities of unmanned ground vehicles (UGV) and unmanned aerial vehicles (UAV) in complex environment.
Visual place recognition (VPR) <cit.> or loop closure detection (LCD) helps robots to find loop closure in SLAM framework and is an essential element for accurate mapping and localization.
Although many methods

have been proposed in recent years, 
VPR is still a challenging problem under varying environmental conditions.
Traditional VPR approaches that use handcrafted features to learn place descriptors for local scene description, often fail to extract valid features when encountering significant changes <cit.> in environmental conditions, such as changes in season, weather, illumination, as well as viewpoints.




Ideally, the place recognition method should be able to capture condition-invariant features for robust loop closure detection, since the appearance of scene objects (e.g., roads, terrains and houses) is often highly related to environmental conditions, and that each object has its own appearance distribution under variant conditions.
To the best of our knowledge, there are few VPR methods that have explored how to improve the place recognition performance against variant environmental conditions <cit.>. A major drawback of these methods is that the change in environmental conditions affects the local features, resulting in decreased accuracy of VPR.
In this paper, we propose the condition-directed visual place recognition method to address this issue.
Our work consists of two parts: feature extraction and feature separation.















Firstly, in the feature extraction step, we utilize a CapsuleNet-based network <cit.> to extract multi-domain place features, as shown in Fig. <ref>.
Traditional convolutional neural network (CNN) is efficient in object detection

, regression

and segmentation
, but as pointed out by Hinton, the inner connections of objects are easily lost with the deep convolutional and max pooling operations.
For instance, in face detection tasks, even if the facial objects (nose, eyes, mouth, lips) are in incorrect layouts, the traditional CNN method may still consider the image as a human face, since it contains all the necessary features of a human face.
This problem also exists in place recognition tasks, since different places may contain similar objects but with different arrangements. 
CapsuleNet uses an dynamic routing method to cluster the shallow convolutional layer features in an unsupervised way.

In this paper, we demonstrate another application of CapsuleNet, which could capture feature distribution under specific conditions.








The main contributions of this work can be summarized as follows:

	
  * We propose the use of CapsuleNet-based feature extraction module, and show its robustness in the conditional feature learning for the visual place recognition task.
	
	
  * We propose a feature separation method for the visual place recognition task, where features are indirectly separated based on the relationship between condition-related and condition-invariant features in an information-theoretic view.



The outline of the paper is as follows: Section <ref> introduces the related works on visual-based place recognition methods. 
Section <ref> describes our conditional visual place recognition method, which has two components: feature extraction and feature separation. 
In Section <ref>, 
we evaluate the proposed method on two challenging datasets: the NORDLAND <cit.> dataset which has same trajectories under multiple season conditions and a GTAV dataset which is generated on the same trajectory under different weather conditions in a game simulator.
Finally, we provide concluding remarks in Section <ref>.




§ RELATED WORK



Visual place recognition (VPR) methods have been well studied in past several years, and can be classified into two categories: feature- and appearance-based.
In feature-based VPR, descriptive features are transformed into local place descriptors.

Then, place recognition can be achieved by extracting the current place descriptors and searching similar place indexes in the bag of words.
On the contrary, appearance-based VPR uses feature descriptors that are extracted from the entire image, and performs place recognition by assessing feature similarities.
SeqSLAM <cit.> describes image similarities by directly using the sum of absolute difference (SAD) between frames, while  vector of locally aggregated descriptors (VLAD) <cit.> aggregates local invariant features into a single feature vector and uses Euclidean distance between vectors to quantify image similarities.



Recently, many works have investigated CNN-based features for appearance-based VPR tasks. 
Sünderhauf et al. <cit.> first used pre-trained VGG model to extract middle-layer CNN outputs as image descriptors in the sequence matching pipeline. 
However, a pre-trained network can not be further trained for place recognition task, since the data labels are hard to define in VPR task. 
Recently, Chen et al. <cit.> and Garg et al. <cit.> address the conditional invariant VPR as an image classification task and rely on precise but expensive human labeling for semantic labels. 
Arandjelovic et al. <cit.> developed NetVLAD, which is a modified form of the VLAD features, with CNN networks to improve the feature robustness. 
 

The approach that comes closest to our method is the work of Porav et al. <cit.>, 
where they learn invertible generators based on the CycleGAN <cit.>, 
The original CycleGAN method can transform the image from one domain to another domain, but such transformation is limited to only two domains. 
Thus, for multiple domain place recognition task, the method of Porav et al. requires transformation model between each pair of conditions. In contrast, our method can learn more than two conditions in the same structure.







§ PROPOSED METHOD


In this section, we investigate the details of two core modules in our conditional visual place recognition method.



 §.§ Feature Extraction

















  §.§.§ VLAD


VLAD is a feature encoding and pooling method, which encodes a set of local feature descriptors extracted from an image by using a clustering method such as K-means clustering. 
For the feature extraction module, we extract multi-domain place features from the raw image, by utilizing a CapsuleNet module.
Let q_ik be the strength of the association of data vector x_i to the cluster μ_k, such that q_ik≥0 and ∑_k=1^Kq_ik=1, where K is the clusters number. 
VLAD encodes feature x by considering the residuals

    v_k=∑_i=1^Nq_ik(x_i-μ_k),

and the joint feature description {v_1, v_2, v_3,..., v_N}, where N is the local features number.

Assume we can extract N lower feature descriptors (each is denoted as x_i) from the raw image, we can construct a new VLAD like module with the following equation,

    v_k=∑_i=1^NQ_k(l_i)r(x_i, μ_k),

where r(x_i, μ_k) is the residual function measuring similarities between x_i and μ_k, and Q_k(l_i) is the weighting of capsule vector l_i involved with the k^th cluster center. 









  §.§.§ Modified CapsuleNet

In order to transform Eq.<ref> into an end-to-end learning block, we consider two aspects: 

	
  * Constructing the residual function r(x_i, μ_k);
	
  * Assigning the weights Q_k(l_i).

With lower layer features extracted from the shallow convolution layer, 
we use N× D_property matrix to map lower level features into higher level features, where N is the CNN unit number in the shallow convolution layer. 

If we want to integrate the lower-higher feature mapping within a single layer, the local lower level feature x_i should have 
a linear mapping layer to represent the residual function Q_i

    f(x_i, μ_k)   =𝐖_ikx_i + b_k
    
     r(x_i,μ_k)   = f(x_i, μ_k) - 1/N∑_k f(x_i, μ_k),

where 𝐖_ik and b_k are the linear transformation weighting and bias for the k^th capsule center. 

Furthermore, to estimate the local capsule features weighting Q_k(x_i), we apply a soft assignment estimation defined as

    Q_k(x_i)=exp(b_ik)/∑_j^Kexp(b_ij),

where b_ik is the probability that the i^th  local capsule feature belonging to k^th  capsule cluster c_k. Therefore, Eq.<ref> can be written in the following format,

    v_k=∑_i=1^Nexp(b_ik)/∑_j^Kexp(b_ij)(f(x_i, μ_k) - 1/N∑_k f(x_i, μ_k)).


In order to learn the parameters b_ij, W_ik, and c_k, we apply the iterative dynamic routing mechanism as described in <cit.>.
For the output of N_object higher level features, we assume the last D_C dimensions are assigned as the condition features, e.g. D_C=4 is in the case where the condition is season.










 §.§ Feature Separation

In the previous section, we described the feature extraction module p_θ. In this section, we use an additional decoder module q_ϕ, and two reconstruction modules on feature ℒ_Feature and image ℒ_Image domain to achieve the feature separation. 
Naturally, condition-invariant feature Z_G and condition-related feature Z_C are highly correlated.
Fig. <ref> shows the relationship between information Z_G and Z_C. H(Z_G, Z_C) and I(Z_G;Z_C) are the joint entropy and the mutual entropy respectively, while H(z_G|z_C) and H(z_C|z_G) are the conditional entropy. 
From the view of information theory, feature separation can be achieved in the following ways:

	
  * Decrease the conditional entropy H(z|x): less conditional entropy enforces the unique mapping from x∈χ to z∈ (Z_G, Z_C);
	
  * Improve the geometric feature extraction capability: the more accurate geometry we capture, the higher LCD accuracy we can achieve;
	
  * Reduce the mutual entropy I(Z_G;Z_C|x): use environmental conditions to direct feature extraction.

We add these three restrictions in our feature separation module.







  §.§.§ Conditional Entropy Reduction

H(z|x) measures the uncertainty of feature z given the data sample x. The conditional entropy H(z|x)=0 can be achieved, if and only if z is the deterministic mapping of x.
Thus, reducing H_p_θ(z|x) can improve the uniqueness mapping from x to z, where p_θ is the parameter in the encoder module. 
However, improving the condition entropy H_p_θ(z|x) is intractable, since we can not access the data-label pair (x,z) directly.
An alternative approach is to optimize the upper bound of H_p_θ(z|x), and the upper bound can be obtained through the following equation,

    min_θ ,ϕH_p_θ(z|x)    ≜min_θ ,ϕ -∑ p_θ(z|x)log(p_θ(z|x)) 
       =min_θ ,ϕ -∑ p_θ(z|x)[log(q_ϕ(z|x))]   
           -∑ p_θ(z|x)[log(p_θ(z|x))-log(q_ϕ(z|x))]  
       =min_θ ,ϕ  H_p_θ(z|x)[log(q_ϕ(z|x))] 
           -E_p_θ(z|x)[𝐊𝐋(p_θ(z|x)(q_ϕ(z|x)))],

where 𝐊𝐋 is the Kullback-Leibler divergence.
And H_p_θ(log(q_ϕ(z|x))) measures the uncertainty of the predicted feature with a given sample data x.
Since we can not extract features from the q_θ directly, we add an additional feature encoder module after the decoder module (see Fig .<ref>). Eq. <ref> can be converted into

    min_θ ,ϕH(z|x)    ≤min_θ ,ϕ H_p_θ(z|x)[log(q_ϕ(z|x))] 
       ≜min_θ ,ϕ H_ẑ∼ p_θ(z|x), x̂∼ q_ϕ(x|ẑ)[log(p_θ(z=ẑ|x̂))] 
       =ℒ_Feature(z, ẑ),

where ℒ_Feature is the Feature Reconstruction Loss between feature extracted from the raw data and the reconstructed data.
As we can see in Eq. <ref>, the original H_p_θ(z|x) is transformed into its upper bound ℒ_Feature(z, ẑ), 
and the upper bound is reduced  only when the feature domain and data domain are perfectly matched.   



  §.§.§ Feature Extraction Improvement

Condition entropy reduction sub-module can restrict the mapping uncertainty from data domain to the feature domain, this restriction is highly related to the generalization ability of the encoder module.
For the place recognition task, there will be highly diverse scenes in practice, however, we can only generate limited samples for network training.
In theory, the GAN method use a decoder and discriminator module can learn the the potential feature-to-data transformation with limited samples.
Thus, we improve the data generalization ability by applying GANs.

    ℒ_GAN 
    = min_ϕmax_ωE(log(D_ω(x)) +
    
    E_x∼ q_ϕ(x|z)(log(1-D_ω(x))).

As demonstrated by Goodfellow et.al <cit.>, with iterative updating of the decoder module q_ϕ and the discriminator module D_ω, GAN could pull the generated data distribution closer to the real data distribution, and improve the generalization ability of the decoder module q_ϕ.



  §.§.§ Mutual entropy reduction


I(z_G;z_C|x) is the mutual entropy, which can be extended by

    I(z_G;z_C|x)=
        H(z_G|x)+H(z_C|x) -H(z_G,z_C|x),

where, reducing the mutual entropy is equivalent to reducing the right-hand term in the above equation. 
Since the conditional entropy satisfies 0 ≤ H(z_G, z_C|x) , we can find the upper bound of I(z_G;z_C|x) by ignoring  H(z_G, z_C|x)

    min_θI(z_G;z_C|x)   ≤min_θ(H(z_G|x)+H(z_C|x)).

 
For the condition-related features, we apply a soft-max based classification module ℒ_Cond, to reduce the conditional entropy H(z_C|x).
Furthermore, we apply an L_2 image reconstruction loss to further restrict the uncertainty H(z_G|x) given a sample data x,

    ℒ_Image = x_raw-x_reco,

where x_raw and x_reco are the raw image and reconstructed one respectively.

By combining Eq.[<ref>, <ref>, <ref>] and ℒ_Cond, the joint loss function can be obtained as

    ℒ_Joint =    ℒ_Feature + ℒ_GAN + ℒ_Cond+
    ℒ_Image.







§ EXPERIMENT RESULTS


In this section, we analyze the performance of our method on two datasets and compare it with three feature extraction methods for the visual place recognition task. 

The experiments are conducted on a single NVIDIA 1080Ti card with 64G RAM on the Ubuntu 14.04 system.
For our method, our can inference the local place image with just 30ms, and each feature is just 1 kb.







 §.§ Datasets

The datasets we used here are the Nordland dataset <cit.> and the GTAV dataset <cit.>. 
The Nordland dataset was recorded on a train in Norway during four different seasons, and each sequence follows the same track. 
In each sequence, we generate 17885 frames from the video at 12 Hz, and the first 16885 frames of each sequence is used for training, and the last 1000 frames for testing.  Note that we train on all four Norldand seasonal datasets, using the seasonal labels to find the condition dependent/invariant features, and then test on the last 1000 frames of each dataset.
In the training procedure, we randomly select frames and their corresponding status labels from the four sequences.

The second dataset GTAV <cit.> contains trajectories on the same track under three different weather status (sunny, rainy and foggy).
This dataset is more challenging than the Nordland dataset, since the viewpoints are variant in the GTAV dataset. 
We generate more than 10,000 frames in each sequence, 9000 frames are used as training data, and the remaining 1000 frames for testing. 

For each dataset, all images are resized to 64×64×3 in RGB format. 
The loop closure detection mechanism is followed as in the original SeqSLAM method; sequences of image features are matched instead of a single image. For more details about the structure of the SeqSLAM, we refer the reader to <cit.>.



 §.§ Accuracy Analysis

To investigate the place recognition accuracy, we compare our feature extraction method with three methods in sequential matching: 
the original feature in SeqSLAM that uses sum of absolute difference as local place feature description; 
convolution layer feature from VGG network, which is trained on the large-scale image classification dataset <cit.>; 
adversarial feature learning-based unsupervised feature obtained from the generative adversarial networks <cit.>;
The place is considered as being matched when the distance between current frame and target frame is limited within 10 frames.
We evaluate the performances in the precision-recall curve (PR-curve), area under curve (AUC) index, inference time, and storage requirement.





Fig.<ref> and  <ref> show the precision-recall curve and AUC index respectively for all the methods on the Nordland datasets and the GTAV datasets. 
In Fig. <ref>, the label spr-sum, spr-fall, etc. refer to the performance using the same network and same model, with different  testing sequences.

In general, all the methods perform better in the Nordland dataset than in the GTAV dataset, since the viewpoints are stable and the geometric changes are smooth due to the constant speed of the train. 
In contrast, test sequences in GTAV datasets have significant viewpoints differences. Furthermore, limited field of view and multiple dynamic objects in GTAV also introduces additional feature noises, which causes a significant difference in scene appearance.

VGG features perform well under normal conditions, such as summer-winter in Nordland, but perform poorly under unusual conditions, which indicates that the VGG features trained on normal environmental conditions do not generalize.
BiGAN does not perform well on either datasets, and this is mainly because it does not take into account the condition of the scene and considers all the images as a joint manifold. For example, the same place under different weather conditions will be encoded differently using BiGAN.
Since SeqSLAM uses gray images to ignore the appearance changes under different environmental conditions, the image based features in SeqSLAM are robust against changing conditions as we can see in the Nordland dataset.
However, its matching accuracy decreases greatly in the GTAV dataset, since raw image features are very sensitive to the changing viewpoints.

In general, MDFL outperforms the above features in most cases of the NORDLAND and GTAV datasets, and can handle complex situation well, 
but is not the best in some situations, such as spring-summer in NORDLAND and rain-summer in GTAV.
One potential reason is that, in each dataset, we only consider one type of environmental condition (Season or weather), but we did not take into account the illumination changes. Since the illumination changes continuously, it is not easy to set this type of environmental condition in training.
The geometric features instructed by the environment labels can capture more common geometry details among the multiple weather or season conditions. 
Another advantage lies in the structure of the CapsuleNet-like architecture that enables MDFL to cluster lower level geometry features into high level descriptions.
The benefit of this mechanism can be better seen in the GTAV dataset, where the extracted features are more robust to the viewpoints difference. 
Table <ref> shows the average AUC results of different methods in both datasets, and our MDFL method outperforms all the other methods.








































































§ CONCLUSION
 

In this paper, we propose a novel multi-domain feature learning method for visual place recognition task. 
At the core of our framework lies the idea of extracting condition-invariant features for place recognition under various environmental conditions.
We use a CapsuleNet-based module to capture multi-domain features from the raw image, and apply a feature separation module to indirectly separate condition-related and condition-invariant features.
Based on the extracted condition-invariant features, experiments on the multi-season condition NORDLAND dataset and the multi-weather condition GTAV datasets, demonstrate the robustness of our method.
The major limitation for our method is that the shallow layer CapsuleNet-based module can only cluster lower level features, and can not capture the semantic descriptions for the place recognition. 
In our future work, we will investigate hierarchical CapsuleNet network module to extract higher level semantic features for place recognition.




plainnat




