title: Each Part Matters: Local Patterns Facilitate Cross-view Geo-localization
authors: Tingyu Wang Zhedong Zheng Chenggang Yan Jiyong Zhang Yaoqi Sun Bolun Zheng Yi Yang
text: 
  plain
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  Journal of  Class Files, Vol. 14, No. 8, August 2015
  Shell et al.: Bare Demo of IEEEtran.cls for IEEE Journals
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  Each Part Matters: Local Patterns Facilitate 
   Cross-view Geo-localization
      Tingyu Wang,
          Zhedong Zheng,
          Chenggang Yan,
          Jiyong Zhang, Member, IEEE,
          Yaoqi Sun,
          Bolun Zheng,
          and Yi Yang, Senior Member, IEEE
  Copyright  2021 IEEE. Personal use of this material is permitted. However, permission to use this material for any other purposes must be obtained from the IEEE by sending an email to pubs-permissions@ieee.org.
  Tingyu Wang, Chenggang Yan, Jiyong Zhang, Yaoqi Sun and Bolun Zheng are with the Intelligent Information Processing Lab, Hangzhou Dianzi University, Hangzhou 310018, China (e-mail: wongtyu@hdu.edu.cn; cgyan@hdu.edu.cn; jzhang@hdu.edu.cn; syq@hdu.edu.cn; blzheng@hdu.edu.cn). Chenggang Yan is the Corresponding Author.
  Zhedong Zheng, Yi Yang are with the Australian Artificial Intelligence Institute, University of Technology Sydney, NSW 2007, Australia (e-mail: Zhedong.Zheng@student.uts.edu.au; Yi.Yang@uts.edu.au).
  This work was partially sponsored by Zhejiang Lab’s International Talent Fund for Young Professionals (No.ZJ2020GZ021).
  
      
  ====================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================
  
  
  
  
  
  
  Cross-view geo-localization is to spot images of the same geographic target from different platforms, , drone-view cameras and satellites. It is challenging in the large visual appearance changes caused by extreme viewpoint variations. Existing methods usually concentrate on mining the fine-grained feature of the geographic target in the image center, but underestimate the contextual information in neighbor areas. In this work, we argue that neighbor areas can be leveraged as auxiliary information, enriching discriminative clues for geo-localization. Specifically, we introduce a simple and effective deep neural network, called Local Pattern Network (LPN), to take advantage of contextual information in an end-to-end manner. 
  Without using extra part estimators, LPN adopts a square-ring feature partition strategy, which provides the attention according to the distance to the image center. It eases the part matching and enables the part-wise representation learning. Owing to the square-ring partition design, the proposed LPN has good scalability to rotation variations and achieves competitive results on three prevailing benchmarks, , University-1652, CVUSA and CVACT. Besides, we also show the proposed LPN can be easily embedded into other frameworks to further boost performance.
  
  
  
  Geo-localization, Image Retrieval, Agriculture, Deep Learning.
  
  
  
  
  
  
  § INTRODUCTION
  
  
  
  Cross-view geo-localization is to retrieve the most relevant images from different platforms, which could be applied to many fields, such as accurate delivery, autonomous driving, robot navigation, event detection, and so on <cit.>. 
  For instance, given a drone-view image, 
  the system intends to search images of the same location in the candidate images of the satellite. The satellite-view images are automatically annotated with geo-tags. Obtaining the true-match satellite-view image, we could localize the building in the drone-view image. Besides, the image-based cross-view geo-localization can facilitate the positioning devices, , GPS, to provide a more robust and accurate result.
  
  
  In recent years, cross-view geo-localization has obtained a significant development due to the advance in deep learning. 
  Most works <cit.> explore the deep neural network with the metric learning to learn the discriminative feature. Specifically, the network is to learn one feature space that brings matched image pairs closer and pushes non-matched pairs far apart <cit.>. The attention mechanism and orientation information are also widely used in the network design <cit.>. However, most existing methods only consider the global information via pooling functions, ignoring the contextual information (see Figure <ref>).  
  
  Generally, the aerial-view platform, , drone or satellite, captures the scene image with a wide angle. When the platform acquires a geographic target, the contextual information around the target is also captured as a by-product. 
  
  When existing works usually ignore such information, we argue that the contextual information provides a key clue for cross-view geo-localization. 
  For instance, when there is no apparent difference between two geographic targets, such as two straight roads, the human visual system is challenging to identify the true-match target. However, the task is much easier with the help of contextual information, , neighbor houses. Mining and utilizing the contextual information in the image can improve the accuracy of the cross-view geo-localization.  
  Our work is inspired by the procedure that the human visual system interprets and matches the same scene of different viewpoints <cit.>.
  
  When recognizing a geography scene of two different platforms, the human visual system generally adopts a hierarchical processing manner to improve the accuracy of judgement. Specifically, the human visual system first pays attention to whether the same geographic target is contained in different viewpoint scenes. Then, the human visual system will check the contextual information around the geographic target to verify the correctness of the match. When there is no remarkable landmark, people usually resort to the map to find discriminative neighbor areas. Imitating the process mentioned above, we design a Local Pattern Network (LPN), which is an effective way to explicitly explore the contextual information in an end-to-end learning manner. Specifically, we divide the high-level feature into several parts in a square-ring partition, as shown in Figure <ref>. Because the geographic target is generally located in the center of the image with the contextual information surrounded. Our partition method can obtain not only the geographic target information (the region of A) but also several contextual-information parts (the region of B and C) with different distances from the geographic target. Therefore, we can explicitly exploit contextual information to optimize LPN. We also observe that our partition strategy is robust to the image rotation in nature. For instance, when rotating the left image in Figure <ref> as the right image, the three regions (A, B, and C) still contain the same semantic information as corresponding regions of the left image. Therefore, the network designed according to the square-ring manner has good scalability to image rotation.
  To verify the effectiveness of the proposed method, we conduct experiments on three public datasets, , University-1652 <cit.>, CVUSA <cit.> and CVACT <cit.>. LPN achieves the Recall@1 accuracy of 75.93% for the task of drone-view target localization (Drone → Satellite) and Recall@1 accuracy of 86.45% for the task of drone navigation (Satellite → Drone), which is higher than the baseline work <cit.> by 17.44% and 15.27% respectively. Similar results are also observed on CVUSA and CVACT. Compared with the baseline model <cit.>, the Recall@1 accuracy increases from 43.91% to 79.69% (+35.78%) on CVUSA and 31.20% to 73.85% (+42.65%) on CVACT.
  
  Besides, the proposed method is complementary to most previous works. The proposed method could be easily fused with the state-of-art methods, , SAFA <cit.>, and boost the performance from 89.84% Recall@1 accuracy to 92.83% (+2.99%) Recall@1 accuracy on CVUSA and 81.03% Recall@1 accuracy to 83.66% (+2.63%) Recall@1 accuracy on CVACT.
  In summary, the main contributions of this paper are as follows:
  
  
  
  
    * We propose a simple and effective model, called Local Pattern Network (LPN). Different from existing works, LPN explicitly takes contextual patterns into consideration and leverages the surrounding environment around the target building. Specifically, the model deploys the square-ring partition strategy and learns contextual information in an end-to-end manner. 
  
  
    * We demonstrate the effectiveness of our method on three prevailing cross-view geo-localization datasets, , University-1652 <cit.>, CVUSA <cit.> and CVACT <cit.>. Our method outperforms the strong baseline on both benchmarks by a large margin. Furthermore, we show that the proposed method is complementary to existing works, and can be fused with the state-the-art approaches to further boost the  performance.
  
  
  We organize the rest of this paper as follows. In Section <ref>, we briefly introduce some of the relevant works. Section <ref> presents our designed LPN in detail. Experimental results are presented in Section <ref> and followed by the conclusion in Section <ref>.
  
  
  
  § RELATED WORK
  
  In this section, we briefly review related previous works, including deep cross-view geo-localization and part-based representation learning.
  
  
  
   §.§ Deep Cross-view Geo-localization
  
  Cross-view geo-localization has been attracting more attention in recent years due to a large number of potential applications. Some pioneering approaches <cit.> focus on extracting hand-crafted features. Inspired by the great success of the deep convolutional neural networks (CNNs) on ImageNet, researchers resort to the deeply-learned feature in recent years. 
  Workman  <cit.> are among the first attempts to utilize a pre-trained CNN to extract features for the cross-view localization task. They demonstrate that features from the high-level layer of CNN contain semantic information about the geographic location. To take one step further, Workman  <cit.> fine-tune the pre-trained network by reducing the feature distance between pairs of ground-level images and aerial images, yielding better performance. 
  Inspired by the face verification approaches, Lin  <cit.> adopt a modified Siamese Network <cit.>, which optimizes network parameters by the contrastive loss <cit.>. 
  Zhai  <cit.> plug the NetVLAD <cit.> into a Siamese-like architecture, making image descriptors robust against large viewpoint changes. 
  Liu  <cit.> stress the importance of orientation information and encode corresponding coordinate information into the network to boost the discrimination of the feature. 
  In a recent work, Shi  <cit.> use the spatial layout information to make up the shortcoming of the global aggregation step in feature extraction. Furthermore, Shi  <cit.> improve the performance of cross-view geo-localization through domain alignment and spatial attention mechanism. Besides, DSM <cit.> considers a limited Field of View setting and adopts a dynamic similarity matching module to align the orientation of cross-view images. 
  
  Another line of works considers the metric learning and designs different training objectives to learn the discriminative representation.
  Vo  <cit.> design an orientation regression loss, yielding the performance improvement. 
  
  Hu  <cit.> employ a weighted soft margin ranking loss, which not only speeds up the training convergence but also improves the retrieval accuracy. Different from adopting metric learning loss (, contrastive loss <cit.> and triplet loss <cit.>), Zheng  <cit.> regard the cross-view image retrieval as a classification task. They apply the instance loss <cit.> to optimize the network and has achieved a competitive result. 
  
  
  However, these methods usually concentrate on exploring the global information but ignore the contextual information as shown in Figure <ref>. Different from existing works, the proposed method intends to take advantage of the neighbor areas. We deploy the feature-level partition strategy, which facilitates the end-to-end learning on the contextual information.   
  
  
  
   §.§ Part-based Representation Learning
  
  The local feature has been widely studied in the design of hand-crafted algorithms <cit.>. Ojala  <cit.> propose a local binary pattern (LBP) descriptor to extract the rotation-invariant feature. Lowe  <cit.> develop a Scale Invariant Feature Transform (SIFT) descriptor for the image-based match. SIFT is invariant to translations, rotation, and scaling transformations by summarizing description of the local image structures in a local neighborhood around each interest point. 
  In the spirit of the conventional part-based descriptor, some researchers also explore the local pattern learning in the deep-leaned models.
  One line of works divides the features based on an extra estimator, such as landmark detection, human pose estimated, and human parsing. 
  Spindle Net <cit.> leverages the landmark points of the human body to obtain semantic features from different body regions. 
  Xu  <cit.> propose a pose-guide part attention module to learn a confidence map. 
  Guo  <cit.> acquire the accurate human part-aligned representation by the human parsing model to enhance the robustness of the feature.
  
  Another line of works does not need an extra pose estimator and deploys a coarse part alignment, such as horizontal matching. Li  <cit.> capture the three parts information corresponding to the head-shoulder, upper body, and lower body by Spatial Transformer Network (STN) <cit.>. Zhao  <cit.> utilize the attention mechanism to learn aligned part information from the input image automatically. A strong Part-based Convolutional Baseline (PCB) <cit.> shows a uniform partition strategy to extract high-level features. Then, by correcting within-part inconsistency of all the column vectors according to their similarities to each part, the performance of this work becomes better. Currently, some state-of-the-art works <cit.> extend the PCB with more partitions or optimization losses. 
  Our work also studies a part-based representation learning on the convolutional layer, but is different in two aspects: 
  Different from works of the first line <cit.>, the proposed method does not need an extra part estimator. Different from works of the second line <cit.>, our partition method makes the network have good scalability to image rotation (see Figure <ref>).
  
  
  
  
  
  § PROPOSED METHOD
  
  In this section, we introduce the Local Pattern Network (LPN) (see Figure <ref>). We first illustrate the network architecture for feature extraction, followed by the partition strategy for feature maps and the optimization objective. Finally, we provide a discussion on our intuition and special cases for different datasets. 
  
  Problem formulation. Given one geo-localization dataset, we denote the input image as x, and y represents the corresponding label. We apply the subscript j to denote the platform where the data x_j is collected, and j ∈{1, 2, 3}. 
  In particular, x_1 denotes the sample from  the satellite view, x_2 denotes the drone-view data, and x_3 denotes the ground-view image. The label
  y ∈ [1, C], where C indicates the number of categories. For instance, a dataset includes 701 buildings and each building contains multiple images. We number 701 buildings into 701 different indexes. Each index represents a category, , the label y ∈ [1, 701]. For cross-view geo-localization, we intend to learn one mapping function, which could project images from different platforms to one shared semantic space. The images of the same location are close, while the images from different location are apart from each other. 
  
  
  
  
  
  
  
  
  
  
  
  
   §.§ Local Pattern Network
  
  Feature extraction.
  The proposed model, , Local Pattern Network (LPN), contains three branches, which extends from the Siamese network <cit.>. From top to bottom in Figure <ref>, the three branches are the satellite-view branch, the drone-view branch and the ground-view branch respectively. LPN can deploy various network architectures as backbones to extract features, such as VGG <cit.>, and ResNet <cit.>. For illustration, we choose ResNet-50 <cit.> as the network architecture of each branch if not specified. ResNet-50 contains five blocks named conv1, conv2, conv3, conv4, conv5, one average pooling layer, and one fully connected layer. 
  We remove the final average pooling layer and the fully connected layer, and obtain intermediate feature maps for subsequent partition processing. 
  Following  <cit.>, we share weights between the satellite-view branch and the drone-view branch, since input images of both branches are from the aerial viewpoint. Three branches have the same feature extraction manner. Specifically, given an input image of size 256 × 256, we can acquire feature maps with the shape of 16 × 16 × 2048 in each branch. We denote this function as ℱ_backbone, and the process of feature extraction can be formulated as:
  
      f_j   = ℱ_backbone(x_j),
  
  where f_j stands for the extracted feature map of the input image x_j. 
  
  
  Feature partition strategy.
  To explicitly take advantage of contextual information, we apply the square-ring partition strategy to divide feature maps. 
  
  
  We observe that the geographic target is generally distributed in the center of the image, and the contextual information is radiantly distributed around. 
  Based on this assumption of semantic information distribution, the center of the square-ring partition can be approximately aligned at the center of the feature maps. As shown in Figure <ref> (A) (green box), we separate images into four parts according to the distance to the image center. In practice, we separate the global feature maps f_j to four feature parts f_j^i (i∈{1,2,3,4}). The superscript i represents the i-th part from the center. 
  
  Then we apply the average pooling layer to transform each part f_j^i with different shapes into a 2048-dim part feature g_j^i. The process can be formulated as:
  
      f_j^i    = ℱ_slice(f_j, i),
  
  
      g_j^i    = 𝒜vgpool(f_j^i),
  
  where ℱ_slice indicates the square-ring partition, and 𝒜vgpool represents the average pooling operation.
  
  
  Optimization objective.
  
  Now we have obtained part features from different sources. Since the features are extracted from different branches, they may have different distribution, which could not be directly used for matching. 
  To solve this limitation, we set up a mapping function that maps features of all sources into one shared feature space. 
  In this shared space, features of the same geo-tag will have a closer distance, while features of different geo-tags are apart from each others.
  
  This classifier module consists of following layers: a fully connected layer (FC), 
  a batch normalization layer (BN), 
  
   a dropout layer (Dropout), 
  and a classification layer (Cls), which is a fully-connected layer. 
  The classifier module is deployed to predict the geo-tag of each image based on part features. 
  
  Given the part features g_j^i as the input, the classifier module outputs a column vector z_j^i. The dimension of z_j^i equals the number of geo-tag categories C.
  
      z_j^i    = ℱ_classifier(g_j^i).
  
  The cross-entropy loss could be formulated as:
  
      p̂(y|x_j^i)    = exp(z_j^i(y))/∑_c=1^Cexp(z_j^i(c)),
  
  
      Loss    = ∑_i,j-log(p̂(y|x_j^i))),
  
  where z_j^i(y) is  the logit score  of the ground-truth geo-tag  y. We apply the softmax function (Equation <ref>) to obtain the normalized probability score p̂(y|x_j^i) in [0,1]. p̂(y|x_j^i) is the predicted probability that x_j^i belongs to the geo-tag  y. 
  In Equation <ref>, we accumulate the losses on the image of different parts and different platforms to optimize the whole network. 
  
  
  
   §.§ Discussion
  
  Our method is inspired by the mechanism of the human visual system on matching images of different viewpoints. In the ancient time, people compare the map and the surrounding environment to know where they are. The contextual information plays an important role. 
  Nowadays, cameras on different platforms typically use wide-angle lenses to obtain complete geographic targets, and the contextual information around the geographic target is also collected in the image. We argue that the contextual information, as a by-product, can facilitate the discriminative representation learning. For example, the neighbor building also could help to predict the target location.
  Instead of dividing images in the pixel level, we split the feature maps in practice, which could not only improve the efficiency but also enable the larger receptive fields as well as the part alignment. The square-ring partition strategy is also robust to the rotation variants. Case 1 (see Figure <ref>) shows the application of the square-ring partition strategy on three images of different views in University-1652, , satellite view, drone view and ground view. The orientation of these three-view images is not aligned. However, we can observe that the geographic target (orange point) is generally located in the image center. Because of the random orientation of three-view images, the contextual information with the same semantics (blue point) may not be distributed in the same orientation but can be located in the same part.
  We note that the sequential partition strategy is a special case of the square-ring partition strategy. The sequential partition strategy takes into account the geometric correspondence <cit.> for a north aligned satellite-and-ground panorama image pair, such as data on CVUSA <cit.>/CVACT <cit.>. Specifically, we apply the square-ring partition to satellite images and the row partition to ground panoramas.
  As shown in Case 2 of Figure <ref>, image pairs on CVUSA/CVACT have different visual appearances. But the same semantic information (, the blue point pair) from the true-matched image pair can still be roughly located in the same part of the divided feature maps. Besides, the column partition is also a variation of our method. The column partition can be adapted when the orientation and the spatial semantics of the matching image pair are roughly aligned. For example, images have been pre-processed by Optimal Transport theory <cit.> or the polar transform <cit.>. Case 3 (see Figure <ref>) provides an example that the column partition is applied to a polar-transformed satellite image and a ground panorama.
  
  
  
  
  
  
  
  § EXPERIMENT
  
  We first introduce three large-scale cross-view geo-localization datasets, two small-scale landmark retrieval datasets and the evaluation protocol. Then Section <ref> describes the implementation detail. We provide the comparison with the state of the arts in Section <ref>, followed by the ablation study in Section <ref>.
  
  
   §.§ Datasets and Evaluation Protocol
  
  We mainly train and evaluate our method on three large-scale geo-localization datasets, , University-1652 <cit.>, CVUSA <cit.> and CVACT <cit.>. Table <ref> shows the image number of query and gallery sets for testing different tasks using these three datasets.
  University-1652 <cit.> is a multi-view multi-source dataset containing satellite-view data, drone-view data and ground-view data. 
  It collects 1652 buildings of 72 universities around the world. 
  The training set includes 701 buildings of 33 universities, and the testing set includes the other 951 buildings of the rest 39 universities. There are no overlapping universities in the training and test set. 
  Since some buildings do not have enough ground-view images to cover different aspects of these buildings, the dataset also provides an additional training set. Images in the additional training set are collected from the Google Image, and they have a similar view as the ground-view images. Therefore, the additional training set can be used as a supplement of the ground-view images. 
  The dataset is employed to study two new tasks, , drone-view target localization (Drone → Satellite) and drone navigation (Satellite → Drone). There are 701 buildings with 50,218 images for training. 
  In the drone-view target localization task (Drone → Satellite), there are 37,855 drone-view images in the query set and 701 true-matched satellite-view images and 250 satellite-view distractors in the gallery. There is only one true-matched satellite-view image under this setting. In the drone navigation task (Satellite → Drone), there are 701 satellite-view query images, and 37,855 true-matched drone-view images and 13,500 drone-view distractors in the gallery. There are multiple true-matched drone-view images under this setting.
  CVUSA <cit.> provides the data collected from two views, , the ground view and the satellite view. Specifically, it contains 35,532 ground-and-satellite image pairs for training and 8884 image pairs for testing. All ground-view panoramic images are collected from Google Street View. Meanwhile, corresponding satellite-view images are downloaded from Microsoft Bing Maps. 
  CVACT <cit.> is a large-scale cross-view dataset. Same as CVUSA, CVACT provides 35,532 ground-and-satellite image pairs for training, and ground-view images are panoramas. Besides, CVACT provides a validation set with 8884 image pairs named CVACT_val and a testing set with 92,802 image pairs denoted as CVACT_test. A query image only has one true-matched image in the gallery for CVACT_val, while for CVACT_test, a query image may correspond to several true-matched images in the gallery.
  Oxford5k <cit.> & Paris6k <cit.> are two prevailing landmark retrieval datasets collected from Flickr. Oxford5k consists of 5062 images that belong to 11 different Oxford landmarks, and Paris6k contains 6412 images of 12 particular Paris buildings. There are 55 query images in Oxford5k and 12 queries in Paris6k. 
  Evaluation protocol. 
  In our experiments, we use the Recall@K (R@K) and the average precision (AP) to evaluate the performance of our model. R@K represents the proportion of correctly matched images in the top-K of the ranking list. A higher recall score shows a better performance of the network. We also calculate the
  area under the Precision-Recall curve, which is known
  as the average precision (AP), which reflects the
  precision and recall rate of the retrieval performance. 
  
  
  
   §.§ Implementation Details
   
  We employ the ResNet-50 <cit.> with pre-trained weights on ImageNet <cit.> to extract visual features. Following  <cit.>, we modify the stride of the second convolutional layer and the last down-sample layer in conv5_1 of the ResNet-50 from 2 to 1. The newly-added layers in LPN, , the classifier module, are initialized with kaiming initialization <cit.>. We resize each input image to a fixed size of 256 × 256 pixels during training and testing. In training, we employ random cropping and flipping to augment the input data. For the optimizer, we adopt stochastic gradient descent (SGD) with momentum 0.9 and weight decay 0.0005 with a mini-batch of 32. The initial learning rate is 0.001 for backbone layers and 0.01 for the new layers. We train our model for 120 epochs, and the learning rate is decayed by 0.1 after 80 epochs. 
  During testing, we utilize the Euclidean distance to measure the similarity between the query image and candidate images in the gallery. Our model is implemented on Pytorch <cit.>, and all experiments are conducted on one NVIDIA RTX 2080Ti GPU.
  
  
  
   §.§ Comparison with the State-of-the-arts
  
  Results on University-1652. 
  As shown in Table <ref>, we compare the proposed method with other competitive approaches on University-1652. 
  The proposed LPN has achieved 74.18% Recall@1 accuracy and 77.39% AP on Drone → Satellite and 85.16% Recall@1 accuracy and 73.68% AP on Satellite → Drone without using the additional Google training data. 
  The performance has surpassed the reported result of other competitive methods such as <cit.>, and the proposed method outperforms the best method, , instance loss <cit.> by a large margin, , about 14% AP improvement.
  If the extra training data, , noisy data collected from Google Image, is added into the training set <cit.>, we could further boost the retrieval performance. In the drone-view target localization task (Drone → Satellite), the accuracy of Recall@1 increases from 74.18% to 75.93% and the value of AP goes up from 77.39% to 79.14%; in the drone navigation task (Satellite → Drone), the accuracy of Recall@1 increases from 85.16% to 86.45% and the value of AP raises from 73.68% to 74.79%. For the drone-view target localization task, there are 951 satellite-view images in the gallery. To make this retrieval task more challenging, we add 8884 satellite-view images collected from the testing set of CVUSA into the gallery of University-1652 as the distractors. Although the distractors would decrease the overall performance, indicated by Rank@1 and AP accuracy, the results are still competitive. This demonstrates the robustness of our proposed method against distractors.
  
  
  
  
  
  Results on CVUSA.
  The comparison with other competitive methods on CVUSA is detailed in Table <ref>. 
  Ground-view images on CVUSA are panoramas, in which, the contextual information is generally distributed on both sides of the geographic target. Basing on the discussion in <ref>, we deploy the sequential partition strategy to explicitly mine the contextual information on CVUSA (see Figure <ref>). The sequential partition strategy is a specific case of the square-ring partition strategy.
  As shown in Table <ref>, we could observe two points. 
  First, we deploy category recognition as the pretext task to conduct geo-localization on CVUSA. In particular, we regard 35,532 pairs as 35,532 location categories to train the model. The proposed method, whether using VGG16 <cit.> or ResNet-50 <cit.> as the backbone, surpasses most existing methods. Specifically, when using VGG16 as the backbone, our method achieves 79.69% R@1, 91.70% R@5, 94.55% R@10 and 98.50% R@Top1% on CVUSA. Since the feature expression capability of ResNet-50 is powerful than VGG16, our method with ResNet-50 backbone obtains 85.79% R@1, 95.38% R@5, 96.98% R@10 and 99.41% R@Top1% on CVUSA.
  Second, the proposed method is complementary to existing methods. 
  For instance, our method can combine with the CVFT <cit.> and the SAFA <cit.> orthogonally. We re-implement CVFT and SAFA. Specifically, we keep the VGG16 as the backbone on both models unchanged and divide feature maps basing on Case 3. For CVFT, we divide the final aligned feature maps into 8 parts and compute the loss of each corresponding part. For SAFA, the polar transform is retained. We first divide the feature maps into 8 parts, and then we use one SPE in SAFA to deal with each part separately before computing the loss. 
  Combined with our partition strategy, CVFT+Ours boosts the R@1 accuracy from 61.43% to 68.20% (+6.77%) and the R@Top1% accuracy from 99.02% to 99.30% (+0.28%). Similarly, SAFA+Ours can further improve the R@1 accuracy from 89.84% to 92.83% (+2.99%) and the R@Top1% accuracy from 99.64% to 99.78% (+0.14%).
  
  
  
  
  Significant performance improvements suggest that our method helps to mine more contextual information, yielding discriminative features.
  Results on CVACT. CVACT has a similar data pattern with CVUSA. Our method with ResNet-50 <cit.> backbone achieves 79.99% R@1, 90.63% R@5, 92.56% R@10 and 97.03% R@Top1% on CVACT. For a fair comparison, our method using VGG16 as the backbone also acquires competitive results. CVFT <cit.>+Ours obtains the improvement with the R@1 accuracy from 61.05% to 62.90% (+1.85%) and the R@Top1% accuracy from 95.93% to 97.22% (+1.29%). SAFA <cit.>+Ours increases the R@1 accuracy from 81.03% to 83.66% (+2.63%) and the R@Top1% accuracy from 98.17% to 98.41% (+0.24%). The experimental results demonstrate that our method is still effective on CVACT.
  
  
  
   §.§ Ablation Studies
  
  To verify the effectiveness of components in our model, we design several ablation studies.
  Effect of the number of parts. 
  The number of parts n is one of the key parameters in our network. By default, we deploy n=4. 
  
  When n = 1, the model is employed to global average pooling. At this time, the model equals the baseline with global attention  <cit.>.
  As shown in Figure <ref>, with the increment of n, both the Recall@1 and AP values have a significant improvement, since more contextual information has been taken into consideration. 
  
  Intuitively, concatenating more contextual information parts can improve the discriminability of the final feature descriptor. However, we note that, as n increases, each part contains less receptive fields with limited semantic information. 
  As a result, a higher value of n compromises the discriminability of the image representation. When n = 6 or 8, the performance gains slowly or even slightly degrades. Therefore, we use n = 4 as the default choice for our network, which balances the mining of the contextual information and the appropriate size of the receptive field. 
  
  
  
  
  
  
  Effect of the input image size.
  A small training size compresses the fine-grained information of the input image, which compromises the discriminative representation learning. 
  In contrast, a larger input size introduces more memory costs during training. 
  To balance the input image size with the memory usage, we study the effect of the input image size. We just change the image size and the covered region of the image is not changed in the experiment. As shown in Table <ref>, in both tasks, , (Drone → Satellite) and (Satellite → Drone), as the input image size from 224 to 384, we observe that the performance gradually improves. 
  When we continue to enlarge the input size to 512, the improvement is not clear on the Drone → Satellite task. We hope this study could help the real-world application in selecting the appropriate input size, when computation sources are limited.  
  
  
  
  
  
  
  
  
  
  
  Is LPN robust to rotation variants?
  Satellite-view images in University-1652 are north aligned and the orientation of drone-view images is random. In the training phase, the rotation augmentation is applied in the satellite-view branch but not in other branches. To verify the scalability of LPN for image rotation, we conduct experiments on rotating the query image to retrieval the true-matched images. We do not rotate gallery images but query images. The experimental results are shown in Table <ref>. The 0^∘ denotes the input query image without rotation. 
  
  For the task of drone navigation (Satellite → Drone), LPN obtains robust features for unseen satellite-view query images against different rotation angles. In contrast, for the task of drone-view target localization (Drone → Satellite), we do not train the model on the rotated drone-view data. The LPN still achieves one competitive performance without a significant performance drop. In addition, we also attempt to rotate different angles on query and gallery images to further test our model. The experimental results suggest that LPN has good scalability to rotation variations.
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  Does LPN learn complementary part features?
  The square-ring partition strategy divides the feature maps into four parts in LPN. We use the numbers 1, 2, 3, and 4 to represent the four parts of the feature maps, as shown in Figure <ref>. Subsequently, we conduct experiments by choosing one or a combination of the four parts. The experimental results demonstrate LPN has learned complementary features (see Table <ref>). We observe that using only one part has obtained one fairly good result in two tasks, , (Drone → Satellite) and (Satellite → Drone). When we further concatenate two or three parts, the accuracy of Recall@1 and AP gradually increases. 
  When all parts are leveraged, we obtain the best performance in both tasks. It demonstrates that LPN has learned complementary part feature, enriching the model capability. But the Recall@1 accuracy and AP drop dramatically in both tasks if query features and gallery features cover different parts. The results reflect from the side that the learned semantic information between the parts is complementary. Because of the complementary in each part, the semantic information contained in different parts is not overlapping. When there are different parts in the query and gallery feature (, the true-matched image pair covers significant different areas), the different parts become the distractors for the final visual feature, resulting in terrible retrieval performance. 
  Is LPN robust to the shifted query image? In the realistic scenario, there is usually an offset in the geographic target location of the query image and true-matched images in the gallery. To explore whether LPN can cope with the offset of the geographic target location in a true-matched image pair, we carry out experiments on shifting the query image during testing. Specifically, we shift the query image to the right in pixels and keep images in the gallery intact (see Figure <ref>). Table <ref> shows the experimental results. 0 indicates that the input query image is not offset. When the input query image is shifted 10 pixels, we can hardly observe a performance drops for drone navigation and drone-view target localization tasks. While the shifted pixels is 20, the performance on both tasks decreases slightly. The experimental results suggest that LPN is robust when there is a small offset of the geographic target location for a true-matched image pair in retrieval.
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  Geo-localization between satellite-view images and ground-view images. In University-1652, geo-localization between satellite-view images and ground-view images is a challenging task. We can sum up the difficulties in the following points. (1) Ground and satellite images that are collected from different viewpoints naturally have a distinct visual appearance. (2) Unlike ground-view images in CVUSA and CVACT, which are panoramas, the ground-view image in University-1652 only covers part of the whole building. (3) Ground-view images in University-1652 contains vast obstacles, , trees and cars. From Table <ref>, we can notice that geo-localization between satellite-view and ground-view images do not work well. Using satellite-view images to retrieve the ground-view images, our method achieves the best results in Recall@1 accuracy and AP. Whether employing drone-view images or not, there is a performance decrease in the ground-to-satellite localization compared with baseline <cit.>. But using drone-view images in LPN obtains better results than without using these.
  Effect of the drone distance to the geographic target. The scale of the satellite-view image in University-1652 is fixed, while the scale of the drone-view image changes dynamically with the drone distance to the geographic target. We adopt drone-view images with different distances to the geographic target as queries to study the impact of the changed scale for LPN. As shown in Table <ref>, when the drone-view image is captured in a middle distance to the geographic target, we obtain the best performance. When the drone distance is short to the geographic target, we can observe that the results are still competitive compared with using all drone-view query images. The scales of these drone-view images are close to satellite-view images. Another reason is that these drone-view images mainly contain the target building without extra trees and other buildings.
  
  
  
  
  Transfer learning from University-1652 to small-scale datasets. 
  To study the generalization ability of LPN trained on University-1652, we evaluate three models on two small-scale landmark retrieval datasets, , Oxford5k <cit.> and Paris6k <cit.>. The first model is ResNet-50 <cit.> trained on ImageNet <cit.>. The second model is baseline <cit.> and LPN is the third model. During the evaluation, three models have not been fine-tuned on these two datasets. For baseline and LPN, we choose two different branches, , ℱ_s and ℱ_g to extract features, since these two branches focus on different low-level patterns of input images. Weights on ℱ_s are trained by the satellite-view images, while ℱ_g is learned on the ground-view images. From Table <ref>, we observe that the extracted feature from LPN shows better performance on both two datasets than features obtained from ResNet-50 and baseline. This result also demonstrates that the square-ring partition strategy can enhance the generalization ability of our model.
  We also note that in the same model, ℱ_g has a superior generalization ability than ℱ_s. 
  It is because that images in Oxford5k and Pairs6k are closer to the Google Street View images, which are similar to the ground-view images collected from Google Image. Besides, ℱ_s is trained by the aerial-view data, which viewpoint is perpendicular to the ground plane. In contrast, the data viewpoint in Oxford5k or Paris6k is parallel to the ground plane.
  
  
  
  
  
  
  
  
   §.§ Qualitative Result
   
  As an additional qualitative evaluation, we visualize some heatmaps created by our and compared methods. Figure <ref> (a) shows heatmaps generated by baseline <cit.> and LPN in the drone and satellite platforms. Compared with the baseline, our approach activates the region of the geographic target and neighbor areas containing the contextual information. Figure <ref> (b) shows the ground-view heatmaps generated by the original SAFA <cit.> and SAFA fused our partition strategy (Ours). SAFA only activates the position of the road, while our method further places emphasis on contextual information next to the road position. Our method is more consistent with the processing of the human visual system to locate an unfamiliar road.
  
  Moreover, we show some retrieval results for different tasks on University-1652 and CVUSA in Figure <ref>. On University-1652, we observe that LPN can adapt to retrieve the reasonable images based on the content in both drone-view localization and drone navigation tasks. One failure case is also shown in the second row of Figure <ref> (1), in which LPN can not recall the matched image in top-1. We notice that it is challenging in that the recalled top-1 image has a very similar pattern with the query image, especially the appearance of the geographic target in two images. 
  
  On CVUSA, we observe a similar result. Our method with SAFA <cit.> has successfully retrieved the relevant satellite-view images.
  
  
  
  § CONCLUSION
  
  In this paper, we identify the challenge in cross-view geo-localization, and propose a simple and effective deep neural network, called Local Pattern Network (LPN), to explicitly mine the contextual information. Specifically, we introduce a square-ring partition strategy for learning complementary spatial features according to the distance to the image center. The contextual information enhances the discriminability of the image representation with more fine-grained patterns. Our approach has achieved competitive accuracy on three cross-view geo-localization benchmarks, , University-1652, CVUSA and CVACT. Moreover, the proposed LPN has good scalability to rotation variation, which is close to the real-world application. The square-ring partition strategy also can be easily embedded into other frameworks to boost performance. In the future, we will investigate applying a module, such as STN <cit.>, to estimate the scale of the drone-view image.
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  IEEEtran
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  