Dual Attention Matching Network for Context-Aware Feature Sequence based Person Re-Identification
Jianlou Si Honggang Zhang Chun-Guang Li Jason Kuen Xiangfei Kong Alex C. Kot Gang Wang








Dual Attention Matching Network for Context-Aware Feature Sequence based Person Re-Identification
    
Jianlou Si1, Honggang Zhang1, Chun-Guang Li1, Jason Kuen2, 
Xiangfei Kong2, Alex C. Kot2, Gang Wang3

1 Beijing University of Posts and Telecommunications, Beijing, China

2 Nanyang Technological University, Singapore

3 Alibaba AI Labs, Hangzhou, China









    
===========================================================================================================================================================================================================================================================================











Typical person re-identification (ReID) methods usually describe each pedestrian with a single feature vector and match them in a task-specific metric space. However, the methods based on a single feature vector are not sufficient enough to overcome visual ambiguity, which frequently occurs in real scenario. In this paper, we propose a novel end-to-end trainable framework, called Dual ATtention Matching network (DuATM), to learn context-aware feature sequences and perform attentive sequence comparison simultaneously. The core component of our DuATM framework is a dual attention mechanism, in which both intra-sequence and inter-sequence attention strategies are used for feature refinement and feature-pair alignment, respectively. Thus, detailed visual cues contained in the intermediate feature sequences can be automatically exploited and properly compared. We train the proposed DuATM network as a siamese network via a triplet loss assisted with a de-correlation loss and a cross-entropy loss. We conduct extensive experiments on both image and video based ReID benchmark datasets. Experimental results demonstrate the significant advantages of our approach compared to the state-of-the-art methods.








§ INTRODUCTION



Person Re-Identification (ReID) aims at associating the same pedestrian across multiple cameras <cit.>, which has attracted rapidly increased attention in the computer vision community due to its importance for many potential applications, such as video surveillance analysis and content-based image/video retrieval.



A typical person ReID pipeline usually describes each pedestrian image or video footage with a single feature vector firstly and then match them in a task-specific metric space, where the feature vectors of same pedestrian are expected to have smaller distances than that of different pedestrians, , <cit.>. Recently, benefited from the success of deep learning, 
feature vector based methods have obtained significant performance improvements <cit.>. However, when the individuals undergo drastic appearance changes or when they are dressed in similar clothes, it becomes quite difficult to use single feature vector based representation for reliable person ReID. As shown in Fig. <ref> (a), different individuals are very similar to each other in appearance, except for some local patterns on skirts. Unfortunately, the single feature vector based methods usually pay more attention on the overall appearance rather than the local discriminative parts and thus fail to yield accurate matching results. Moreover, as shown in Fig. <ref> (d), there are also some interference frames in each video sequence, which will heavily contaminate the whole feature vector 
and thus lead to mismatching.









An alternative way to address these problems is to describe each person with a set of feature vectors and match them based on feature set or feature sequence.[In this paper, we refer to a group of feature vectors as feature sequence if they have spatial/temporal adjacent relations; otherwise as feature set.] For example, in <cit.>, the spatial-patch based local feature sequences or body-part based semantic feature sets are extracted from pedestrian images and matched according to some heuristic correspondence structures; in <cit.>, multiple sub-segment or frame level comparisons are computed and aggregated for matching pedestrian based on video.

Among these methods, matching the local feature sequences based on a universal correspondence structure might easily fail when encountering heavily sequence misalignments, , caused by the spatial displacements as shown in Fig. <ref> (b) or local interferences as shown in Fig. <ref> (d). Besides, matching the semantic feature sets based on human body structure might also fail when encountering body-part occlusions 
as shown in Fig. <ref> (c). 



To tackle the challenges mentioned above, we propose a novel end-to-end trainable framework, named Dual ATtention Matching network (DuATM), to jointly learn context-aware feature sequences and perform attentive sequences comparison. Our framework consists of two cascaded modules, one for feature sequence extraction and one for feature sequence matching, as illustrated in Fig. <ref>.
The feature sequence extraction module is built on a spatial/bi-recurrent convolutional neural network (CNN) for image/video inputs to extract spatial/temporal-spatial context-aware feature sequences. 
The sequence matching module is based on a dual attention mechanism—which contains one attention strategy for intra-sequence refinement and one attention strategy for inter-sequence alignment—the former refines each corrupted feature vectors by exploiting the contextual information within sequence and the later aligns feature-pair by selecting semantically consistent counterparts cross paired sequences.

After feature sequences refinement and alignment, the holistic sequence distance score is computed by aggregating multiple local distances between the refined and aligned pairwise feature vectors of each paired sequences.
We train the proposed DuATM as a siamese network with a triplet loss 

plus a de-correlation loss and a cross-entropy loss, to reduce the feature sequence redundancy and enhance the feature sequence discrimination. 



The main contributions of the paper are as follows.


  * We propose a novel end-to-end trainable framework for person ReID, which can jointly learn context-aware feature sequences and perform sequences comparison with dual attention mechanism.


  * We use a dual attention mechanism to perform intra-sequence feature refinement and inter-sequence feature-pair alignment simultaneously.


  * We train DuATM as a siamese network with a triplet loss, plus a de-correlation loss and a cross-entropy loss, and evaluate the effectiveness of each part.


  * We conduct extensive experiments on both image and video based benchmark datasets and demonstrate the effectiveness of our proposal.











































§ RELATED WORKS


Person ReID systems usually consist of two major components: a) feature extraction and b) metric learning. Previous works on person ReID focus on either constructing informative features, or finding a discriminative distance metric. 


According to the used representation forms in matching stage, we roughly divide the existing methods into two groups:
feature vector based methods, , <cit.>; and feature set or feature sequence based methods, , <cit.>.

In feature vector based methods, an image or video is represented by a feature vector and the metric learning is performed based on feature vectors. For example, in <cit.>, hand-crafted local features are integrated into a feature vector, and distance metric is learned by simultaneously maximizing inter-class margins and minimizing intra-class variations. Meanwhile, many recent works directly learn comparable feature embedding from the raw input data via a neural network. For example, in <cit.>, high-quality local patterns are explored from images or videos firstly and then aggregated into informative feature vectors; in <cit.>, local features of recurrent appearance data are extracted and integrated using temporal-pooling strategy; in <cit.>, to enhance the generalization capability of the learned embeddings, the pairwise similarity criterion is extended to triplet or quadruplet. Although these methods mentioned above are able to learn task-specific compact embeddings, these methods still suffer from the mismatching problem, especially when some vital visual details fail to be captured. 







Different from the feature vector based methods, feature set or feature sequence based methods are capable of preserving more detailed visual cues by leveraging complementary feature vectors or spatial information. 

For example, in <cit.>, local spatial constraints are adopted when computing spatial-patch based feature sets similarity;
in <cit.>, dense element-wise correspondences are employed when computing the distance of temporal feature sequences;
in <cit.>, spatial correspondence structures are explored via the patch comparison layer inserted in a deep network;
in <cit.>, the body structure information is utilized to facilitate the semantic alignment of feature sequences.
While these methods mentioned above exploit heuristic correspondence structures to compare feature sequences, they might easily fail when heavy misalignments or interferences occur in feature sequences. 


Recently, attention mechanism has been proposed 
in many tasks of matching sequences or learning representations, , <cit.>. In <cit.>, attention mechanism is used to softly align word embeddings between each text-sequence pair in the task of natural language processing. In <cit.>, glimpse representation is produced for each image via neural attention, so that each input pair can be compared progressively. In <cit.>, fixed-dimension feature vectors are learned from variable length videos for face recognition and person ReID by attentive aggregation, respectively.
However, these methods either consider a single intra-sequence attention for feature selection from feature set/sequence,
or consider a single inter-sequence attention for feature sets/sequences matching.
While inter-sequence attention is able to tackle the sequence misalignment problem, it might fail when interferences or corruptions occur. On the other hand, intra-sequence attention is able to tackle corruptions but it is not able to align sequences.















In this paper, we exploit the attention mechanism into feature sequence based person ReID.  
Unlike the existing methods, that compare sequences via heuristic correspondence structures, we attempt to compare two sequences via dual attention processes, in which an inter-sequence attention process is used to perform sequence alignment

and an intra-sequence attention process is used simultaneously to perform sequence refinement.





























§ OUR PROPOSAL: DUAL ATTENTION MATCHING NETWORK (DUATM)


This section will present an end-to-end trainable framework—DuATM, which consists of two modules: one for extracting feature sequences and one for matching feature sequences, as illustrated in Fig. <ref>. 









 §.§ Feature Sequence Extraction Module





In DuATM, we adopt DenseNet-121 <cit.> as the backbone of the feature sequence extraction module. Owning to the direct connections between each layer to all the subsequent layers in DenseNet, local details are better propagated to the outputs to enrich the final feature sequences. Specifically, the network architectures for image and video inputs are slightly different. 



  * Given an image 𝒳∈ℝ^H × W× 3, as illustrated in Fig. <ref> (a), the convolutional feature maps are obtained from the outputs of DenseNet-Block4. 
    Each feature vector at a specific position across all channels contains both the spatial details and semantic contexts due to its large receptive field size. Then, these feature vectors are rearranged by locations to form a feature sequence and each feature vector is further transformed into a compact embedding space via a Fully Connected (FC) layer.


  * Given a video footage 𝒳∈ℝ^H × W × 3 × T, of length T, as illustrated in  Fig. <ref> (b), each frame in video at a time-step is passed to a DenseNet to produce the frame-level feature vector. Then, a bidirectional recurrent layer is introduced to encode both the temporal-spatial appearance details and the complementary motion cues, by allowing information to be passed between time-steps. 
    Finally, all hidden states from different time-steps compose the final feature sequence for the video.




For convenience, we denote the feature sequence extraction as 𝐗 = (; Θ_), where 𝐗 is the extracted feature vectors sequence which encodes spatial or temporal information and ℱ(·; Θ_) represents the feature extraction module parameterized with Θ_. 
More specifically, we denote 𝐗={𝐱^i∈ℝ^D}_i=1^S as a feature sequence of length S. 
Each feature vector 𝐱^i is normalized to unit ℓ_2 norm before passing it to the next module.





 §.§ Sequence Matching Module








Sequence matching module is the most important component of DuATM. Note that there is no supervision information available to force the feature extraction module to learn semantically aligned feature sequences, thus one of the goals of this module is to compare each pair of possibly unaligned feature sequences (𝐗_a, 𝐗_b), where 𝐗_a = {𝐱_a^i}_i=1^S_a and 𝐗_b = {𝐱_b^j}_j=1^S_b. However, each sequence may also contain a certain amount of corrupted feature vectors (, caused by the noisy inputs). A naive 
method is to transform feature sequences into comparable 
feature vectors via average pooling, in which the misalignment or corruptions are ignored. 
Instead, we propose to refine and align each feature sequence pair at first, then compute and aggregate the distances of multiple feature pairs.


Since that the intermediate feature sequences obtained from our feature extraction module contain abundant contextual information, we 
use these contexts 
to remove the feature corruptions and 
compare feature sequences. 
Specifically, we attempt to exploit the contextual information to help feature sequence refinement and feature sequence pair alignment via the attention mechanism. 
To be more specific, if one of a feature sequence pair is treated as the memory, the refinement of each feature vector within this sequence can be achieved by an intra-sequence attention process; oppositely, if the other sequence is treated as the memory, the alignment of each feature vector can be achieved by an inter-sequence attention process; and vice versa. 

For clarity, we illustrate the sequence matching module of DuATM in Fig. <ref> (a), in which two types of attention procedures are integrated into a dual attention block for feature sequence refinement and alignment, as illustrated in Fig. <ref> (b). After refinement and alignment, the holistic sequence distance score is computed by aggregating multiple local distances between refined and aligned pairwise feature vectors of each sequence pair.











































  §.§.§ Dual Attention Block
























The dual attention block is composed of one transform layer and one attention layer, in which the transform layer is used to produce the feature-aware filter and the attention layer is used to generate the corresponding attention weights.

Without loss of generality, as an example to present the dual attention block in detail, we describe the generation process of (𝐱̅_a^i,𝐱̂_b^i), as illustrated in Fig. <ref> (b). Specifically, let 𝐱_a^i be the reference feature to be refined and aligned.



  * At first, the filter is computed through the transform layer as follows:


    𝐪_a^i  =  ReLU(BN(𝐖𝐱_a^i + 𝐛)),


where 𝐖 and 𝐛 are the weight matrix and bias vector of a linear layer, BN and ReLU represent Batch Normalization <cit.> and rectified linear unit (ReLU) function, respectively.


  * Then, the attention significance for intra-sequence refinement and inter-sequence alignment can be computed separately through the attention layer as follows:


    e̅_a^i,m = ⟨𝐪_a^i, 𝐱_a^m⟩,     ê_b^i,n = ⟨𝐪_a^i ,  𝐱_b^n⟩,


where ⟨·,·⟩ denotes the inner product.


  * Finally, the semantically refined and aligned feature vector pair (𝐱̅_a^i,𝐱̂_b^i) is obtained by linearly combining elements within the corresponding sequences, respectively, via normalized attention weights as









    𝐱̅_a^i = ∑_m=1^S_aσ(e̅_a^i,m)𝐱_a^m,    𝐱̂_b^i = ∑_n=1^S_bσ(ê_b^i,n)𝐱_b^n,

where σ(·) is a soft-max function for normalization in which σ(t_j)=exp(t_j)/∑_j=1^S exp(t_j) for 𝐭∈ℝ^S.



Following 
Eq. (<ref>) to (<ref>), the 
comparable feature pairs with respect to each feature vector 
can be obtained.





  §.§.§ Distance Computation and Aggregation




Owing to the feature sequence refinement and alignment performed in the dual attention block, it 
is reasonable to directly compute the distance between two refined and simultaneously aligned features, and aggregate the computed distances of feature-pairs into a holistic sequence distance.


In DuATM, the dual attention is bidirectional, , the dual attention process is carried out twice with respect to {𝐱_a^i} and {𝐱_b^j}, respectively. Thus, we use the the distances of sequence-pair in both two comparison directions 
to define the distance of the holistic sequences.


Specifically, we use the Euclidean distance to compute the distance between feature pair, ,


    d_a^i = _a^i - _b^i _2,   i=1,⋯,S_a,
    
    d_b^j = _b^j - _a^j _2,   j=1,⋯,S_b.


And then, we aggregate these distances via the average-pooling to define the distance of feature sequences 𝐗_a and 𝐗_b as follows:


    𝐗_a - 𝐗_b __ = 1/2S_a∑_i=1^S_a d_a^i + 1/2S_b∑_j=1^S_b d_b^j,


where 𝐗_a - 𝐗_b __ is the distance defined by the sequence matching module.


For convenience, we denote all parameters (, 𝐖, 𝐛, and the parameters in the BN layer) in the sequence matching module as Θ__.










 §.§ Loss Functions for Training DuATM


 


To train the whole network to perform person ReID and generalize well on unseen data, we use the siamese architecture with triplet loss during the training period as shown in Fig. <ref>. Moreover, 
to make the learned intermediate feature sequences 
compact, robust, and more discriminative, we also combined with two auxiliary losses, de-correlation loss and cross-entropy loss. Thus, the overall loss function is defined as:


    ℓ = ℓ^(0)(, Θ_,Θ_) + λ_1 ℓ^(1)(, Θ_) + λ_2 ℓ^(2)(, Θ_, θ),

where λ_1>0 and λ_2>0 are two tradeoff parameters. 





Triplet Loss The 
objective of using triplet loss is to force 
the network to make the distance between the positive pairs smaller than the negative ones.

Given a triplet of person images/videos, the extraction module extracts spatial/temporal-spatial context-aware feature sequences via a three-branch siamese subnet, and the matching module attentively computes the distances between the positive and negative pair via a two-branch siamese subnet.


Let =(𝒳_, 𝒳_+, 𝒳_-) be a triplet input. To force the network to predict the distance of  positive pair smaller than the negative pair with a 
margin γ, we define the triplet loss ℓ^(0)(, Θ_, Θ_) as: 


    max{0, γ + (𝒳_)-(𝒳_+)__ -(𝒳_),(𝒳_-)__},


where γ > 0 (, γ=0.2 in our experiments).



De-Correlation Loss
In <cit.>, de-correlating representations has been studied as a regularizer to reduce over-fitting in deep networks.
In this paper, we formulate a similar but different de-correlation loss to make our feature sequence more compact. 
Specifically, we 
impose a constraint on the intra-sequence correlation matrix as follows:


    ℓ^(1)(, Θ_) = 1/N^2||𝐈 - (𝒳)^T (𝒳)||_F^2,

where 𝐈 is an identity matrix and N is the total number of feature vectors in the sequence.

Cross-Entropy Loss with Data Augmentation To learn more informative and robust feature sequences, we also use a cross-entropy loss with data augmentation approach. Specifically, we use the data with the same labels to generate more data for training.

Suppose 𝐗=(𝒳) is an intermediate feature sequence, we achieve this goal by first pooling the sequence as =∑_i=1^Sω_i 𝐱_i, where ∑_i ω_i=1 and ω_i ≥ 0, and then passing the aggregated vector to a FC layer followed by a cross-entropy loss:


    ℓ^(2)(, Θ_, θ)=-lnσ(𝐰_c + b_c),


where c is the same label as the input 𝒳, {𝐰_c, b_c} refer to the c_th row of the FC layer's weight matrix and bias vector, respectively, and θ contains the parameters in the new FC layer. Note that, instead of generating  by simply average pooling, we propose to introduce a random convex combination strategy into the pooling stage by randomly generating ω_i∈ [0,1] and even reset it to 0 with the probability p>0, but keeping ∑_i ω_i = 1. This can be regarded as a simplified version of the interpolation method <cit.> to augment training dataset.






















§ EXPERIMENTS




To evaluate our proposal, we conduct extensive experiments on three large-scale data sets, including Market-1501 <cit.>, DukeMTMC-reID <cit.>, and MARS <cit.>.




 §.§ Datasets, Evaluation, and Implementations


Datasets Description Market-1501 is collected from 6 cameras, which contains totally 1,501 identities and 32,668 bounding boxes generated by a DPM-detector. It is split into non-overlapping  train/test sets of 12,936/19,732 images as defined in <cit.>, and single-query evaluation mode is adopted in our experiments. DukeMTMC-reID is a subset of DukeMTMC <cit.> captured with 8 cameras for cross-camera tracking. It includes 1,404 identities, in which 
one half for training and one half for testing. Specifically, there are 2,228 queries, 17,661 galleries, and 16,522 training images, respectively.  MARS 
is an extension of Market-1501 for video-based ReID. It is composed of 8,298 tracklets for 625 identities for training, and 12,180 tracklets for 636 identities for testing as defined in <cit.>, where 
the tracklets usually contain 25-50 frames.


Evaluation Protocol
For performance evaluation, we employ the standard metrics as in most person ReID literatures, namely the cumulative matching cure (CMC) and the mean Average Precision (mAP). To compute these scores, we re-implement the evaluation code provided by <cit.> in Python.

Implementation Details

We use the DenseNet-121 <cit.> trained on ImageNet to initialize the DenseNet part in DuATM, and train our network with stochastic gradient descent (SGD) method.

To be more specific, we freeze the pre-trained DenseNet parameters and train our model for the first 100 epochs, and continue the training of the entire network for other 200 epochs. The learning rate is initialized as 0.01 and changed to 0.001 in the last 50 epochs.

An obstacle 
in training DuATM 
with triplet loss is 
lack of positive pairs compared with negative ones.

To alleviate the data imbalance issue, 
we adopt the hard triplet mining strategy <cit.> to generate triplet mini-batches.

Specifically, each mini-batch contains P persons with V images/tracklets, and all of them are regarded as anchor points to select the corresponding hard positives and negatives. In experiments, 
we set (P=10, V=4) with size 256 ×128 for image dataset, and set (P=7,V=3) with size 128 × 64 for video dataset by default.

Besides, we follow the common practices to augment image dataset by using random horizontal flips and random crops 
<cit.>, and to augment video dataset by randomly selecting video sub-sequences of 16 consecutive frames. 



The dimension D of feature vectors within each sequence is set to 256 for both image and video inputs. Besides, the hyper parameters of loss functions, , λ_1, λ_2 and corruption ratio p, are set as λ_1=0.1,λ_2=0.5, and p=0.2 when comparing with the baseline. They are tuned to more proper 
values in the parameter analysis experiments.

During the evaluation, we discard the data augmentation process except when comparing with state-of-the-art methods, and use the sub-sequences of 64 consecutive frames for video ReID.[If the tracklet has less frames, 
we circularly sample the sequence.] All experiments are implemented with PyTorch on 2 Nvidia Titan-X GPUs.





 §.§ Evaluations on Performance of DuATM



DuATM Trained with Different Losses To evaluate the contribution of each loss and the dual attention block, we train DuATM and report the results with the following four settings: a) DuATM+ℓ^(0), b) DuATM+ℓ^(0)+ℓ^(1), c) DuATM+ℓ^(0)+ℓ^(2), and d) DuATM+ℓ^(0)+ℓ^(1)+ℓ^(2).



Note that DuATM is built on DenseNet. Thus, as the baseline, we take DenseNet to extract feature sequence, use an average pooling layer to form the holistic feature vector and use Euclidean distance to compare feature vectors. The baseline is trained with the triplet loss ℓ^(0), denoted as AvePool+ℓ^(0). Experimental results are presented in Table <ref>.








As observed from Table <ref> that, the results of DuATM+ℓ^(0) consistently outperform that of AvePool+ℓ^(0) on all three data sets. This confirms the effectiveness of using dual attention block in DuATM: using context-aware feature sequences with dual attentive matching mechanism is more effective than the average-pooling based single feature vector method.
The performance is further improved when adding the de-correlation loss ℓ^(1) and the cross-entropy loss ℓ^(2).


Since that the de-correlation loss ℓ^(1) does not bring any extra supervision information for discrimination, the performance gain of DuATM+ℓ^(0)+ℓ^(1) over DuATM+ℓ^(0) is minor. Interestingly, when the cross-entropy loss is added, the performance is significantly improved. This could be accounted to the supervision information brought by the identity labels. Finally, when combining all three loss functions, the accuracy is further improved.




Ablation Study of DuATM To verify the effects of intra- and inter-sequence attentions in DuATM, we evaluate each of them separately on Market1501, denoted as Intra+ℓ^(0) and Inter+ℓ^(0). Experimental results are listed in Table <ref>, where DuATM+ℓ^(0) is nothing but Intra+Inter+ℓ^(0). We can observe from Table <ref> that, jointly using the two attentions, , the dual attention, leads to improvements in the performance than using only one type attention. This confirms the importance of using dual attention mechanism.















































Evaluation on Parameters in DuATM In the loss function of DuATM, there are two parameters λ_1 and λ_2. In training the cross-entropy loss, there is also a parameter p to control the corruption ratio in generating auxiliary data. To evaluate the influence of these parameters, we conduct experiments on three data sets by changing one parameter while fixing the other two. Experimental results are shown in Fig. <ref>.

From these results, we can draw three conclusions: a) while a moderate value λ_1 can enforce the sequences more compact, an over-large λ_1 harms the contextual relationships between feature vectors leading to slightly degenerated performance; b) a moderate value λ_2 can bring extra supervision information, but an over-large λ_2 might lead to over-fitting; 
c) the performance is not sensitive to parameter p. Also, we find that DuATM achieves the best performance with the settings of (λ_1, λ_2, p) as (0.3,0.9,0.4), (0.1,0.5,0.6), and (0.5,0.5,0.4), on Market-1501, DukeMTMC-reID, and MARS, respectively. We list these results in the bottom two rows of Table <ref>.

In addition, we conduct experiments on Market-1501 and MARS to evaluate the impact of feature dimension D and sequence length T of a video. Experimental results are shown in Fig. <ref>. For image based ReID, when each person is represented by a sequence with size D × T, even using a lower dimension, the whole sequence can still contain enough discrimination information, , the results at rank-1 still keep at 78.50% or 87.71% for D=16 or D=32, respectively. For video based ReID, since that the feature sequence length is determined by the tracklet size, a longer sequence will contain more visual cues captured at different time points and thus lead to higher accuracy, , mAP is improved from 21.87% with T=1 to 59.42% with T=96.

















Comparison to Other Attention Methods To demonstrate the effectiveness of our dual attention mechanism, in Table <ref>, we compare our method with several existing attentive methods, including CAN <cit.>, HP-Net <cit.>, ST-RNN <cit.>, and QAN <cit.>, in which the salient local patterns are extracted by the attention strategy and aggregated into a single comparable feature vector. Instead, we keep all informative local patterns during feature extraction, and use dual attention mechanism to perform local pattern refinement and pattern-pair alignment during the matching stage. Our model performs a more 
reasonable comparisons and thus achieves superior performance.




Comparison to Other Feature Sequence / Feature Set based Methods In Table <ref>, we compare the performance of our method to several existing sequence / set based methods <cit.>. Since that, DuATM can not only adaptively infer the semantic correspondence structure between local patterns but also automatically remove local corruptions within sequence, our method achieves better performance than body-part based (, SpindleNet <cit.>, DRL-PL <cit.>) and densely-matching based (, SCSP <cit.>) methods on dataset Market-1501.









Comparison to State-of-the-art Methods[Note that different backbones are adopted in different methods, , DenseNet is used in DuATM, ResNet is used in <cit.>, combined multiple networks are used in <cit.>. Thus, a comprehensive evaluation on the performance with different backbones is a worth future work.] In Table <ref>, Table <ref>, and Table <ref>, we compare our approach against the state-of-the-art methods on Market-1501, DukeMTMC-reID, and MARS, respectively. The proposed DuATM achieves superior performance on all of them, that further confirms the effectiveness of our attentively deep context-aware feature sequences based approach.

Specifically, in Market-1501 and DukeMTMC-reID, DuATM surpasses all stepwise models and end-to-end networks, and obtains rank-1 accuracy at 91.24% and 81.37% for each dataset. In MARS, DuATM is still better than most approaches above. If our DuATM is trained with a larger image size 256×128 as in <cit.>, the rank-1 will be 81.16%, which surpasses the state-of-the-art result. 












Visualization of Dual Attention Mechanism
To better understand the dual attention mechanism used in our DuATM, we display some intermediate visualization results in Fig. <ref>. 
Since that the learned feature vectors within each sequence are context-aware, with reference to each feature vector, the intra-sequence attention can concentrate on its context-related body-parts or gaits from the same image or video to refine itself, and the inter-sequence attention can simultaneously concentrate on the semantically consistent body-parts or gaits from the opposite image or video to generate its aligned counterpart, even when the reference feature is derived from a corrupted region as in Fig. <ref> (c). Consequently, in DuATM, the feature sequences are semantically refined and aligned, and thus properly compared.









§ CONCLUSIONS







We proposed an end-to-end trainable framework, namely Dual ATtention Matching network (DuATM), to learn context-aware feature sequences and to perform dually attentive comparison for person ReID. The core component of DuATM is a dual attention block, which simultaneously performs feature refinement and feature-pair alignment. DuATM is trained via a triplet loss assisted with a de-correlation loss and a cross-entropy loss. Experiments conducted on large-scale image and video data sets have confirmed the significant advantages of our proposal.

 



§ ACKNOWLEDGMENTS



This work was carried out at the Rapid-Rich Object Search (ROSE) Lab, Nanyang Technological University, Singapore. The ROSE Lab is supported by the National Research Foundation and the Infocomm Media Development Authority, Singapore. J. Si,  H. Zhang, and C.-G. Li are supported by Beijing Municipal Science and Technology Commission Project under Grant No. Z181100001918005. C.-G. Li is also partially supported by the Open Project Fund from Key Laboratory of Machine Perception (MOE), Peking University. The authors would like to thank the support of the NVIDIA AI Technology Center for their donation/contribution of Titan X GPUs used in our research.

ieee







