Multi-Granularity Reference-Aided Attentive Feature Aggregation for Video-based Person Re-identification
Zhizheng Zhang Cuiling Lan Wenjun Zeng Zhibo Chen
cs.CV









Multi-Granularity Reference-Aided Attentive Feature Aggregation
 for Video-based Person Re-identification
    Zhizheng Zhang^1This work was done when Zhizheng Zhang was an intern at MSRA.     Cuiling Lan^2Corresponding author.
    Wenjun Zeng^2 
    Zhibo Chen^1†
^1University of Science and Technology of China     ^2Microsoft Research Asia

	zhizheng@mail.ustc.edu.cn   {culan,wezeng}@microsoft.com   chenzhibo@ustc.edu.cn












    version of December 30, 2023
==========================================================================================================================================================================================================================================================================================================================================

empty



    Video-based person re-identification (reID) aims at matching the same person across video clips. It is a challenging task due to the existence of redundancy among frames, newly revealed appearance, occlusion, and motion blurs. In this paper, we propose an attentive feature aggregation module, namely Multi-Granularity Reference-aided Attentive Feature Aggregation (MG-RAFA), to delicately aggregate spatio-temporal features into a discriminative video-level feature representation. In order to determine the contribution/importance of a spatial-temporal feature node, we propose to learn the attention from a global view with convolutional operations. Specifically, we stack its relations, , pairwise correlations with respect to a representative set of reference feature nodes (S-RFNs) that represents global video information, together with the feature itself to infer the attention. Moreover, to exploit the semantics of different levels, we propose to learn multi-granularity attentions based on the relations captured at different granularities. Extensive ablation studies demonstrate the effectiveness of our attentive feature aggregation module MG-RAFA. Our framework achieves the state-of-the-art performance on three benchmark datasets.
    
    





§ INTRODUCTION


Person re-identification (reID) aims at matching persons in different positions, times, and camera views. Many researches focus on image-based setting by comparing the still images <cit.>. With the prevalence of video capturing systems, person reID based on video offers larger capacity for achieving more robust performance. As illustrated in Figure <ref>, for a video clip, the visible contents of different frames differ but there are also overlaps/redundancy. In general, the multiple frames of a video clip/sequence can provide more comprehensive information of a person for identification, but also raise more challenges, such as the handling of the presence of abundant redundancy, occlusion, motion blurs.


[1]All faces in the images are masked for anonymization.

A typical video-based person reID pipeline <cit.> extracts and aggregates spatial and temporal features to obtain a single feature vector as the video representation. To make the video-level feature representation precise, comprehensive and discriminative, we should latch onto the informative features from a global view, and meanwhile, remove the interference.

Attention, which aims at strengthening the important features while suppressing the irrelevant ones, matches the aforementioned goal well. Some works have studied spatio-temporal attention <cit.> or the attentive recurrent neural networks <cit.> to aggregate spatial and temporal features. They learn the attention weights for spatial and temporal dimensions separately or sequentially <cit.>. However, due to the lack of a global view, they suffer from the difficulty in precisely determining whether a feature of some position is important and what the degree of redundancy is within the entire video clip. A diversity regularization is adopted in <cit.> to remedy this issue, but only alleviates it to some extent. A powerful model is expected which jointly determines the importance levels of each spatio-temporal feature from a global view. Besides, as shown in Figure <ref>, discriminative factors/semantics could be captured at different granularities (regions of different sizes) by human. However, there is a lack of effective mechanisms to explore such hierarchical characteristics.


In this paper, we propose a Multi-Granularity Reference-aided Attentive Feature Aggregation (MG-RAFA) scheme for video-based person reID. For effective feature aggregation of the spatial and temporal positions, we determine the importance of each feature position/node from a global view, and consider the hierarchy of semantics during this process. For each feature position, we use its relations/affinities with respect to all reference feature nodes, which represent the global structural information (clustering-like patterns), together with the feature itself (appearance information) to model and infer the attention weights for aggregation. This is in part inspired by Relation-aware Global Attention (RGA) <cit.> designed for effective image feature extraction. However, a 3D video is rather different from a 2D image, where a video clip generally presents abundant redundancy along the time dimension, and the spatio-temporal structure patterns are complicated due to the diversity of human poses. 


Considering the characteristics of video, we propose to construct a small but representative set of reference feature nodes (S-RFNs) for globally modelling the pairwise relations, instead of using all the original feature nodes. S-RFNs provides a simplified but representative reference for modeling global relations, which not only eases the difficulty of attention learning but also reduces the computational complexity. Moreover, we also take into account that the semantics are diverse in their granularities as illustrated in Figure  <ref>. We propose to hierarchically model relations for attentive feature aggregation, which allows attention learning to be more precise and adaptive with low computational complexity.


In summary, we have three main contributions: 




  * For video-based person reID, we propose a simple yet effective Multi-Granularity Reference-Aided Attention Feature Aggregation (MG-RAFA) module for the joint spatial and temporal attentive feature aggregation.


  * To better capture the discriminative semantics at different granularities, we exploit the relations at multiple granularities to infer attention for feature aggregation.



  * We propose to build a small but representative reference set for more effective relation modeling by compressing the redundancy information of video data. 


We conduct extensive experiments to evaluate our proposed feature aggregation for video-based person reID and demonstrate the effectiveness of each technical component. The final system significantly outperforms the state-of-the-art approaches on three benchmark datasets. Besides, the proposed multi-granularity module MG-RAFA further reduces the computational complexity as compared to its single-granularity version SG-RAFA via our innovative design. Our final scheme only slightly increases the computational complexity over the baseline (<1%).








§ RELATED WORK


In many practical scenarios, video is ready for access and contains more comprehensive information than a single image. Video-based person reID offers larger optimization space for achieving high reID performance and attracts more and more interests in recent years.

Video-based Person ReID. Some works simply formulate the video-based person reID problem as an image-based reID problem, which extracts the feature representation for each frame and aggregates the representations of all the frames using temporal average pooling <cit.>. McLaughlin apply Recurrent Neural Network on the frame-wise features extracted from CNN to allow information to flow among different frames, and then temporally pool the output features to obtain the final feature representation <cit.>. Similarly, Yan leverage LSTM network to aggregate the frame-wise features to obtain a sequence-level feature representation <cit.>. Liu propose a two-stream network in which motion context together with appearance features are accumulated by recurrent neural network <cit.>. Inspired by the exploration of 3D Convolutional Neural Network for learning the spatial-temporal representation in other video-related tasks such as action recognition <cit.>, 3D convolution networks are used to extract sequence-level feature <cit.>. These works treat the features with the same importance even though the features for different spatial and temporal positions have different contribution/importance levels for video-based person reID.  

Attention for Image-based Person ReID. For image-based person reID, many attention mechanisms have been designed to emphasize important features and suppress irrelevant ones for obtaining discriminative features. Some works use the human part/pose/mask information to infer the attention regions for extracting part/foreground features <cit.>. Some works learn the attention in terms of spatial positions or channels in end-to-end frameworks <cit.>. In <cit.>, spatial attention and channel attention are adopted to modulate the features. In general, convolutional layers with limited receptive fields are used to learn the spatial attention. Zhang propose a relation-aware global attention to globally learn attention by exploiting the pairwise relations <cit.> and achieve significant improvement for image-based person reID. Despite the wide exploration in image-based reID, attention designs are under-explored for video-based reID, with much fewer efforts on the globally derived attention.


In this paper, motivated in part by <cit.> which is designed for effective feature learning of an image by exploring relations, we design a multi-granularity reference-aided attentive feature aggregation scheme for video-based person reID. Particularly, to compute the relations, we build a small set of  reference nodes instead of using all the nodes for robustness and computational efficiency. Moreover, multi-granularity attention is designed to capture and explore the semantics of different levels.

Attention for Video-based Person ReID. For video-based person reID, some attention mechanisms have been designed. One category of works considers the mutual influences between the pair of sequences to be matched <cit.>. In <cit.>, the temporal attention weights for one sequence was guided by the information from distance matching with the other sequence. However, given a sequence in the gallery set, it needs to prepare different features corresponding to different query images, which is complicated and less friendly in practical application. 



Another category of works independently determines the features of the sequence itself. To weaken the influence of noisy frames, Liu propose a quality aware network (QAN), which estimates the quality score of each frame for aggregating the temporal features as the final feature  <cit.>. Zhou use learned temporal attention weights to update the current frame features as input of RNN <cit.>. Zhao disentangle the features of each frame to semantic attribute related sub-features and re-weight them by the confidence of attribute recognition for temporal aggregation <cit.>. These works do not simultaneously generate spatial and temporal attention from a global view for feature aggregation. Recently, spatial-temporal map which is directly calculated from the feature maps is used to aggregate the frame-level feature maps, without using any additional parameters <cit.>. However, as attention is computed in a pre-defined manner, the optimization space is limited.

Considering the lack of effective joint spatio-temporal attention mechanisms for the feature aggregation in video-based person reID, we intend to address this by proposing a multi-granularity reference-aided global attention, which jointly determines the spatial and temporal attention for feature aggregation.




§ MULTI-GRANULARITY REFERENCE-AIDED ATTENTIVE FEATURE AGGREGATION


We propose an effective attention module, namely Multi-Granularity Reference-aided Global Attention (MG-RAFA), for spatial-temporal feature aggregation to obtain a video-level feature vector. In  Section <ref>, we introduce the preliminary. Then, we describe our proposed reference-aided attentive feature aggregation under single-granularity setting in Section <ref> and elaborate the multi-granularity (our final design of MG-RAFA) in Section <ref>. We finally present the loss functions in Section <ref>.




 §.§ Overview



For video-based person reID, we aim at designing an attentive feature aggregation module that can comprehensively capture discriminative information and exclude interference from a video which in general contains redundancy, newly revealed contents, occlusion and blurring. To achieve this goal, a joint determination of attention for the spatio-temporal features from a global view is important for robust performance.


We propose to learn the attention for each spatio-temporal position/node by exploring the global relations with respect to a set of reference feature nodes. Particularly, for the global relations modeling of a target node, we build a small set of representative feature nodes as reference, instead of using all the feature nodes, to ease optimization difficulty and reduce the computational complexity. Moreover, the discriminative information may physically spread over different semantic levels as illustrated in Figure <ref>. We thus introduce hierarchical (multi-granularity) relation modeling to capture the semantics at different granularities.


Figure <ref> gives an illustration of our overall framework. For a video tracklet, we sample T  frames as 𝒱={I_1, I_2,⋯, I_T}. Through a single frame feature extractor (, ResNet-50 backbone), we obtain a set of feature maps F_all={F_t | t=1,2,⋯,T}, where F_t ∈ℝ^H × W × C includes H × W feature nodes, (H, W, C represent the height, width and number of channels, respectively). Based on the proposed multi-granularity reference-aided attention, all feature nodes in this set are weighted summed into a feature vector v as the final video-level feature representation for matching by l_2 distance. For clarity, we first present our proposed reference-aided attentive feature aggregation under the single-granularity setting in Subsection <ref> and introduce the multi-granularity version in Subsection <ref>.



 §.§ Reference-aided Attentive Feature Aggregation




The extracted feature set F_all={F_t | t=1,2,⋯,T} consists of K=H × W × T feature nodes, each of which is a C-dimensional feature vector. To determine the importance level of a feature node, it would be helpful if all the other feature nodes are also “seen", since intuitively people can determine the relative importance of something by comparing it with all others. For a feature node, to determine its importance level, we prepare its relation/affinity with every node as the ingredient to infer the attention. For any node i, when stacking its relations with all nodes (, in raster scan order), the number of relation elements is D=H× W × T.


Taking into account the existence of appearance variations (, caused by pose, viewpoint variations) and large redundancy among frames, the distribution space for relation vectors is large and may cause difficulty in mining the patterns for accurately determining the attention. Thus, we propose to ease the difficulty by choosing a small set of representative feature nodes, instead of all the nodes, as the reference for modeling relations. As we know, for a video tracklet, there is usually large redundancy across temporal frames. For video action recognition, Bobick <cit.> propose to use a static vector-image where the vector value at each point is a function of the motion properties at the corresponding spatial location of a video sequence to  compactly represent the information of a video. Motivated by this, we adopt average pooling along the temporal frames to fuse F_all={F_t|t=1,⋯,T} into a feature map F_R ∈ℝ^H × W × C. Different from action recognition where the motion/temporal evolution is important, the temporal motion and evolution in general has no discriminative information for person ReID while the appearances are the key. Thus we simply average the temporal frames to obtain F_R as the reference, , the representative set of reference feature nodes (S-RFNs), to model global relations, which consists of D=H× W feature nodes. 

For a feature node x^i∈ℝ^C in F_all, we calculate the relations/affinities between it and all feature nodes in the reference set F_R to model its corresponding relations. A pairwise relation is formulated as the correlation of the two nodes in embedding spaces:

    r_i,j = r(x^i, y_r^j)=μ(x^i)^T·ν(y_r^j),

where y_r^j∈ℝ^C denotes a feature node in the reference set F_R, i (and j) identifies the node index. We define  μ(x^i)=ReLU(W_μx^i) and ν(y_r^j)=ReLU(W_νy_r^j), where W_μ∈ℝ^(C/s)× C and W_ν∈ℝ^(C/s)× C are learned weight matrices, where s is a positive integral which controls the dimension reduction ratio. We implement it by adopting a 1× 1 convolutional filter followed by Batch Normalization (BN) and ReLU activation, respectively. Note that we omit BN operations to simplify the notation. By stacking the pairwise relations of the feature node x^i with all the nodes (, scanned in raster scan order) in the reference set F_R, we have the relation vector as

    𝐫_i = [r_i,1, r_i,2,⋯,r_i,D] ∈ℝ^D,

which compactly reflects the global and clustering-like structural information. In addition, since the relations are stacked into a vector with a fixed scanning order with respect to the reference nodes, the spatial geometric information is also contained in the relation vector. 











Intuitively, a person can have a sense of the importance levels of a node once he or she obtains the affinity/correlation of this node with many other ones. Similarly, the relation vector which describes the affinity/relation with all reference nodes provides valuable structural information.    
Particularly, the original feature x^i represents local appearance information while the relation feature r^i models global relations. They complement and reinforce each other but in different semantic spaces. We thereby combine them together in their respective embedding space and jointly learn, model, and inference the level of importance (attention scores) of the feature node x^i through a modeling function as

    a^i=θ([ϕ(x^i), ψ(r^i)]),

where ϕ(·) and ψ(·) are two embedding functions, [·,·] represents the concatenation operation, and θ(·) represents a modeling function to inference the attention vector a^i∈ℝ^C corresponding to x^i. We define ϕ(x^i)=ReLU(W_ϕx^i), ψ(r^i)=ReLU(W_ψr^i), and θ([ϕ(x^i), ψ(r^i)])=ReLU(W_θ([ϕ(x^i), ψ(r^i)]), where W_ϕ∈ℝ^(C/s) × C, W_ψ∈ℝ^(D/s) × D and W_θ∈ℝ^C × ((C/s)+(D/s)) are learned weight matrices. We implement them by performing 1× 1 convolutional filtering followed by BN and ReLU. For each feature node x^i in F_all (nodes corresponding to all the spatial and temporal positions), we obtain an attention score vector a^i. For all nodes in F_all, we have 𝒜 = [a^1,a^2,⋯,a^K].


We normalize the learned attention scores via the Softmax function across different spatial and temporal positions (node indexes) and obtain the final attention â^i, i=1,2,⋯,K. Afterwards, we use the final attention as the weights to aggregate all the feature nodes (from all spatial and temporal positions) in F_all. Mathematically, we obtain the final sequence-level feature representation v∈ℝ^C by

    v=∑_k=1^Kâ^i⊙x^i,   â^i=Softmax(a^i),

where symbol ⊙ represents element-wise multiplication. 




 §.§ Multi-Granularity Attention




Human can capture the different semantics (such as gender, body shape, clothing details) of a person at different granularity levels (, in terms of viewing distance or image resolution). Some types of semantics (, whether the person wears glasses or not) may be easier to capture at fine granularity while some others (, body shape) may be easier to  capture at coarse granularity by excluding the distraction from fine details. Motivated by this, we propose the Multi-Granularity Reference-aided Attentive Feature Aggregation (MG-RAFA) which derives the attention and introduces a hierarchical design, aiming at capturing the discriminative spatial and temporal information at different semantic levels. Basically, we distinguish the different granularities by modeling relations and deriving attention on feature maps of different resolutions. 

Following the earlier notations, for both reference nodes in F_R and the nodes to be aggregated in F_all, we split them along their channel dimensions into N splits/groups. Each group corresponds to a granularity. In this way, we reduce the computational complexity in comparison with the single granularity case. For the m^th granularity, we perform spatial average pooling with a ratio factor m on both the m^th split features of F_R and F_t, t=1, 2, ⋯, T. We obtain the factorized reference feature F_R,m∈ℝ^H_m×W_m×C/N of D_m = H_m×W_m nodes, where H_m = H/2^m-1 and W_m = W/2^m-1. Similarly, we obtain the factorized feature map on frame t as F_t,m∈ℝ^H_m×W_m×C/N and the spatial and temporal feature node set as F_all,m = {F_t,m| t=1, 2, ⋯, T}. 


Then, we employ the reference-aided attentive feature aggregation as described in Section <ref> for each group separately. Thereby, the relation modeling in Eq. (<ref>) and the attention modeling function in Eq. (<ref>) can be expanded into their multi-granularity versions as 

    r(x_m^i, y_m^j)=μ_m(x_m^i)^T·ν_m(y_m^j),


    a_m^i=θ_m([ϕ_m(x_m^i), ψ_m(r_m^i)]),

where subscript m identifies the index of granularity, x_m^i denotes the i^th node in F_all,m and y_m^j denotes the j^th nodes in the reference feature  map F_R,m. Similar to the feature aggregation under single granularity in Section <ref>, we normalize the attention scores via Softmax function and weighted sum the feature nodes (across different spatial-temporal positions). Finally, we concatenate the fused feature of each split/group (denoted by v_m) to obtain the final sequence-level feature representation v = [v_1, v_2, ⋯, v_N].




 §.§ Loss Design



We add the retrieval-based loss, , the triplet loss with hard mining L_Tr, and the ID/classification loss (cross entropy loss with label smoothing <cit.>) denoted by L_ID, on the video feature vector v. Each classifier consists of a Batch Normalization (BN) layer followed by a fully-connected (FC) layer. Specially, to encourage the network to aggregate discriminative features at each granularity, we add the two loss functions on the aggregated feature of each granularity v_g, g=1,⋯,N. The final loss is:

    L = L_ID(v) + L_Tr(v) +1/N∑_g=1^N( L_ID(v_g) + L_Tr(v_g) ).






§ EXPERIMENTS




 §.§ Datasets and Evaluation Metrics


































































We evaluate our approach on three video-based person reID datasets, including MARS <cit.>, iLIDS-VID <cit.>, and PRID2011 <cit.>. Table <ref> gives detailed information. Following the common practices, we adopt the Cumulative Matching Characteristic (CMC) at Rank-1 (R-1), to Rank-20 (R-20), and the mean average precision (mAP) as the evaluation metrics. For MARS, we use the  train/test split protocol defined in  <cit.>. For iLIDS-VID and PRID2011, similar to <cit.>, we report the average CMC across 10 random half-half train/test splits for stable comparison. 










 §.§ Implementation Details


Networks. Similar to <cit.>, we take ResNet-50 <cit.> as our backbone for per-frame feature extraction. Similar to <cit.>, we remove the last spatial down-sampling operation in the conv5_x block for both the baseline and our schemes. In our scheme, we apply our propose MG-RAFA after the last residual block (conv5_x) for the attentive feature aggregation to obtain the final feature vector v. We build Baseline by taking the feature vector v obtained through global spatial-temporal average pooling.



Experimental Settings. We uniformly split the entire video into T=8 chunks and randomly sample one frame per chunk. For the triplet loss with hard mining <cit.>, in a mini-batch, we sample P identities and each identity includes Z different video tracklets. For MARS, we set P=16 and Z=4 so that the mini-batch size is 64. For iLIDS-VID and PRID2011, we set P=8 and Z=4. We use the commonly used data augmentation strategies of random cropping <cit.>, horizontal flipping, and random erasing <cit.> (with a probability of 0.5 at the sequence level for both the baselines and our schemes. Sequence-level data augmentation is much superior to frame-level one. This is closer to the realistic data variation and
will not break the inherent consistency among frames. We set the input  resolution of images to be 256 × 128 with T=8 frames. Adam optimizer is used. Please find more details in the supplementary.






 §.§ Ablation Study






  §.§.§ Effectiveness Analysis
 



We validate the effectiveness of our proposed multi-granularity reference-aided attention (MG-RAFA) module and show the comparisons in Table <ref>.

MG-RAFA vs. Baseline. Our final scheme MG-RAFA (N=4) outperforms Baseline by 2.9%, 2.1% and 3.4% on Mars, iLIDS-VID, and PRID2011, respectively. This demonstrates the effectiveness of our proposed attentive aggregation approach. 



Effectiveness of using the Global Relations. In Table <ref>, MG-AFA (N=4) denotes the scheme when we use the visual feature alone (without using relations) to learn the attention. Our scheme MG-RAFA (N=4) which uses relations outperforms MG-AFA (N=4) by 2.2%,  1.9%, and 3.3% in Rank-1 on Mars, iLIDS-VID, and PRID2011, respectively, indicating the effectiveness of leveraging global relations for learning attention.


Single Granularity vs. Multiple Granularity. 
SG-RAFA(S=1) also takes advantage of relations but ignores the exploration of semantics at different granularities. In comparison with SG-RAFA(S=1), our final scheme MG-RAFA(N=4) that explores relations at multiple granularities achieves 1.0%, 1.5%, 2.3% Rank-1 improvements on Mars, iLIDS-VID, PRID2011, respectively. MG-RAFA is effective in capturing correlations of different granularity levels.






Moreover, we study the effects of different numbers of granularities by comparing MG-RAFA(N=4) with MG-RAFA(N=2). The results show that finer granularity delivers better performance. Note that the spatial resolution of the frame features F_t is 16×8. The spatial resolution ratio between two adjacent granularity levels is set to 4, which allows the maximum number of granularity levels N to be 4 (, 16×8, 8×4, 4×2, and 2×1) in this work. In the subsequent description, we use MG-RAFA to refer to MG-RAFA(N=4) unless otherwise specified.

To further demonstrate that the improvements come from the relation modeling at varying granularities instead of multiple attention masks, we compare MG-RAFA(N=4) with the single-granularity setting SG-RAFA(S=4). In this setting, the features are divided into four splits(groups) along channels with each split having an attention mask rather than a shared one. Each attention mask is derived from the same fine gruanularity. The results show that MG-RAFA(N=4) is superior to SG-RAFA(S=4).


Complexity. Thanks to the channel splitting and spatial pooling, the computational complexity (FLOPS) of our multi-granularity  module MG-RAFA(N=4) is only 9.2% of that of the single granularity module SG-RAFA(S=1).
 





  §.§.§ Selection of the Reference Feature Nodes




In our scheme, we take a set of feature nodes (S-RFNs) as reference to model pairwise relations. Instead of taking all feature nodes in the frame features as reference, considering the larger temporal redundancy, an average pooling operation along the time dimension is performed to reduce the number of nodes for easing optimization and reducing computational complexity. For clarity, we investigate different strategies for building the S-RFNs under single granularity setting (, SG-RAFA) and show the results in Table <ref>.





The spatial resolution of frame feature is H× W, where H=16, W=8 in this paper, resulting in 128 feature nodes for each frame feature F_t. The number of temporal  frames is T=8. 
Ours: we obtain S-RFNs by fusing frame features F_t, t=1,⋯,T along the time dimension and obtain a feature map with 16 × 8 = 128 feature nodes. S-P: we fuse feature nodes to obtain the reference set via average pooling along spatial dimensions. T-P: we perform average pooling along the temporal dimension with different ratios to obtain different settings in Table <ref>. ST(16×8×8): we take all the spatial and temporal nodes as the reference set.

We have the following observations. (1) Ours outperforms schemes S-P(8×1×8), S-P(8×2×8), S-P(4×4×8), S-P(8×4×8) with spatial pooling by 1.2%, 1.2%, 1.0% and 0.9% in mAP respectively, where spatial pooling may remove too much useful information and result in inferior S-RFNs. (2) Ours also outperforms those with partial temporal pooling. The performance increases as the temporal pooling degree increases. (3) Ours outperforms the scheme ST(16×8×8) without any pooling by 0.8% in mAP. Using all nodes as reference results in a larger optimization space and the diversity of temporal patterns is complex which makes it difficult to learn. In contrast, through temporal average pooling, we reduce the pattern complexity and thus ease the learning difficulty and computational complexity.

Complexity. Thanks to the selection of S-RFNs, in comparison with ST(16×8×8) which uses all the feature nodes as reference, the computational complexity in terms of FLOPs for our aggregation module is reduced from 6.697G to 2.301G while the performance is 0.8% higher in mAP.



















































  §.§.§ Comparison with Non-local




Non-local block <cit.> explores long-range context, which weighted sums the features from all positions to refine the current position feature. Both our approach and non-local can explore the global context. However, non-local block uses a deterministic way, , weighted summation (without parameters), to exploit the global information, which limits its capability. In contrast, ours could mine the structural pattern and semantics from the stacked relations by leveraging a learned model/function to inference the importance of a feature node as attention, being more flexible and having large optimization space. 

Table <ref> shows the comparisons to non-local schemes. We added a non-local module on the feature maps F_t, t=1,⋯,T for feature refinement followed by spatio-temporal average pooling. NL(ST) denotes that non-local is performed on all the spatio-temporal features and NL(S) denotes that non-local is performed within each frame. Our SG-RAFA outperforms NL(ST) and NL(S) significantly by 2.4% and 1.9% in mAP, respectively. NL(ST) is inferior to NL(S) which may be caused by the optimization difficulty when spatio-temporal dimensions are jointly considered.              






  §.§.§ Extension of MG Design to Other Attention




Different semantics could be suitably captured at different granularities (as illustrated in Figure <ref>). Our proposed multi-granularity design can also be applied to other attention mechanisms. We conduct experiments by applying several different attention designs on the extracted feature maps F_t, t=1,⋯, T and show the results in Table <ref>. Compared with the single gruanularity versions, multi-granularity design brings 1.5%, 1.4%, and 1.7% improvements in mAP respectively for RGA-SC <cit.>, SE <cit.>, and CBAM <cit.>, demonstrating the effectiveness of the proposed multi-granularity design. In addition, our proposed MG-RAFA outperforms  RGA-SC(MG), SE(MG), CBAM(MG) by 0.9%, 1.6%, and 1.3% in mAP respectively.







 §.§ Comparison with State-of-the-arts


Table <ref> shows that our MG-RAFA significantly outperforms the state-of-the-arts. On Mars, compared to STA <cit.>, our method achieves 5.1% improvements in mAP. On iLIDS-VID and PRID2011, ours outperforms the second best approach by 2.3% and 0.4% in Rank-1, respectively. 








 §.§ Visualization Analysis






We visualize the learned attention values at different spatial-temporal positions at different granularities in Figure <ref>. We have two observations from (a). (1) The learned attention tends to focus on different semantic regions from different frames, which gets rid of a lot of repetitions (redundancy). (2) Interestingly but not surprisingly, our attention is able to select the better represented areas and exclude the inferences (, see the 3^rd and 4^th columns of the right sub-figures in (a) where there are occlusions). We believe our attention modeling is an effective method to capture and learn discriminative spatial and temporal representation. (b) shows MG-RAFA captures different semantics at different granularities, which tends to capture more details at finer granularities and larger body parts at coarser granularities.




[2]We do not include results on DukeMTMC-VideoReID <cit.> since this dataset is not publicly released anymore.





§ CONCLUSION

In this paper, we propose a Multi-Granularity Reference-aided Attentive Feature Aggregation scheme (MG-RAFA) for video-based person re-identification, which effectively enhances discriminative features and suppresses identity-irrelavant features on the spatial and temporal feature representations. Particularly, to reduce the optimization difficulty, we propose to use a representative set of reference feature nodes (S-RFNs) for modeling the global relations. Moreover, we propose multi-gruanularity attention by exploring the relations at different granularity levels to capture semantics at different levels. Our scheme achieves the state-of-the-art performance on three benchmark datasets. 




§ ACKNOWLEDGEMENTS

This work was supported in part by NSFC under Grant U1908209, 61632001 and the National Key Research and Development Program of China 2018AAA0101400.

ieee_fullname



