Spatial and Temporal Mutual Promotion for Video-based Person Re-identification
Yiheng Liu Zhenxun Yuan Wengang Zhou Houqiang Li






Spatial and Temporal Mutual Promotion for Video-based Person Re-identification
    Yiheng Liu^1,  Zhenxun Yuan^2,  Wengang Zhou^1,  Houqiang Li^1

^1CAS Key Laboratory of Technology in GIPAS, EEIS Department, University of Science and Technology of China 
 ^2 School of Electrical and Computer Engineering, Purdue University

lyh156@mail.ustc.edu.cn,  yuan141@purdue.edu,  {zhwg,lihq}@ustc.edu.cn


    December 30, 2023
===============================================================================================================================================================================================================================================================================================================================


Video-based person re-identification is a crucial task of matching video sequences of a person across multiple camera views. Generally, features directly extracted from a single frame suffer from occlusion, blur, illumination and posture changes. This leads to false activation or missing activation in some regions, which corrupts the appearance and motion representation. How to explore the abundant spatial-temporal information in video sequences is the key to solve this problem. To this end, we propose a Refining Recurrent Unit (RRU) that recovers the missing parts and suppresses noisy parts of the current frame's features by referring historical frames. With RRU, the quality of each frame's appearance representation is improved. Then we use the Spatial-Temporal clues Integration Module (STIM) to mine the spatial-temporal information from those upgraded features. Meanwhile, the multi-level training objective is used to enhance the capability of RRU and STIM. Through the cooperation of those modules, the spatial and temporal features mutually promote each other and the final spatial-temporal feature representation is more discriminative and robust. Extensive experiments are conducted on three challenging datasets, i.e., iLIDS-VID, PRID-2011 and MARS. The experimental results demonstrate that our approach outperforms existing state-of-the-art methods of video-based person re-identification on iLIDS-VID and MARS and achieves favorable results on PRID-2011.




§ INTRODUCTION

Person re-identification aims to identify persons across different cameras views.
Recently, this topic has drawn more and more attention thanks to its significant applications in video surveillance analysis and retrieval.
This task is very challenging due to background clutter, blur, occlusion, as well as the dramatic variation in illumination, pedestrian's postures and viewpoints.
Generally, person re-identification is approached with either image or video data for representation.
Many image-based person re-identification methods have achieved impressive progress.
However, those methods are susceptible to the quality of images.
Limited amount of information in a single image results in a lower tolerance to noise.
For similar pedestrians, if the discriminative patches are lost due to occlusion or blur, it will easily lead to misidentification.

Compared with a single image to represent a person, a video sequence contains richer information <cit.>.
The information of different video frames complements each other, so that it is more robust to noise.
More importantly, motion context in video sequences is useful for identifying pedestrians.
Therefore, how to make full use of spatial-temporal information is the key to video-based person re-identification.
Some methods <cit.> use recurrent networks to fuse temporal information.
Another alternative is to predict a quality score for whole or part of each video frame <cit.>.
However, a key effect of temporal information is ignored by these methods: the temporal information is useful to resist spatial noise.
Usually, in a video sequence, although a region in one frame is corrupted by noise, the missing information can be recovered by regions of the same location in some other frames.
Given a video frame, the difficulty lies in how to refine the noisy regions with the features of previous video frames.

In this paper, we propose a new approach to handle the difficulty of video-based person re-identification.
Instead of directly using the features of each video frame to extract temporal features,
we first propose a refining recurrent unit (RRU) to recover the missing parts
and suppress noisy parts based on the appearance and motion context from historical video frames.
After that, with the refined feature representation, we introduce a spatial-temporal clues integration module (STIM) to integrate spatial information and temporal information simultaneously.
Meanwhile, the proposed multi-level training objective further enhances the capability of RRU and STIM.
The cooperation of those modules enables the network to learn more robust and discriminative feature representation for accurate person re-identification.



§ RELATED WORKS

Most prior works on person re-identification  are based on still images and dedicated to two key issues: discriminative feature representation learning and distance metric learning.
Features that are both discriminative and invariant to background and viewpoint changes are crucial to person re-identification.
Handcrafted features such as color histograms <cit.>, texture histograms <cit.> and Local Binary Pattern <cit.> are widely utilized.
With feature representation, distance metric learning approaches are widely explored to accurately measure the similarity between pedestrians.
Following such a paradigm, many effective methods have been proposed, such as LADF <cit.>, RankSVM <cit.>, XQDA <cit.>.

In recent years, the rapidly developing Convolutional Neural Networks (CNN) have greatly advanced the progress of person re-identification.
Subramaniam et al. compute the normalized correlation between two patch matrices to handle inexact matching problems <cit.>.
Wang et al. propose an approach to learn single image representation and cross-image representation simultaneously <cit.>.
In <cit.>, a multi-scale and saliency-based model is proposed to learn deep discriminative feature representations at different scales.
Si et al. design a dual attention network to learn context-aware feature sequences and apply dually attentive comparison <cit.>.
Meanwhile, many works try to improve the representation capability of models based on local parts.
Wei et al. use a part extraction module to generate part regions and learn discriminative features based on different part regions <cit.>.
Sun et al. propose a refined part pooling to enhance the consistency in each part <cit.>.
Zhao et al. use a pretrained human landmark generation model to extract body regions
and then use a tree-structured fusion strategy to integrate the full-body features and different body subregion features <cit.>.

On the other hand, video-based person re-identification has also gained considerable attention.
Wang et al. propose a model that automatically selects more discriminative video fragments from the whole video sequence and learns cross-view matching by ranking <cit.>.
You et al. design a top-push distance learning model to enlarge the inter-class margin and minimize intra-class variations in order to improve the matching accuracy <cit.>.
Recently, some other methods <cit.> have been proposed to predict the quality scores for the features of video frames or local regions.
They average the frame or region features in a weighting manner based on the quality scores as the final representation of a video sequence.
Although their strategies alleviate the noise problem, the neglect of important temporal context limits their capability.
To make better use of the temporal information in video sequences,
RNN or LSTM <cit.> is adopted to fuse the feature vectors extracted from video frames <cit.>.
The temporal average pooling <cit.> is applied at each time step to generate the final representation of video sequences.
Although the temporal clues are captured with the recurrent models, the motion context of different regions in video sequences is lost, which results in limited performance.

Different from the above methods, our work aims to take full advantage of spatial-temporal information in video sequences. To this end, we propose two effective modules, i.e., refining recurrent unit (RRU) and spatial-temporal clues integration module (STIM). The former resists noise and recovers missing activation occurred in different regions while the latter integrates both spatial and temporal clues in a unified framework. Further, we apply a multi-level training objective to simultaneously optimize them. To the best of our knowledge, this is the first attempt to refine appearance representation with spatial-temporal information in person re-identification task which achieves promising performance.











§ METHOD

In this section, we first introduce the overall architecture of the proposed method.
Then we elaborate the key components in our framework, including refining recurrent unit (RRU), spatial-temporal clues integration module (STIM) and multi-level training objective, separately.



 §.§ Framework Overview

The overall architecture of the proposed model is shown in Fig. 1.
V_i = {V_i,k}_k=1^K represents K video sequences of person i.
Each video sequence contains T frames.
We use V_i, k, t to represent the t^th frame of V_i, k.

We adopt Inception-v3 <cit.> as the backbone of feature extraction module.
Given a video sequence V_i, k, each frame V_i, k, t is fed to Inception-v3 module
to extract frame-level feature maps X_i, k, t∈R^C × H × W from the outputs of the final inception block.
Feature maps of all frames in a video sequence are fed into RRU for refinement.
The refined feature maps are then fed into STIM to generate the final video-level feature representation
f_i,k∈R^256.



 §.§ Refining Recurrent Unit

In the involved video data for person re-identification, it is common that some regions of a target person suffer from occlusion, blur and varied postures.
In the degraded regions, the raw feature maps X_i, k, t are easily polluted.
However, the appearance and motion information remembered from regions of the same position in other frames can help to recover the lost information.
Inspired by such observation, we design a recurrent unit to remove noise and recover missing activation regions by implicitly referring the appearance and motion information extracted from previous video frames.

As illustrated in Fig. 2(a), for time step t, RRU takes three inputs: the raw feature X_i, k, t-1 and the refined feature S_i, k, t-1 of the last frame,
as well as the raw feature X_i, k, t of current frame.
In the first time step, X_i, k, 0 and S_i, k, 0 are initialized with X_i, k, 1.
The update gate model g contained in RRU is used to decide how to update S_i, k, t.

The refined appearance representation S_i, k, t-1 of last frame lends itself as a good reference to evaluate the quality of current frame.
So we use (X_i, k, t - S_i, k, t-1) as an input of g to model the appearance differences.
Between neighboring frames, pedestrian's movement causes translation of feature responses in spatial space.
With such consideration, we use (X_i, k, t- X_i, k, t-1) as another input of g to capture motion context.
These two terms are concatenated to form Z_i∈R^2C × H × W for g to determine the update gate of different regions.
The update gate Z is defined as follows,

    Z = g(Z_i) = g( [X_i, k, t- S_i, k, t-1, X_i, k, t - X_i, k, t-1]) .


Fig. 2(b) illustrates the detailed structure of update gate model g.
The first layer of the update gate model g named transition layer consists of a convolutional layer with 256 filters of size 1 × 1, a batch normalization (BN) layer and a rectified linear unit (ReLU).
It is designed to summarize the appearance and motion information of each spatial location and reduce the feature dimension.
Then a spatial attention model and a channel attention model are applied to the transitional feature Z_t∈R^256 × H × W produced by the first layer, separately.

For spatial attention model, we first use a global cross-channel average pooling layer to get the overall response in each spatial position.
Then two FC layers (the first 128-node FC layer is followed by ReLU) are applied to generate the spatial attention maps Z_s∈R^1 × H × W.

It is formulated as

    Z_s = W^2_s × ReLU(W^1_s ×Z^H,W_t)  ,

where Z^H,W_t ∈R^H, W is the result of Z_t after cross-channel average pooling and × is matrix multiplication.


For channel attention model,  we first introduce a global spatial space average pooling layer to get overall response of each channel.
Then a FC layer is applied to get the channel attention maps Z_c∈R^C × 1 × 1, which is formulated as

    Z_c = W_c ×Z^C_t  ,

where Z^C_t ∈R^C is the result of Z_t after global spatial space average pooling.


The overall attention maps of current input feature are the product of spatial attention maps and channel attention maps.
After a sigmoid operation, the overall attention maps are normalized into the range between 0 and 1, formulated as

    Z = sigmoid(Z_s⊙Z_c)  ,

where Z∈R^C × H × W is the update gate of X_i, k, t and ⊙ denotes element-wise multiplication.
Then RRU refines X_i, k, t with the previous refined feature S_i,k,t-1 as

    S_i, k, t = (1 - Z) ⊙S_i, k, t-1 + Z⊙X_i, k, t ,

where S_i, k, t∈R^C × H × W is the refined feature of the raw input X_i, k, t of the current video frame.
The value of each position in Z denotes the probability for the activation value in corresponding position of X_i, k, t to be reserved.
Higher probability value indicates that the update gate model g considers that feature in this location has high quality and should be reserved,
and on the contrary, locations with lower probability will be dominated by previous refined feature.
Given the refined feature S_i, k, t of each frame in a video sequence, we stack them to S_i, k∈R^C × T × H × W, where T is the number of frames in this video sequence.

Unlike the previous recurrent units (RNN or LSTM), RRU is used to upgrade the frame-level feature by referring the spatial-temporal information instead of extracting new features from temporal feature vectors.
From the above formulas, we can see that S_i, k, t and X_i, k, t share the same feature space, which means that RRU refines features in the same feature space.
This allows RRU to be applied to other video-based models to reduce spatial noise and improve quality of feature at each time step.
We will provide justification on this issue in the later experiments section.






 §.§ Spatial-temporal Clues Integration Module

Many previous works use RNN or weighted average pooling to integrate temporal information from feature vectors that are extracted from video frames.
In those methods, the motion context in different spatial locations is ignored <cit.>.
In order to fully exploit spatial-temporal information, we propose a spatial-temporal clues integration module (STIM) to
enable our model to learn the appearance representation and capture motion context simultaneously.

STIM contains two 3D convolution blocks and a global average pooling layer.
Each 3D convolution block is composed of three consecutive operations: a 3D convolutional layer <cit.> with 256 filters followed by a 3D BN layer and ReLU.
In the first 3D convolution block, the kernel size is set to be 1 × 1 × 1, so as to reduce the feature dimension.
The second 3D convolution block with 3 × 3 × 3 kernels outputs the spatial-temporal feature maps O_i,k∈R^256 × T × H × W.
After temporal and spatial space average pooling, we get the final feature representation f_i,k∈R^256 as


    f_i,k = 1/T1/H1/W∑_t=1^T∑_h=1^H∑_w=1^WO_i,k,t .


The 3 × 3 × 3 convolution kernels allows STIM to capture the movement of human body parts in spatial space.
Meanwhile, the simultaneous integration of spatial information and temporal information also helps STIM to resist local spatial noise. 




 §.§ Multi-level Training Objective

The identity classifier and auxiliary classifier of raw Inception-v3 model <cit.> are retained to learn more robust and informative features.
These two cross-entropy losses are denoted as L_c.
We replace the FC layer in identity classifier with a classifier block <cit.>,
which consists of a 512-node FC layer, BN, ReLU, Dropout, another FC layer and a Softmax layer.
We feed f_i,k to classifier block to generate the prediction of identity.

In addition to the cross-entropy losses, we propose a multi-level training objective to further enhance the capability of RRU and STIM,
which consists of the video-level ranking constraint L_v and part-level ranking constraint L_p.
Then, the overall training objective is defined as

    L = L_c + L_v + L_p  .


We apply batch hard triplet loss <cit.> on f_i,k as the video-level ranking constraint, formulated as

    L_v = 1/N1/K∑_i=1^N∑_a=1^K[m    + max_p=1 ⋯ KD(f_i,a, f_i,p) 
       - min_j=1 ⋯ N 
     n=1 ⋯ K 
     j ≠ iD(f_i,a,f_j,n))]_+  ,

where m is the margin and D(· , ·) denotes the Euclidean distance between two feature vectors.
N denotes the number of pedestrians in a mini-batch.
The video-level ranking constraint forces the distance between the overall representation of positive pairs to be smaller than negative pairs.

However, as the overall description of a video sequence, f_i,k ignores the spatial differences.
The most important function of RRU is to refine features of different regions.
So the video-level ranking constraint can not let RRU exert its full capability.
To further enhance the refining capability of RRU in local parts,
we propose the part-level ranking constraint based on the batch hard triplet loss.
We split the original feature maps S_i,k,t output by RRU into H horizontal strips and get the part-level feature representation as follows,

    p^r_i,k = 1/T1/W∑_t=1^T∑_w=1^WS^r_i,k,t ,

where S^r_i,k,t∈R^C × W is the strip feature from r^th row of S_i,k,t.
Given the part-level feature vector p^r_i, k∈R^C, we define the part-level ranking constraint as

    L_p = 1/N1/K1/H∑_i=1^N∑_a=1^K∑_r=1^H[m    + max_p=1 ⋯ KD(p^r_i,a, p^r_i,p) 
       - min_j=1 ⋯ N 
     n=1 ⋯ K 
     j ≠ iD(p^r_i,a,p^r_j,n))]_+  .




Unlike previous methods <cit.>, our part-based method does not require additional parameters, which makes our strategy more efficient and flexible.
For the same strip, part-level ranking constraint pulls features of video sequence belonging to the same identity closer and pushes features of video sequence belonging to different identities farther.
This forces RRU to learn a better update gate model g, which can measure the quality of different regions better.

In our approach, those modules are optimized in a collaborative way.
Given the raw low-quality feature maps of each frame, RRU first refines them based on spatial-temporal information.
Then STIM extracts better spatial-temporal feature representation from high-quality appearance feature maps generated by RRU.

The video-level ranking constraint forces STIM to learn more discriminative spatial-temporal representation for each video sequence.
The part-level ranking constraint helps RRU to focus on different local parts and enhance its refining capability for spatial space.
Through the cooperation between these modules, the final representation is more discriminative and robust to occlusion, blur, illumination and posture changes.








§ EXPERIMENTS

In this section, we evaluate our method on three public video datasets for person re-identification including iLIDS-VID <cit.>, PRID-2011 <cit.> and MARS <cit.>.
We first introduce the experiment setting in Sec. 4.1 and 4.2. Then, we make a ablation study on the effectiveness of each component of our method in Sec. 4.3. After that, in Sec. 4.4, we compare our approach with the state-of-the-art video-based person re-identification methods.


 §.§ Datasets and Protocols

iLIDS-VID consists of 600 video sequences of 300 persons. For each person, there are two video sequences observed from two non-overlapping cameras views at an airport arrival hall.
The video sequences range in length from 23 to 192 frames with an average length of 73.
The challenging factors on this dataset include blur, occlusion and large variations in pose, viewpoints and illumination.

PRID-2011 includes 400 video sequences of 200 identities captured by two camera views.
The length for each video sequence varies from 5 to 675 frames with an average length of 100.
Following <cit.>, video sequences with more than 21 frames are used.
Compared with iLIDS-VID, PRID-2011 is relatively less challenging because of relatively simple backgrounds and rare occlusions.

MARS is the largest video-based person re-identification dataset, which consists of 1261 different pedestrians and around 20,000 video sequences.
Those video sequences are generated by DPM detector and GMMCP tracker, which make MARS more realistic and challenging.

Following <cit.>, the partition for training and testing set in MARS dataset is given. The results are reported in terms of Cumulative Matching Characteristic (CMC) table
and mean average precision (mAP).
Following <cit.>, iLIDS-VID and PRID-2011 datasets are randomly split into two sets with the same number of pedestrians for training and testing.
For testing set, video sequences from one camera view are used as probe set, while video sequences from another camera view are used as gallery set.
We use the average CMC table over 10 trials with different train/test splits to evaluate the performance of different methods on these two datasets.



 §.§ Implementation Details

The Inception-v3 model <cit.> is first pretrained on the ImageNet dataset.


Each input video frame is resized to 299 × 299 pixels.
It is notable that we don't apply any data augmentation strategy.
During training process, we set N = 10, K = 2, T = 8 and m = 0.4.
The dropout rate in classifier block is set to 0.5.
Network is updated by stochastic gradient descent algorithm with a learning rate of 0.01, weight decay of 5 × 10^-4 and nesterov momentum of 0.9.
For the pretrained layers, the learning rate is set to 0.1 × of the base learning rate.
During test process, we extract the feature vector f_i,k using all frames of one video sequence V_i,k and compute the cosine distance with feature vectors of other video sequences.
The code will be made publicly available[https://github.com/yolomax/rru-reid].


 §.§ Ablation Study of the Proposed Method




The baseline approach contains only the Inception-v3 model trained by L_c.
We use average pooling to generate the final representation.
As shown in Table <ref>, after applying video-level ranking constraint L_v, the rank-1 accuracy of the baseline approach is improved by 14.5% on iLIDS-VID and 3.6% on PRID-2011.
This shows that it is very effective to introduce the video-level ranking constraint for the networks.

Analysis on STIM.
When we use a LSTM with a hidden size of 256 directly to integrate temporal information, the performances are even worse than average pooling approach on both datasets because of the noise in raw features and the neglect of spatial information.
If we replace LSTM with STIM, the rank-1 accuracy is improved by a large margin, which means that STIM can better integrate spatial-temporal information.
Meanwhile, it also indicates that it is necessary to preserve the spatial resolution when mining temporal information, which makes STIM more robust to noise than LSTM.



Analysis on RRU.
When we apply RRU to Baseline+L_v, the rank-1 accuracy is improved by 2.8% on iLIDS-VID and 3.3% on PRID-2011.
We show some examples of the feature maps before and after being refined by RRU in Fig. 3.
For the left video sequence fragment, due to the severe blur, the responses to the legs of the raw feature maps in the second frame and third frame are almost lost as shown in the white boxes.
After refinement by RRU, the responses are well recovered.
For the middle video sequence fragment, the model suffers from background clutter.
As shown in the gold box in the second frame, the background part has noisy responses.
After the process of RRU, the noisy responses are well removed.
Meanwhile, the drifting responses of upper body in raw feature maps of second frame and the nearly missing responses in the white box of the fourth frame are well corrected.
For the right video sequence fragment, the hand in the fourth frame is occluded by the sign.
After refinement by RRU, the responses are well recovered, and the almost lost responses in the first frame are also well recovered.
Those examples verify the denoising and recovering capabilities of RRU.

We evaluated the effects of update gate model g with different component setting. As shown in Table <ref>, the combination of Z_s and Z_c achieves better performances than using them alone.
We also investigate the effect of the input information for update gate model g, which is shown in Table <ref>.
When we remove the motion information [X_i, k, t- X_i, k, t-1] and only keep the appearance differences [X_i, k, t- S_i, k, t-1], the performance declines on both datasets.
This indicates the motion information is useful for g to evaluate the quality of feature maps.
When we directly use the concatenated feature maps [X_i, k, t, S_i, k, t-1] as input without explicitly indicating the appearance differences and motion information, the performance also drops.
This indicates that it is more efficient to provide the priori information about appearance differences and motion information than to let g directly learn how to integrate the information from the S_i, k, t-1 and X_i, k, t.

When we insert RRU before LSTM, the rank-1 accuracy is improved by 0.8% on iLIDS-VID and 2.9% on PRID-2011.
When RRU is inserted before STIM, RRU boosts the rank-1 accuracy by 3.2% on iLIDS-VID and 0.4% on PRID-2011.
Specifically, when collaborated with RRU, STIM achieves greater performance improvement than LSTM on challenging iLIDS-VID dataset, since STIM makes better utilization of the refined appearance representation generated by RRU, while LSTM ignores the spatial information.
After applying part-level ranking constraint on RRU, RRU further boosts the rank-1 accuracy of STIM by 3.8% on iLIDS-VID and 1.2% on PRID-2011.
The video sequences of MARS dataset <cit.> are generated by DPM detector and GMMCP tracker, which are not well-aligned and make it more challenging than other datasets. As shown in Table <ref>, RRU and L_p still boost the accuracy. RRU refines feature maps instead of raw RGB images. The large receptive fields of neurons in the backbone model mitigate the impact of imperfect pedestrian detection bounding box.
We can conclude that RRU is well compatible with other models.
Meanwhile, the combination of video-level ranking constraint and part-level ranking constraint is important for further improving the learning capability of networks.



 §.§ Comparison to the State-of-the-Art methods


We compare the performance of our approach with other state-of-the-art methods on three datasets in Table <ref>.



Our approach achieves the best performance on iLIDS-VID and MARS for rank-1 accuracy and mAP accuracy.
Compared with DRSTA, the improvements achieved by our approach are 4.1% and 2.1% for rank-1 accuracy on iLIDS-VID and MARS, respectively.
Specifically, for MARS, our approach exceeds DRSTA by 6.9% for mAP accuracy.
On PRID-2011 dataset, our approach is only slightly lower than DRSTA in terms of rank-1 accuracy.
As discussed before, PRID-2011 is relatively less challenging because of relatively simple background and rare occlusion, which makes the capability of our model fail to be fully expressed.






§ CONCLUSIONS

In this paper, we propose a new network architecture for video-based person re-identification,
which consists of two key modules, i.e., refining recurrent unit (RRU) and spatial-temporal clues integration module (STIM).
The former refines the feature activations with a recurrent paradigm, while the latter integrates the abundant spatial-temporal information.
Those modules are collaboratively optimized with a multi-level training objective.
Extensive experiments on three popular benchmark video datasets demonstrate the effectiveness of our approach.



§  ACKNOWLEDGMENTS

This work was supported in part to Prof. Houqiang Li by 973 Pro-gram under contract No. 2015CB351803 and NSFC under contract No. 61390514, and in part to Dr. Wengang Zhou by NSFC under Contract 61822208 and Contract 61632019, the Fundamental Research Funds for the Central Universities, the Young Elite Scientists Sponsorship Program by the CAST under Grant 2016QNRC001, and the Youth Innovation Promotion Association CAS under Grant 2018497.

aaai

