Person re-identification with fusion of hand-crafted and deep pose-based body region features
Jubin Johnson Shunsuke Yasugi Yoichi Sugino Sugiri Pranata Shengmei Shen






















Person re-identification with fusion of hand-crafted and deep pose-based body region features
    Jubin Johnson1 Shunsuke Yasugi2 Yoichi Sugino2 Sugiri Pranata1  Shengmei Shen1

	1Panasonic R&D Center           2Panasonic Corporation 
 Singapore Core Element Technology Development Center

	            Japan

	http://www.prdcsg.panasonic.com.sg/http://www.prdcsg.panasonic.com.sg/
	
    December 30, 2023
=================================================================================================================================================================================================================================================================================================





   Person re-identification (re-ID) aims to accurately retrieve a person from a large-scale database of images captured across multiple cameras. Existing works learn deep representations using a large training subset of unique persons. However, identifying unseen persons is critical for a good re-ID algorithm. Moreover, the misalignment between person crops to detection errors or pose variations leads to poor feature matching. In this work, we present a fusion of handcrafted features and deep feature representation learned using multiple body parts to complement the global body features that achieves high performance on unseen test images. Pose information is used to detect body regions that are passed through Convolutional Neural Networks (CNN) to guide feature learning. Finally, a metric learning step enables robust distance matching on a discriminative subspace. Experimental results on 4 popular re-ID benchmark datasets namely VIPer, DukeMTMC-reID, Market-1501 and CUHK03 show that the proposed method achieves state-of-the-art performance in image-based person re-identification.        





§ INTRODUCTION

Person re-ID is an important task in video surveillance and has various applications in security and law-enforcement. The main goal in re-ID is to retrieve all instances of a probe person from a large gallery set. In the instance of a security breach or event, law enforcement can use a re-ID system to automatically identify and track persons-of-interest, which would save hundreds of man-hours.     

Apart from the obvious challenges posed by viewpoint variations and occlusions, re-ID can be regarded as a zero-shot learning problem where the training and testing classes are different. It is therefore, crucial to learn discriminative representations for unseen persons. Existing approaches handle this challenge by either learning discriminative subspaces/metrics <cit.>, or generating discriminative features <cit.>. The success of deep learning in computer vision has led to emergence of works that learn discriminative features using CNNs <cit.> and through metric learning losses such as contrastive loss <cit.>, triplet loss <cit.>, and more recently quadruplet loss <cit.>. 


A majority of these approaches learn a global feature representation for the person crop, without considering the spatial information contained in the image. Previous works <cit.> have shown that the representations learned by a deep classification model on the global image are focused mainly on one body region- the upper body. The drawbacks of such global representations are 
 illustrated in Fig. <ref>. Since person crops for large-scale datasets like Market1501 and CUHK03 are generated from video frames using detectors such as DPM <cit.>, inaccurate detection boxes might impact feature learning, e.g., Fig. <ref> (a-d).  The top half of the images in Fig. <ref>(a) correspond to the head-shoulder region and background respectively. Directly comparing the feature maps learned from the global image would impede the similarity score. The pose change or non-rigid body deformation makes the metric learning difficult, e.g., Fig. <ref> (e-f). Moreover, occluded parts of the human body might introduce irrelevant context into the learned feature and it is non-trivial to emphasize local differences in a global feature, especially
when we have to distinguish two people with very similar appearances. To explicitly overcome these drawbacks, recent studies have paid attention to part-based, local feature learning. Some works <cit.> divide the whole body into a few fixed parts, without considering the alignment between parts. However, it still suffers from inaccurate detection box, pose variation, and occlusion. 

In the proposed re-ID framework, pose information is utilized to guide the feature learning process for various body parts. Pose has been recently used to guide local learning for better alignment of features <cit.>. Our work is inspired by <cit.> where multiple body regions are used to guide the feature learning followed by a tree-based feature fusion step to effectively combine the different local features. We show that such detailed level of granularity in the body parts is not always beneficial to the learning process. A simple concatenation of the features learned from three body regions - head , body and leg, along with the global features generalize well across datasets captured from different domains. Second, we adopt a disjoint framework which not only
utilizes the powerful feature extraction capability of CNN but also brings into
play the exclusive superiority of complementary handcrafted features and advanced metric learning method.       



§ RELATED WORK

We review work closely related to our method. An extensive survey of person re-ID can be found at <cit.>. Most ReID pipelines are composed of two main steps,
feature learning and metric learning. For feature learning, several effective approaches have been proposed,
for example, the Ensemble of Local Features (ELF) <cit.>, the Local Maximal Oc-
currence (LOMO) <cit.>. These handcrafted features have made impressive improve-
ments over robust feature representation, and advanced the person re-id research. The promising performance of CNN on ImageNet classification
indicates that classification network extracts discriminative
image features. Therefore, several works <cit.> fine-tuned the classification networks on target
datasets as feature extractors for person re-ID. For example,
Xiao  <cit.> propose a novel dropout strategy to train a
classification model with multiple datasets jointly. Wu  <cit.> combine the hand-crafted histogram features and
deep features to fine-tune the classification network.
Besides classification network, siamese network and triplet network are two other popular networks for person ReID.
The siamese network takes a pair of images as input, and
is trained to verify the similarity between those two images <cit.>. Ahmed  <cit.> and Zheng  <cit.> employ the siamese network to infer the description
and a corresponding similarity metric simultaneously.
Shi  <cit.> replace the Euclidean distance with Mahalanobis
distance in the siamese network. Varior  <cit.>
combine the LSTM and siamese network for person ReID.
Some other works <cit.> employ triplet networks to
learn the representation for person ReID. Recently, many works generate deep representation from
local parts <cit.>. Li  <cit.> employ
Spatial Transform Network (STN) for part localization,
and propose Multli-Scale Context-Aware Network to infer
representations on the generated local parts.  


Zhang  <cit.> align the local features with the global features in a mutual learning framework. Pose invariant
embedding (PIE) <cit.> aligns pedestrians to a standard pose to
reduce the impact of pose variation. A Global-Local Alignment
Descriptor (GLAD) <cit.> does not directly align
pedestrians, but rather detects key pose points and extracts
local features from corresponding regions.
<cit.> presents a deep mutual learning
strategy where an ensemble of students learn collaboratively
and teach each other throughout the training process.
DarkRank <cit.> introduces a new type of knowledge-cross
sample similarity for model compression and acceleration,
achieving state-of-the-art performance. These methods use
mutual learning in classification

Our work is inspired by the part-based approach introduced by Zhao  <cit.>. They firstly extract human body parts with fourteen
body joints, then fuse the features extracted on body
parts using a feature fusion network that is trained end-to-end. We argue that the feature fusion network and the micro-body regions actually hamper the performance in test images. Experimental evaluations are provided in section <ref> to validate this claim.



§ PROPOSED METHOD

The proposed re-ID framework involves two stages for learning the feature representation of an input person image. The overall framework is illustrated in Fig. <ref>. First, the three body regions, namely head, upper body and lower body are learned using a pose prediction network. The body regions are used as guidance to the network for learning region-specific deep features in the next step through a ROI pooling layer as illustrated in Fig. <ref>. The handcrafted features are learned and combined with the deep features in the next step. Finally, metric learning is used to generate a discriminative subspace for accurate comparison between the query and probe features.



 §.§ Body region prediction

Similar to Zhao  <cit.>, human body region information is extracted using a pose prediction network (PPN) which will be used to guide the feature learning process for each sub-region. A sequential framework inspired by the Convolutional Pose Machines (CPM) <cit.> generates body joint 
response maps in a coarse-to-fine manner. In each stage, a
convolutional network is used to extract image features and
then combine the response maps from the previous stage,
to produce increasingly refined estimations for body joint
locations.  Given an 
input image, the PPN predicts fourteen response maps F=ℝ^W^'× H^'× 14. The body joint locations are then obtained by max pooling each of the response maps, followed by grouping them to generate three rectangle region proposals
representing the head, the upper-body and lower-body regions, respectively. A few examples are shown in Fig. <ref>. More details can be found in <cit.> with the exception that we do not mine out micro body regions.



 §.§ Multi-part representation learning for re-ID

The body parts generated in the previous step guides the learning for the multi-part representation The general flowchart for training the network is shown in
Fig. <ref>. The network takes the person image together with the
3 body regions as input and computes one global feature
and three part features. Details of the network and structure are present in <cit.>. It essentially contains three inception modules and an ROI pooling stage for pooling out the body region features.   Each feature is 256 dimensional which is concatenated to obtain the  final feature vector. During testing this multi-part feature is extracted and used
to distinguish different persons.
 


 §.§ Fusion of handcrafted features and metric learning
 
  In addition to the deep features, low-level features like color texture and illumination are crucial factors for describing images of persons. In addition to HSV color histogram, the Scale Invariant Local Ternary
  Pattern (SILTP) descriptor is utilized for illumination invariant texture de-
  scription. In order to extract spatial information, sliding windows (subwindow,
  i.e., 10×10 size) are applied to describe local details of a person image. Adopting
  pyramid representation, within each subwindow, we extract two scales of SILTP
  histograms and a joint HSV histogram. To address viewpoint changes, we check
  all subwindows at the same horizontal location, and maximize the local occur-
  rence of each pattern (i.e., the same histogram bin) among those subwindows,
  it is called Local Maximal Occurrence (LOMO) features. For 128×48 image,
  all the computed local maximal occurrences are concatenated into the final de-
  scriptor with 26960 dimensions and it is called LOMO. Finally, a log transform
  is applied to suppress large bin values, and both HSV and SILTP features are
  normalized. For metric learning, we adopt the widely used Cross-view Quadratic Discriminant Analysis (XQDA) <cit.> to learn the subspace.



§ EXPERIMENTS




 §.§ Datasets

The proposed framework is evaluated on the most widely used datasets in the past year for image-based re-ID namely, VIPeR <cit.>, CUHK-03 <cit.>, Market-1501 <cit.> and DukeMTMC-reID <cit.>.

VIPeR is one of the earliest and widely used datasets for person re-ID. It contains 632 unique identities captured across two cameras, with each identity having one image per camera. The dataset is split randomly into equal halves for training and testing.

CUHK-03 consists of 14,097 cropped images from
1,467 identities. For each identity, images are captured from
two cameras and there are about 5 images for each view.
Two ways are used to produce the cropped images, i.e., human
annotation and detection by DPM <cit.>. Our evaluation is based on the combined dataset. We use the standard experimental setting
to select 1,367 identities for training, and the rest 100 for
testing.

Market-1501 contains 32,668 images from 1,501
identities, and each image is annotated with a bounding box
detected by DPM. Each identity is captured by at most six
cameras. We use the standard training, testing, and query
split provided by the authors in <cit.>. 

DukeMTMC-reID is a subset of the DukeMTMC <cit.> for image-based re-identification, in the format of the Market-1501 dataset. It contains 16,522 training images of 702 identities, 2,228 query images of the other 702 identities and 17,661 gallery images. We report mAP and Rank-1 precision on Market-1501 and DukeMTMC-reID datasets, while rank-1, -5, and -10 accuracies are reported on the other 2 datasets. Similar to <cit.>, we learn a single model using the training splits of all the four datasets. The statistics of the training and testing splits are shown in Table <ref>.







 §.§ Comparisons with other methods
 
We compare the performance of our proposed method on the standard evaluation protocols for the 4 datasets. The results are reported in Tables  <ref> - <ref>. In the tables, Ours represents our results using cosine similarity for matching, while Ours+LOMO+XQDA represents the full framework with the addition of handcrafted features and metric learning.

On Market-1501, the proposed method achieves state-of-the-art performance with Rank-1 accuracy of 92.09% as shown in Table <ref>. Our mAP of 79.1% is only slightly lower than 79.3% of AlignedReID <cit.>. The influence of  metric learning is not significant for datasets with large gallery sets, nevertheless, the inclusion does bring about a slight increase in performance. 

Table <ref> shows the performance on DukeMTMC-reID, obtaining accuracy of 80.36% and 64.8% on Rank-1 and mAP measures. ,respectively. Although some recent methods <cit.> perform better than the proposed method on this dataset, they have lower scores on the other three datasets. This shows that our method is able to generalize well across datasets captured from different domains.

VIPeR and CUHK-03 follow the single-shot evaluation strategy, where only one query and gallery image is selected for each identity. Hence, we use the CMC ranking evaluation metric to compare with other methods. The comparisons on CUHK-03 are summarized in Table <ref>. The proposed method obtains accuracy of 92.7%, 99.1% and 99.6% on Rank- 1, 5 and 10 measures which is the state-of-the-art. We even outperform methods such as  <cit.> which use multiple images of the same identity in the gallery. 

On VIPeR dataset, we achieve 63.6% rank-1 accuracy, which is around 7% more than the current state-of-the-art. As can be seen from Table <ref>, for small-scale datasets, a big factor for this improvement is the metric learning step.      




 


 §.§ Comparisons with feature fusion of SpindleNet
 
SpindleNet <cit.> uses a tree fusion structure to combine the features obtained from the multiple body regions and showed that their fusion strategy based on feature competition and fine-tuning obtained the best performance. In our experiments, we observed that the addition of finer (micro) body regions deteriorated the performance of the overall framework. Table <ref> shows the experimental results validating this on the Market-1501 dataset. We trained SpindleNet and our network on the training split of this dataset. The addition of the micro body regions is shown to bring down the performance of both the feature fusion network as well as the linear concatenation. It is to be noted that the global features perform better than the FFN of SpindleNet. The use of the macro body regions - head leg and body, utilize complementary information that cannot be well represented by the global features, improving the performance by 2% on both mAP and Rank-1. 





 §.§ Comparison of backbone CNN architecture

The base network for deep representation learning contains convolutional layers followed by Inception modules. Table <ref> shows the effect of using ResNet-50 architecture <cit.> for our task of multi-body representation. ResNet-50 is deeper and takes a larger input image size of 224×224 for learning the feature representations. However, this does not equate to a better representation for the unseen test images as the mAP measures are consistently better for both the datasets under consideration. 





 §.§ Qualitative comparison

Fig. <ref> and <ref> show the retrieval results obtained using queries from all the datasets. For Market-1501 and DukeMTMC-reID, the gallery contains multiple images of the query identity captured from a camera different to that of the query. As can be seen from Fig. <ref>, our method is able to capture most person belonging to the probe, despite a huge variation in appearance and pose. The single-shot setting followed by VIPeR and CUHK-03, where only a single instance of the query identity is present in the gallery is more difficult for person re-ID. Although the person is partially occluded (row 5 in Fig. <ref>) or the appearance changes drastically (last row in Fig. <ref>), our method is consistently able to retrieve the correct person.     



§ CONCLUSION
 
In this paper, we have shown that deep classification models that are trained on the global image tend to learn poor representations for unseen test images. It is shown that a fusion of handcrafted features and deep feature representation learned using multiple body parts to complement the global body features achieves high performance on such zero-shot learning problems. Experimental evaluations on four benchmark datasets in person re-ID show that the proposed method performs among the state-of-the-art. 



ieee



