{'fraction_non_alphanumeric': 0.030674404354251523, 'fraction_numerical': 0.01975273041848394, 'mean_word_length': 4.831021437578815, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
DeepSkeleton: Skeleton Map for 3D Human Pose Regression

Despite recent success on 2D human pose estimation, 3D human pose estimation still remains an open problem. A key challenge is the ill-posed depth ambiguity nature. This paper presents a novel intermediate feature representation named skeleton map for regression. It distills structural context from irrelavant properties of RGB image e.g. illumination and texture. It is simple, clean and can be easily generated via deconvolution network. For the first time, we show that training regression network from skeleton map alone is capable of meeting the performance of state-of-theart 3D human pose estimation works. We further exploit the power of multiple 3D hypothesis generation to obtain reasonbale 3D pose in consistent with 2D pose detection. The effectiveness of our approach is validated on challenging in-the-wild dataset MPII and indoor dataset Human3.6M.


Introduction
A prevalant family of human pose estimation works generally fall into two groups: 2D human pose estimation and 3D human pose estimation. Last year has witnessed the revolution of 2D human pose estimation, thanks to the development of heatmap-based network and the availability of deep residual network [16]. However, the research on 3D human pose estimation has been significantly lagging behind its counterpart.
Prior works on 3D human pose estimation can be roughly categorized into two families: regression methods and reconstruction methods. Regression methods directly learn a mapping function from input image to the target 3D joint locations [22,23,24,33,36,42,43,57,44,32]. As popularized in 3D human pose estimation, the performance is not as excellent as expected. Reconstruction methods typically follow a two-stage schema where in the first stage a 2D pose estimator is employed and then 3D pose is reconstructed aiming to minimize the reprojection error via optimization [1,4,37,13,59] or matching [18,7]. While optimization can generate only one 3D output, recent work of Jahangiri and Yuille [18] has highlighted the importance of having multiple hypotheses under the widely known problem of depth ambiguity.
A fundamental challenge limiting previous methods from attacking 3D human pose estimation is the insufficient training data. Most top-performing methods in 3D human pose estimation are restricted in laboratory environment where the appearance variation is far less than outdoor scene. On the other hand, there exists no accurate 3D ground truth for in-the-wild dataset to date. Fusing 2D and 3D data, dates back to at least [56], therefore has become an emerging trend [42,57]. This naturally brings us to a question, for accurate 3D human pose estimation, is mixing different data sources really indispensible?
In this work, we argue that combined training is not necessary by better exploiting the data that we already have. We put forth a novel expressive intermediate feature representation called skeleton map. For an example see Figure 1. The core insight is that human being can easily reason about 3D human pose given skeleton map by our prior knowledge about human kinematics, anatomy and anthropometrics, rather than image observation. This suggests that most of the image cues, e.g. lightning, texture, and human-object interaction are useless for 3D pose inference.
Skeleton map is simple, compact and effective. Unlike full human body part segmentation map which requires pixel-wise segmentation, it only models the connection be-tween adjacent joints in the human skeleton. It contains rich structural information and exhibits inherent occlusionaware property for regression.
Skeleton map is general for both indoor and outdoor scenario. Researchers previously argued that 2D and 3D data complements each other and mixing them is central to better shared feature learning. But with our pure and succinct skeleton map representation, we are able to achieve near state-of-the-art performance. We then take a leap forward to generate multiple hypotheses, sharing the same spirit of "wisdom of the crowd" [27]. Different scales of skeleton maps are conveniently generated by deconvolutional networks for subsequent regression. It can be intepretered as implicit data augmentation without data fusion or data synthesis. This desirable property further resolves the depth ambiguity.
To the best of our knowledge, this is the first work to • Perform regression directly from skeleton map alone.
• Generate multiple hypotheses without 3D library.
• Unite segmentation, pose regression and heatmap detection into a framework we call DeepSkeleton for 3D human pose estimation.

Related Work
For a comprehensive review on the literature of 2D human pose estimation we refer the readers to [15,30,41]. Here we survey on previous works most relevant to our approach.
3D Human Pose Estimation Li and Chan [22] pioneered the work of direct 3D human pose regression from 2D image. Several approaches have been proposed to learn the pose structure from data. Tekin et al. [43] learn latent pose representation using an auto-encoder. Zhou et al. [58] integrate a generative forward kinematics layer into the network to learn the joint angle parameters. Realizing the difficulty to minimize per-joint error, [36,32,42] advocate to predict relative joint locations to multiple joints. More recently, Rogez et al. [38] simplify the problem by local residual regression in the classified pose class. Another research direction has been focused on inferring 3D pose from 2D estimates. The underlying premise is that 2D human pose estimation can be regarded as nearly solved [49,34,6,5,10,11,14,27,55]. Consequently, the challenge of 3D pose estimation has been shifted from predicting accurate 2D towards predicting depth from RGB image. As an example, Moreno-Noguer [33] employs FCN [29] to obtain pairwise 3D distance matrix from 2D observation for absolute 3D pose recovery. Tekin et al. [44] fuse image space and heatmap space to combine the best of both worlds. Tome et al. [46] adopt an iterative multi-stage architecture where 2D confidence is progressively lifted to 3D and projected back to 2D, ensuring the match between 2D observation and 3D prediction. Zhou et al. [60] combine heatmap and 3D geometric prior in EM algorithm to reconstruct 3D skeleton from 2D joints. Zhou et al. [57] output depth from 2D heatmap and intermediate feature maps in a weakly-supervised fashion. Nevertheless, the inherent depth ambiguity presents the foremost obstacle impeding the progress of 3D human pose estimation. Recent techniques ameliorate this issue through dual-source training. Examples include [56,57,42]. Our proposed DeepSkeleton stands in contrast to the latent assumption of these works that shared feature learning from both 2D and 3D data is essential. Our results suggest that 2D and 3D data sources may be completely indepedent, which is in line with recent observation [46,31].
Exemplar based pose estimation Most previous work on 3D human pose estimation rely on generating only one single 3D pose. This is problematic as multiple 3D poses may have similar 2D projection. With early reference to [9], generating multiple hypotheses has been repurposed for 3D human pose estimation in [18]. In their work, a generative model is responsible for generating multiple 3D poses from 3D MoCap library. Similarly, Chen and Ramanan [7] retrieve 3D pose using neareast neighbour search. DeepSkeleton shares some resemblence to these works expect for the need of large-scale offline MoCap dataset.
Semantic Segmentation Fully convolutional network(FCN) [29] made the first attempt to solve semantic segmentation using deep neural network. More recent works enjoy the benefits of residual connection [28,8]. However, accurate pixel-wise annotation of segmentation map is time-intensive. Different from full human body part segmentation [35,51,26] , synthesizing ground truth of our proposed skeleton map only requires 2D annotation.
Joint Semantic Segmentation and Pose Estimation Segmentation and pose estimation are two intrinsically complementary tasks in the sense that pose esmation benefits from the topology of semantic parts, while the estimated pose skeleton offers natural cue for the alignment with part instance. One of the most significant works that jointly solve these two problems is Shotton et al. [40], which derives an intermediate body part segmentation representation for 3D joint prediction from depth image. A large body of literature has been devoted to this field [12,21,54,20,2]. Tripathi et al. [48] demonstrate the impact of human pose prior on person instance segmentation. Perhaps the most related approach to ours is Xia et al. [52]. Their skeleton label map serves as a prior to regularize the segmentation task, and is derived from 2D pose prediction. In contrast, our skeleton map is independent of 2D detection and is directly taken as input to the subsequent regression module. Another major difference is that they aim to solve multiperson 2D pose estimation, while we target at single-person 3D pose estimation.

Methodology
Given a RGB image I ∈ R 224×224 of a person, our goal is to output 3D joint locations X ∈ R K×3 with joint number K = 16. We break the problem down into three steps: • Segmentation(Section 3.2): • Regression(Section 3.3): • Matching(Section 3.4): To match with 2D observation x ∈ R K×2 , the hypothesis X * ∈ H with minimum projection error to 2D joint detection is selected as final output.

Skeleton Map
Skeleton map S f i (S g i ) draws a stick with width l i between neighbouring joints in the human skeleton and assigns different colors to distinguish body parts, an example of which is depicted in Figure 1. The human skeleton we use follows the paradigm of [42] expect that we define thorax as root in all our experiments, in which 15 body parts are defined. Skeleton map, like body part segmentation map, encodes part relationship in the body segments and imposes strong prior on human pose. However, training networks for full human body semantic segmentation needs labor-intensive dense and precise annotations, which is impractical for large human pose dataset e.g. Human3.6M [17]. The simplicity of skeleton map naturally addresses this issue. But what is a good architecture for skeleton map generation?

Deconvolution for Generating Skeleton Map
Deconvolutional Network Design A simple choice is to employ the encoder-decoder architecture. In practice, we apply ResNet-50 [16] in a fully convolutional manner, producing pixel-level real values. We replace the fully connected layer after pool5 with deconvolutional layers. The network structure shown in Figure 2 starts with a 224×224 image and extracts features along the downsampling process. Herein only res2c, res3d, res4f and res5c are sketched for brevity. The last fully connected layer is simply discarded. The devolutional module built upon pool5 gradually processes feature maps to the final output: a threechannel 56×56 skeleton map. It comprises of repeated blocks of upsampling layer (with initial weights set to bilinear upsampling) followed by residual module. To encourage the learning of local and global context, high-level feature maps are combined with low-level feature maps through skip connections, analogous to those used in deep networks [29,34]. The output composes of three channels representing the skeleton map. Rather than performing perpixel classification into body part classes, we found that perpixel regression results in better segmentation accuracy. We opt for sigmoid cross entropy loss in training. However, one common problem in training deep network is the notorious vanishing gradient. To remedy this issue, each blob feeding into the pixel-wise summation layer branches off and connects to a residual block. Intermediate supervision is then applied on the output of each residual block, allowing for learning skeleton map at multiple resolutions.
After the initial performance gain brought by common pratice i.e. skip connection and intermediate supervision, investigations on prevalant conv-deconv network architectures e.g. RefineNet [28] and Stacked Hourglass [34] did little to further improve the segmentation accuracy.
Deal with trunction Truncation, that is, partial visibility of human body joints caused by image boundary, poses a noticeable challenge especially for in-the-wild 3D human pose estimation. In our case, this is supported by the fact that the deconvolutional network is uncertain about whether to plot the segments associated with cropped endpoint joint i.e. wrist or ankle, due to lack of image evidence. A standard way to deal with this is to use multiple image crop, which is made possible by multiplying the provided rough person scale in the dataset with the rescaling factor i.e. c i ∈ {1.0, 1.25, 1.5}. The 2D joint ground truth is rescaled in the cropped window accordingly. Note that indoor dataset faces no truncation problem, and crop scale c i is always set to 1.0.
Deal with stick width Several endeavours have been made to effectively excavate features for human pose estimation, most of which focus on multi-stream learning [53]. DeepSkeleton differs in that it directly modifies the target output by changing stick width. Our design of multi-scale skeleton map is motivated by the claim that each convolutional layer is responsible for skeleton pixel whose scale is less than the receptive field [39]. In concrete terms, only convolutional layers with receptive field size larger than stick width can capture features of body parts. Hence, coarse segmentation(large stick width) and fine segmentation(small stick width) feature varying combinations of lowlevel and high-level features. Note that care has to be taken when deciding the body segment width of ground truth skeleton map. We consider two practial concerns:(1)the parts should be small enough to localize semantic body parts.
(2)the parts should be large enough for convolution and deconvolution. The stick width i.e. l i is empirically set to be in the range [5,15].
Deal with occlusion Severe occlusion hinders accurate human pose estimation. Answers to simple queries such as "Is the left upper leg in front of right upper leg?" can actually provide important occlusion cues for regression. Motivated by inside/outside score map [25], in this work, foreground skeleton map S f i displays body parts that are occluding others, while background skeleton map S b i displays those occluded by others. That said, skeleton map explicitly models the occlusion relationship of two overlapping body parts. As far as we know, this straightforward formulation has never been done in the literature of human pose estimation. In more detail, recall that each 2D endpoint joint on the bone results in a ray oriented towards the camera optical center.
Denote the point with smaller depth(closer to camera in Z direction) as id, and the other as id. Foreground skeleton map assigns color of bone B id to the pixel {x, y}. In contrast, background skeleton map assigns color of bone B id , pretending bone B id is occluding bone B id . See Figure 3 for an example. This inherent occlusion-aware property of skeleton map is important for regression.

Regression
Plagued by the cluttered background of RGB image, a longstanding research direction in 3D human pose estimation has been exploiting better features from raw RGB input. We show that regression from skeleton map alone is feasible. We employ state-of-the-art ResNet-50 [16] as the backbone network. Since skeleton maps (S b i , S f i ) generated by deconvolutional network Deconv i are 56 × 56, they are rescaled to 224 × 224 first and concatenated together, which is then taken as input to Deconv i and processed along the downsampling path. The last fully connected layer is repurposed to output 3D position X i = X i k |k = 1, 2, ..K of all K joints. Euclidean distance loss is applied for back propogation. Multiple 3D predictions H = X i |i = 1, ...n are made by training independent regression networks for different skeleton maps input. We want to emphasize that naïvely concatenating all the skeleton maps S as input (early fusion) or learning parallel streams for multiple input and fusing the feature responses subsequently (late fusion) will not help. As an alternative option, one might consider to concatenate skeleton map along with raw RGB image, which however, does not boost the performance in our observation. Therefore, we stick with the original design i.e. learning 3D solely from intermediate skeleton map feature representation.

Matching
Now we have multiple 3D pose hypotheses H, the problem boils down to select the optimal hypothesis X * ∈ H as final 3D output. The simpliest way is to choose the candidate whose projection best matches the 2D pose detection results. Writing Proj as the camera projection matrix and x as 2D pose detection, we seek to find the optimal 3D pose X * ∈ H via minimizing the reprojection error: We use the pre-trained state-of-the-art 2D detector Stacked Hourglass [34] for generating x. No finetuning is employed. We remark that our 3D hypotheses are completely independent of 2D pose detection, rather they are learnt from multi-level discriminative skeleton maps.

Discussions
In principle, any expressive intermediate representation can be used for regression. Heatmap, for example, has been explorered in [50] to bridge the gap between real and synthetic dataset. Carreira et al. [6] stack heatmap with RGB image for coarse-to-fine regression. Yet, an obvious drawback of heatmap is different body joints are encoded in discrete gaussian peaks, thus the dependence between joints is not well exploited. Skeleton map overcomes this problem by explicitly connecting adjacent joints of each bone by a stick. The colored semantic body part offers a strong cue for regression learning. For generating skeleton map, perhaps the easiest way is to firstly detect 2D joints and then draw lines between neighbouring joints. However, this introduces two disadvantages: 1. Occlusion relationship information is completely discarded. 2. The inaccurate 2D detection has impact on the following regression. Our initial exploration shows that this has no apparent benefits.

Experiments
Our approach is evaluated on the largest human pose benchmarks MPII and Human3.6M. MPII [3] is a 2D realworld human pose dataset. It contains around 25k natural images collected from YouTube with a variety of poses and complicated image appearances. Cluttered background, multiple people, severe truncation and occlusion make it the most challenging 2D human pose dataset. For 2D pose evaluation in MPII, we use PCKh [3] metric which measures the percentage of joints with distance to ground truth below a certain threshold.
Human3.6M [17] is a large-scale 3D human pose dataset consisting of 3.6M video frames captured in controlled laboratory environment. 5 male and 6 female actors performing 17 daily activities are captured by motion capture system in 4 different camera views. The image appearance in clothes and background is limited compared to MPII. Following standard practice in [57,22,58,60], five subjects(S1, S5, S6, S7, S8) are used in training. Every 64 th frame of the two subjects(S9, S11) is used in testing. MPJPE(mean per joint position error) [17] is used as evaluation metric after aligning 3D poses to root joint. We represent 3D pose in local camera coordinate system following the methodology of Zhou et al. [57].

Implementation Detail
For training the network we use Caffe [19] with 15 GPUs. Deconvolutional network training starts with base learning rate 0.00001 and mini-batch size 12. For training regression network, we set base learning rate to 0.01 and mini-batch size to 32. Learning rate is dropped by a factor of 10 after error plateu on the validation set. The network is trained until convergence. For optimization, stochastic gradient descent is adopted. Weight decay is 0.0002, and momentum is 0.9. No data augmentation or data fusion is used.

Baseline Settings
To validate the effectiveness of skeleton map and multiple hypothesis, we test two baselines: • Direct RGB It performs regression directly from raw RGB input.

• Ours w/o Mul-Hyp
It performs regression from only one skeleton map.
Unless otherwise specified, we set c i = 1.0, l i = 10 for all the experiments of one hypothesis(skeleton map) both on Human3.6M and MPII.
Our final system is denoted as Ours w Mul-Hyp (equivalent to DeepSkeleton).

Experiments on 3D dataset Human3.6M
Comparison with state-of-the-art For fair comparison, we compare with state-of-the-art methods without mixed 2D and 3D data for training in Table 1. Note that Compositional Pose Regression [42] provides results with and without 2D data. We therefore denote CompBone * as Compositional Pose Regression without extra 2D training data and report the results from the original paper. Table 1 shows that our final system ours w Mul-Hyp outperforms the main competitor Tome et al. [46]. Notably, it surpasses competeting methods in actions Sit, SitDown, Photo by a large margin. The improvement comes from our novel skeleton map representation and the expressiveness of multiple hypotheses. Visualized 3D poses are displayed in Figure 5.
Comparison with Regression from RGB Table 2 shows that Ours w/o Mul-Hyp significantly improves baseline Direct RGB by 12.3mm(relative 10.8%), demonstrating the strength of skeleton map.
Does skeleton map brings better 2D estimation, or better depth estimation? In order to answer this question, we evaluate the average joint error given ground truth depth and ground truth 2D respectively in the following. Without loss of generality, we restrict ourselves to generate one hypothesis.
Impact of Skeleton Map on 2D Estimation We make use of ground truth depth and predicted 2D to recover 3D pose in the camera coordinate system. Table 3 reports the result of average joint error using different input sources for regression network. One can see that 25.9mm(relative 27.9%) error reduction is obtained after feeding predicted skeleton map to regression network. Further 17.4mm decrease is achieved by using ground truth skeleton map for regression. This can be interpreted as skeleton map simplifies the 2D learning procedure and prevents overfitting. Strong shape prior serves as important regularization cue for learning 2D location.
Impact of Skeleton Map on Depth Estimation To gain insight into the importance of skeleton map for depth estimation, we use ground truth 2D and predicted depth to acquire 3D joints. We see in Table 4 that depth regression from predicted skeleton map shows evident superiority over RGB image, yielding 21.5mm(relative 22.7%) error reduction. This indicates that skeleton map is more favorable for depth prediction.
Impact of Multiple Hypotheses Next we elaborate on the effect of using multiple hypotheses. Here we use k = 11 hypotheses. 1 We first assume that the ground truth skeleton map is provided. In Table 5, multiple hypotheses slightly improves the accuracy, but to a lower extent than expected. This implies that ground truth skeleton map is sufficiently powerful to reduce ambiguity. We then move to a realistic scenario where ground truth skeleton map is unavailable. Quite surprisingly, using multiple hypotheses reduces the average MPJPE from 101.9 mm to 86.5mm in Table 2, which largely narrows down the performance gap between ground truth and predicted skeleton map. Generated multiple hypotheses are illustrated in Fig 4. The third hypothesis is chosen as final output based on simple matching. One could argue that similar performance might be accomplished by ensembling multiple runs of the same regression network Regression i . To examine this, we take the regres- 6, 7, 8, 9, 10, 11, 12, 13, 14, 15}  sion outputs of k = 11 different runs from single skeleton map, denoted as Ensemble. The result in Table 6 suggests that our multi-level skeleton map provides more information than single skeleton map. A natural problem arises: What is the performance upper bound of multiple hypotheses? To investigate this problem, we select the optimal 3D hypothesis with minimum 3D error to ground truth 3D pose, producing an error of 68.3mm. This is promising as we are able to excel most state-of-the-art works without offline 3D pose library. However, how to select the optimal 3D hypothesis remains unclear.

Experiments on 2D dataset MPII
We present 2D and 3D pose estimation for in-the-wild dataset MPII. We use MPII validation set [47] including 2958 images for ablation study.
Pseudo 3D Ground Truth MPII only provides 2D annotation, but training our network requires 3D pose ground truth. We use state-of-the-art 3D reconstruction approach [59] to initialize 3D pose from 2D landmark. Note that most of the reconstructed poses are already reasonable despite occasional incorrect inference. We then introduce human assistance, where a human expert is presented with the initialized 3D pose along with input image and asked to manually adjust wrong limb orientation. We stress that the goal of semi-automatic annotation is to resolve the depth ambiguity as far as possible by aligning 3D pose with image observation. Since accurate 3D MoCap pose is impratical for natural images, we call this pseudo 3D ground truth.
Comparison with state-of-the-art Previous approaches generally fall into three families: 2D detection based, 2D regression based and 3D regression based. Our method belongs to 3D regression based. In this family, our closest competitors are [38,42]. The comparison is not completely fair as they both use additional 3D data. Sun et al. [42] integrate a two-stage state-of-the-art 2D regression based method IEF [6]     sion without ad-hoc stage. We observe in Table 7 ours w Mul-Hyp is on par with state-of-the-art 3D regression methods without 3D data or post processing. Qualitative results are shown in Figure 5.  Table 6. Performance gain from multiple hypotheses given predicted skeleton map on Human3.6M. MPJPE metric is used.

Comparison with Regression from RGB
Impact of Multiple Hypotheses Table 9 shows the effect of multi-scale and multi-crop skeleton map. We observe the same conclusion as in Table 6. Using multi-scale skeleton map results in 9.4%(relative 24.2%) improvement of ankle PCKh@0.5. Multi-crop skeleton map yields extra 7.0% improvement. It is noteworthy that ensemble of the same regression network falls behind our final system, indicating multi-level skeleton map is able to capture diverse semantic features from input image.
Performance Upperbound One remaining question is what is the limit of skeleton map applied in natural unconstrained scenario? To assess the upper bound, we perform regression from one single ground truth skeleton map on MPII. We see in Table 10    truth skeleton map achieves 94.5% overall PCKh@0.5. This validates the effectiveness of skeleton map representation.

Conclusion
We have sucessfully shown how to push the limit of 3D human pose estimation using skeleton map without fusing different data sources. Skeleton map is an impressive abstraction of input, which when combined with multiple hypotheses generation is able to achieve compelling results on both indoor and in-the-wild dataset. We also carry out exhaustive experimental evaluation to understand the performance upper bound of our novel intermediate representation. We expect to further narrow down the performance gap between ground truth and predicted skeleton map by better segmentation network. We hope the idea of combining semantic segmentation and pose estimation inspire a new research direction in 3D human pose estimation.