{'fraction_non_alphanumeric': 0.03552716572166798, 'fraction_numerical': 0.020634464939352615, 'mean_word_length': 4.468406593406593, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
PoseNet3D: Unsupervised 3D Human Shape and Pose Estimation

Recovering 3D human pose from 2D joints is a highly unconstrained problem. We propose a novel neural network framework, PoseNet3D, that takes 2D joints as input and outputs 3D skeletons and SMPL body model parameters. By casting our learning approach in a student-teacher framework, we avoid using any 3D data such as paired/unpaired 3D data, motion capture sequences, depth images or multi-view images during training. We first train a teacher network that outputs 3D skeletons, using only 2D poses for training. The teacher network distills its knowledge to a student network that predicts 3D pose in SMPL representation. Finally, both the teacher and the student networks are jointly fine-tuned in an end-to-end manner using temporal, self-consistency and adversarial losses, improving the accuracy of each individual network. Results on Human3.6M dataset for 3D human pose estimation demonstrate that our approach reduces the 3D joint prediction error by 18\% compared to previous unsupervised methods. Qualitative results on in-the-wild datasets show that the recovered 3D poses and meshes are natural, realistic, and flow smoothly over consecutive frames.


Introduction
Accurately estimating 3D pose from 2D landmarks is a classical ill-posed problem in computer vision [15,21]. Previous methods for 3D pose prediction can be classified into model-free and model-based approaches. Typically, modelfree approaches directly learn a mapping from 2D landmarks to 3D joints [6,9,39,40]. Predicting 3D joints from 2D landmarks (lifting) is a challenging task, since there exists an infinite number of 3D poses corresponding to a 2D skeleton due to projective ambiguity [6]. Model-based approaches often fit 3D parametric models such as SMPL [37] to estimate 3D shape and pose. This is typically done by minimizing the 2D error between the projection of the predicted 3D pose and the given 2D landmarks. However, as Equal Contribution shown in [27], 2D reprojection error alone is highly underconstrained and can be minimized via non-natural joint angles. To address these issues, previous 2D to 3D approaches have used various kinds of additional 3D supervision, including paired 2D-3D correspondences [39], unpaired 3D data [27] and synthetic data generated using motion capture (MoCap) sequences [51].
In this paper, our goal is to train a neural network that takes 2D pose (landmarks) as input and outputs SMPL parameters and 3D skeletons, without requiring any additional 3D data or iterative fitting during training. We first train a lifting network (aka teacher) using only 2D inputs to predict model-free 3D poses. The 3D pose output from the teacher is then used as pseudo ground-truth to train a student network to predict SMPL pose parameters. Thus, inspired by the knowledge distillation (KD) framework, our teacherstudent formulation allows training the network in the absence of additional 3D data. In fact, we show that training the student network by directly minimizing the 2D reprojection error (without using knowledge from the teacher) fails due to inherent ambiguities in 2D projection, resulting in incorrect depth predictions and unnatural poses.
Optimization based approaches such as [2] can also fit SMPL meshes to 2D poses. However such approaches are computationally expensive, potentially prone to error and require good initialization as well as 3D pose priors built using MoCap sequences. In contrast, our teacher-student formulation allows training a feed-forward network that can directly output SMPL pose for a given 2D input.
Typically, KD approaches train the teacher network in a supervised manner and the student network is often used to reduce run-time/complexity. Our approach builds upon [6] and trains the teacher (lifter) network in an unsupervised manner without using multi-view images [53,56], depth images [54], or 3D supervision. Compared to previous works, we show improvement by utilizing dilated convolutions to model temporal dynamics.
When using a parametric model such as SMPL, there often exists a semantic gap between the SMPL 3D joints and the 2D landmarks obtained from RGB images (e.g. using OpenPose [4] Figure 1: Overview of the proposed PoseNet3D approach. Input 2D poses are fed to a temporal backbone, followed by a teacher branch and a student branch, which outputs model-free 3D poses and SMPL parameters respectively.
SMPL are close to the center of the pelvis, while in the Hu-man3.6M [23] dataset, the 2D hip joints are close to the body surface. In previous works, this semantic adaptation is learned offline by fitting SMPL meshes to specific 3D datasets and is used during evaluation. Thus, 3D data is also required implicitly for bridging the aforementioned semantic gap. In contrast, we demonstrate that the semantic adaptation can be automatically learned during training to bridge the gap between the SMPL 3D joints and the 2D landmarks during training. Bridging this semantic gap, which is often ignored in previous works is crucial, otherwise the network can minimize joint error by twisting the body, resulting in unnatural poses.
We evaluate our approach on 3D human pose estimation tasks on Human3.6M and MPI-INF-3DHP datasets, reducing the mean per joint position error by 18% compared to the state-of-the-art unsupervised method of [6] (47mm vs 58mm) as shown in Sect. 4. Qualitative results confirm that our method is able to recover complex 3D pose articulations on previously unseen in-the-wild images (e.g. Fig. 4).
Approaches such as [6,56,63] have primarily used 2D joints from single/multi-view images without explicit 3D supervision to learn 3D pose. Chen et al. [6] proposed an unsupervised algorithm for lifting 2D poses to 3D. Their key intuition is that random 2D projections of accurately estimated 3D pose should conform to the real 2D pose distribution. A 2D discriminator is used to give feedback to the lifter about the naturalness of random 2D projections of the predicted 3D skeleton. This helps training the lifter without requiring direct 3D supervision.
Our teacher network builds upon the work of Chen et al. [6], but differs in the following respects. Firstly, in [6], inference uses a single frame as input and weak temporal consistency is enforced using an additional discriminator on frame differences. Their architecture only employs fully connected layers. In contrast, we use dilated convolutions (similar to [52]) to model temporal dynamics in the lifter as well as in the discriminator and train/test on multiframe inputs. Video based approaches such as Li et al. [35] employ 3D trajectory optimization via low rank property and temporal smoothness of video sequences. Instead, we propose novel loss terms to account for the consistency of predicted skeletons on common frames across neighboring time-steps and show improvement in accuracy on the Hu-man3.6M dataset over [6]. Secondly, unlike [6], our ap-proach is also able to estimate SMPL parameters. Deep Learning with SMPL: Deep learning approaches such as [1,27,31,38,45,49,51,59,61] have utilized SMPL to directly regress to the underlying shape and pose parameters by training a feed-forward network. The 3D joints are computed via linear regression on the estimated mesh vertices [27]. Our student network also predicts SMPL parameters but differs from these approaches in following respects.
Firstly, approaches such as [27] minimize the 2D reprojection error between the projection of the SMPL 3D joints and the predicted 2D joints from images. However, as noted in [27], 2D keypoint loss is highly unconstrained and thus [27] learns the limits of joint angles using a dataset of 3D scans. This is done by training a discriminator to bring the distribution of predicted SMPL parameters close to the real distribution of SMPL parameters obtained from 3D scans. Since we do not assume access to any additional 3D data at training time, we address this problem by first training a teacher network to predict 3D joint positions. We then use the output of the teacher as pseudo ground-truth to train the student network to predict SMPL parameters. By using knowledge distillation from the teacher along with simple regularizers on the SMPL parameters, we can recover realistic 3D pose without requiring additional 3D information during training. Our ablation studies show that the proposed strategy significantly outperforms the baseline strategy of directly minimizing the 2D re-projection error.
Secondly, previous works typically ignore the semantic gap between the SMPL 3D joints and the 2D landmarks while training. Instead, a regressor from vertices to joints [1,12,27] is obtained offline by fitting SMPL meshes to specific 3D datasets (e.g. Human3.6M). In contrast, we bridge this gap using semantic joint adaptation (SJA) during training of the student network. We demonstrate that SJA improves the accuracy as well as naturalness of the predicted 3D pose. SMPL based Optimization: Classical optimization techniques has also been used previously to fit the SMPL model to image evidence given by 2D landmarks/silhouettes [2,17,33,41]. The optimization based approaches are typically slow and prone to error. In contrast, our approach trains a feed-forward network allowing for a faster and more robust inference. Recently, Kolotouros et al. [31] (SPIN) have used iterative optimization within the training loop of a deep neural network to generate pseudo groundtruth, which is used to provide direct supervision on SMPL parameters for regression. Instead, our teacher network provides supervision on 3D joints, obtained by SMPL forward kinematics. In contrast to SPIN, we do not use a 3D pose prior learned using CMU MoCap sequences (similar to [2]). Our approach can also be extended to use optimization in the loop to generate additional supervision on SMPL pa-rameters, which we leave for future work. Knowledge Distillation: Distilling the knowledge in neural networks has been used for applications such as network compression, combining an ensemble of models into a single model [20], enhancing privacy [46], large scale semisupervised learning [68] etc. Wang et al. [63] use non-rigid structure from motion (NRSfM) alogrithm as the teacher to learn 3D pose from 2D joints. In contrast, we use a 2D-3D lifter as a teacher and show better results on Human3.6M. Unlike [63], our approach also outputs SMPL parameters.

Proposed Approach
Our PoseNet3D approach is a combination of model-free 3D pose estimation followed by knowledge distillation to predict SMPL pose parameters. As shown in Fig. 1, the input to our network is a set of T 2D skeletons from T consecutive frames of a video. The architecture consists of a temporal backbone, which utilizes dilated convolutions over time to model the temporal dynamics and produces a feature vector. The feature vector is fed to two branches: (a) Teacher branch, which outputs 3D poses, and (b) Student branch, which outputs SMPL parameters. The 3D joints from the student branch are computed as described in Section 3.3. The two sets of 3D joints from the student and teacher branches are compared to ensure consistency. The predicted 3D joints from the teacher and the student branches are re-projected to 2D after random rotations and are fed to a temporal discriminator. In the following sections, we describe the teacher and student networks and associated training losses in detail.

Teacher: Temporally Consistent Lifting
. . , N denote the i th 2D pose landmark of a skeleton in frame j with the root joint (midpoint between the hip joints) as origin. The 2D skeleton for frame j is x j = x j 1 , . . . , x j N . The input to the network at time step t is a set of T 2D skeletons represented as x(t) = x t , . . . , x T −t+1 . For simplicity, we drop the dependence on time-step to describe the lifter. Similar to [6], we assume a perspective camera with unit focal length centered at the origin and fix the distance of the 3D skeleton to the camera to a constant c units. The 2D skeletons are normalized such that the mean distance from the head joint to the root joint is 1 c units in 2D. At each time-step t, the temporal lifting branch predicts a depth offset o j i for each x j i . The 3D joints are computed as The generated skeletons are projected back to 2D via random projections. Let Q be a random rotation matrix. The rotated 3D skeleton Y j i is obtained as where X j r is the predicted root joint of j th skeleton and C = (0, 0, c) T . Let y j i denote the 2D projection of Y j i .

Training Losses for the Teacher Network
Multi-Frame Self-Supervision Loss: Let G T denote the teacher network that predicts the model-free 3D pose X j i = G T (x j i ) as defined in Section 3.1. We also lift the reprojected 2D skeletons to obtain Therefore, we define our multi-frame self-supervision loss as Temporal Consistency Loss: Since we predict T 3D skeletons at each time step t, common frames exists between neighboring time-steps. Using a sliding window with temporal stride 1, we have T − 1 frames in common between time-step t and t + 1. We use an L 2 loss to enforce consistency between these common frames in 3D, Bone Length Loss: At each time step t, we enforce that the bone lengths for the T predicted 3D skeletons be consistent by minimizing the variance of bone lengths over the T frames. Let b(m, n, j) = X j m − X j n denote the bone length between the m th and n th predicted 3D joints for frame j. Bone length loss is defined as where N (m) denotes the set of connected skeleton joints for joint m and Var j denotes variance over T frames. Temporal Discriminator: The discriminator provides feedback to the lifter regarding the realism of projected 2D skeletons. In contrast to [6], which uses a single frame discriminator and a frame-difference discriminator, we use a single temporal discriminator that takes a set of T reprojected/real 2D skeletons as input. Previous approaches have used RNN and LSTM to model sequential/temporal data. A challenge in using RNN/LSTM is delayed feedback which requires the use of a policy gradient to back-propagate the feedback from the discriminator [11]. In contrast, our temporal discriminator uses dilated convolutions and provides feedback at each time-step, simplifying the training. Formally, the discriminator is trained to distinguish between sequences of T real 2D skeletons r(t) = r 1 , . . . , r T (target probability of 1) and fake (projected) 2D skeletons y(t) = y 1 , . . . , y T (target probability of 0). We utilize a standard adversarial loss [16] defined as where Θ T and Θ D denote the parameters of the teacher and the discriminator networks, respectively.

Student: Estimating SMPL Parameters
For our model-based approach, we use the Skinned Multi-Person Linear (SMPL) representation [37]. SMPL is a parametric model that factors human bodies into a shape (body proportions) and pose (articulation) representation. The shape is parameterized using a PCA subspace with 300 basis shapes and shape coefficients (β). The human pose is modeled as a set of 24 local joint angles corresponding to K = 24 3D joints (including root joint) and is represented as 72 axis-angle coefficients. We directly predict the rotation matrix corresponding to each joint from the network, on which we perform a differentiable ortho-normalization. Let R = {R 1 , . . . , R K } denote the set of K rotation matrices. Given a set of parameters β and R, SMPL produces a mesh V = M(β, R), V ∈ R 6890×3 with 6890 vertices, where M is differentiable. Note that the 3D joints by themselves do not fully constrain the shape of the body and it is not possible to predict accurate shape using 3D joints alone. Approaches such as [51] have additionally used silhouettes to estimate shape and thus accurate shape prediction is not a goal of this paper. We only predict the first 10 β parameters (common for all T frames) and set the remaining to zero. Thus, the student network has a total of 10 + 24 × 9 × T = 10 + 216 × T outputs at each time step.

Semantic Joint Adaptation (SJA)
The 3D joints J ∈ R 24×3 are obtained by linear regression from the final mesh vertices V . The linear regressor is a sparse matrix W ∈ R 24×6890 , which represents a convex combination of vertices for each 3D joint. Hence, J = W V . The pre-trained linear regressor in SMPL produces 3D joints that are often semantically different from the 2D joints obtained from 2D pose detectors or annotations on datasets. For example, in SMPL the 3D hip joints are closer to the center of the body. However, in Hu-man3.6M 2D annotations, the hip landmarks are closer to the periphery. Our SJA module learns the adaptation of the SMPL 3D joints to 2D joints used for training.
We first experimented with a linear layer that learns a weight matrix A ∈ R 72×72 and an bias vector b ∈ R 72×1 , which is applied to the 72 × 1 vectorized representation of J to adapt the SMPL joints (referred to as Linear-SJA), However, such an approach fails in practice. Since there is no constraint on joints, the network can potentially learn to minimize the joint error by moving the SMPL joints outside the body (Fig. 2). To avoid such pitfalls, similar to SMPL, we learn a convex combination of vertices W , resulting in 24 × 6890 = 165, 360 additional learnable parameters and obtain the new joints as J = W V . Visualization in Fig. 3 shows that weights for the learned regressor on Human3.6M shifts from the center of body towards the surface, corresponding to a similar shift in 2D hip landmarks. For the rest of the paper, SJA refers to the learned convex combination of vertices.

Training Losses for the Student
The following losses are used to train the student network via knowledge distillation. Knowledge Distillation Loss: We define a loss between the model-free prediction of 3D joints X j i and the 3D joints obtained via the SMPL model. To account for the mismatch between the number of joints, we choose N (14 in our case) relevant joints from the 24 SMPL joints. Let I(i) denote the index of the SMPL joint corresponding to i th input 2D joint. L KD is computed as a sum of individual losses over each joint i and each frame j, where W I(i) denote the row of matrix W corresponding to the regressor weights for joint I(i) and R j denotes the set of predicted rotation matrices for frame j. Regularization of SMPL Parameters: In absence of any 3D data, we use a simple regularizer for pose parameters to avoid over-twisting by penalizing the deviation of the predicted rotation matrices from identity rotation.
where I 3×3 is the 3 × 3 identity matrix. We use a similar L 2 regularizer for β, L β = β 2 , since β = 0 represents the average human shape. The β regularizer is used with a relatively larger weight during training to keep the shape close to the average shape. However, we show in Section 4.5 (Fig. 7) that without SJA, these regularizers by themselves are not sufficient to avoid unnatural predictions. Our novel SJA module helps improve the realism and naturalness of predicted pose parameters.
Discriminator: Similar to the teacher network, the predicted 3D joints from the student network are reprojected to random views and fed to the discriminator. The corresponding discriminator loss is L S D , similar to L T D in Eqn. 5.

Training PoseNet3D
We train PoseNet3D following these steps: 1. Train Teacher: Train the shared temporal convolution backbone and the teacher branch by minimizing L T = λ mss L mss + λ tc L tc + λ bl L bl + L T D . 2. Knowledge Distillation: Freeze the shared temporal backbone and the teacher branch. Train the student branch by minimizing L S = L KD + λ R L θ + λ β L β . 3. Learn SJA: Initialize W to W . Fine-tune W and the student branch by minimizing L S . 4. Fine-tune the entire network by minimizing L = L T + λ S L S + L S D .
Hyper-parameters λ mss , λ tc , λ bl , λ R , λ β , λ S are defined in Sect. 4.1. Note that in step 4, we feed the re-projection of the 3D pose predicted from both the teacher and student networks to the discriminator.

Experiments and Results
We evaluate on the widely used Human3.6M [22] and MPI-INF-3DHP [40] datasets and show quantitative and qualitative results. We also show qualitative visualizations of reconstructed skeletons and meshes on in-thewild datasets such as Leeds Sports Pose [26] (LSP) where ground-truth 3D data is not available. Since our approach takes temporal sequences, for inference on single frame input (e.g. LSP), we simply copy the skeleton T times.

Implementation Details
We use N = 14 joints and randomly sample T = 9 frame sub-sequences of 2D poses from video sequences for training. The input poses are normalized such that the mean distance from the head joint to the root joint is 0.1 units, corresponding to placing the 3D skeletons at c = 10 units from the camera. As shown in Fig. 1, our temporal backbone takes a 2N × T input followed by a conv-block, comprising of convolution filter, batchnorm, ReLU and dropout. Each convolution filter has 1024 channels with a kernel size of 3 × 1 and temporal dilation factor of d = 1. The output of the conv-block is fed to a residual block with two conv-blocks, with a dilation ratio of d = 3 and d = 1, respectively. The teacher branch consists of an additional residual block with 2 fully-connected (FC) layers of size 1024 each. Similarly, the student branch consists of 4 FC residual blocks. The temporal discriminator architecture is identical to the temporal backbone architecture but does not use BatchNorm. We train on TitanX GPUs using the Adam optimizer [29] with batchsize of 6000 and learning rate of 0.0001 for 150 epochs. The loss weights are set as λ mss = 2, λ tc = 1, λ bl = 2, λ R = 30, λ β = 10 and λ S = 2.

Datasets and Metrics
Human3.6M (H3.6M): This is one of the largest 3D human pose datasets, consisting of 3.6 million 3D human poses. The dataset contains video and MoCap data from 11 subjects performing typical activities such as walking, sitting, etc. Similar to previous works [6,13,27], we report the mean per joint position error in mm after scaling and rigid alignment to the ground truth skeleton (P-MPJPE) on subjects S9 and S11 (all cameras). We only use 2D data from subjects S1, S5, S6, S7 and S8 for training a single activityagnostic model. To evaluate the smoothness of predicted 3D pose, we report mean per joint velocity error (MPJVE), which is calculated as the mean per joint error of the first derivative of the aligned 3D pose sequences. We also propose the mean bone-length standard deviation (MBLSTD) metric as the average standard deviation (in mm) of 8 bone segments (corresponding to upper/lower/left/right hand/leg) over all sequences. Lower values of MPJPE, MPJVE, and MBLSTD signify better performance. MPI-INF-3DHP: The MPI-INF-3DHP dataset consists of 3D data captured using a markerless multi-camera MoCap system. The ground truth 3D annotations have some noise, which is a known symptom of markerless MoCap systems. We evaluate on valid images from test-set containing 2929 frames following [27] and report P-MPJPE, Percentage of Correct Keypoints (PCK) @150mm, and Area Under the Curve (AUC) computed for a range of PCK thesholds.

Quantitative Results
We denote our results obtained by taking the average of the predicted 3D poses from the teacher and the student networks as PoseNet3D. For a fair comparison with modelfree methods in Table 1, we also report the results from the teacher branch (PoseNet3D-Teacher) after Step 1 of training (Sect. 3.5). Similarly, results from the student branch after Step 3 of training are denoted as PoseNet3D-Student in Table 2. The corresponding results from the two branches after fine tuning (Step 4, Sect. 3.5) are denoted by PoseNet3D-Teacher-FT and PoseNet3D-Student-FT, respectively.
As evident from  H3.6M only). Our best model reduces the P-MPJPE error from 58mm to 47mm (18% improvement). We also outperform several previous weakly-supervised approaches that use 3D information in training. Similarly, on H3.6M, our results are better than previous model-based approaches such as HMR and SPIN that use unpaired 3D data and produce SMPL meshes as output (Table 2). Finally, Table 3 summarizes results on MPI-INF-3DHP. PoseNet3D model trained on H3.6M outperforms HMR [27] and come close to the results from SPIN [31], both of which were trained on MPI-INF-3DHP and used unpaired 3D data for training. This offers a strong evidence that PoseNet3D generalizes well to out-of-domain datasets (e.g., in this case trained on H3.6M and tested on MPI-INF-3DHP). Figure 4 shows overlay of generated mesh using predicted SMPL parameters on the corresponding image, for a few examples from H3.6M, LSP and MPI-INF-3DHP datasets. As discussed earlier, since our approach uses only 2D landmarks and cannot estimate accurate shape, pro-    Table 4 analyzes the impact of number of frames for the teacher network with and without fine-tuning in terms of MPJVE and MBLSTD metrics. We implemented the approach of Chen et al. [6] to compute similar metrics and our 9-frame teacher network outperforms their approach, reducing MPJVE and MBLSTD by more than 60% and 30%, respectively.

Ablation Studies
For the student branch, we first define two baselines.     [44]. ‡ denotes our implementation of [6].
Since SMPL is a parametric model, a trivial baseline is to train the student network directly by minimizing the 2D re-projection error (Baseline 1). We also propose an additional baseline by employing SJA on top (Baseline 2). As Figure 8: Without knowledge distillation, directly training the student network with 2D re-projection loss results in monster meshes, even when the 2D loss is small.
noted in [1] and specifically in [27], minimizing the 2D reprojection error without any 3D supervision can result in monster meshes with high P-MPJPE. We observe a similar phenomenon. As shown in Table 4 and Fig. 8, these baselines results in a high P-MPJPE and do not predict high quality poses.
Finally, we analyze the effect of SJA on the student network using KD. Linear-SJA (Eqn. 6) results in low 3D error on training data but high 3D error on test data (PoseNet3D-S-LinearSJA). The network severely overfits on the training data by moving joints outside the body (see Fig. 2 for examples). Without using SJA (PoseNet3D-S-noSJA), the error is higher than using SJA (PoseNet3D-S-SJA). Visualization in Fig. 7 compares the output of student network with and without SJA. Notice how the re-projected 3D skeleton is semantically closer to the input 2D skeleton with SJA (especially hip and head joints). SJA results in better 3D pose predictions confirming that in absence of any paired/unpaired 3D supervision, our semantic joint adaptation module is essential for training the student network.

Conclusions
We present a knowledge distillation algorithm to learn SMPL pose parameters from 2D joints, without requiring additional 3D data for training. Our approach trains a feedforward network to predict SMPL parameters and does not require any iterative fitting. We first learn a teacher network to lift 2D joints to model-free 3D pose in a temporally consistent manner. The temporal dynamics are modeled using dilated convolutions in both lifter and discriminator, allowing feedback at every time-step and avoids common pitfalls in using LSTM/RNN in such settings. The teacher network provides pseudo ground-truth to the student network which learns to predict SMPL pose parameters. We demonstrate how to bridge the semantic gap between the SMPL 3D joints and 2D pose landmarks during training, which has been largely ignored in previous literature. We believe that our paper has significantly improved the state-of-art in learning of 3D pose from 2D skeletons in absence of explicit 3D supervision.