Abstract
Depictions of similar human body configurations can vary with changing viewpoints. Using only 2D information, we would like to enable vision algorithms to recognize similarity in human body poses across multiple views. This ability is useful for analyzing body movements and human behaviors in images and videos. In this paper, we propose an approach for learning a compact view-invariant embedding space from 2D joint keypoints alone, without explicitly predicting 3D poses. Since 2D poses are projected from 3D space, they have an inherent ambiguity, which is difficult to represent through a deterministic mapping. Hence, we use probabilistic embeddings to model this input uncertainty. Experimental results show that our embedding model achieves higher accuracy when retrieving similar poses across different camera views, in comparison with 2D-to-3D pose lifting models. The results also suggest that our model is able to generalize across datasets, and our embedding variance correlates with input pose ambiguity.
Introduction
When we represent three dimensional (3D) human bodies in two dimensions (2D), the same human pose can appear different across camera views. There can be significant visual variations from a change in viewpoint due to changing relative depth of body parts and self-occlusions. Despite these variations, humans have the ability to recognize similar 3D human body poses in images and videos. This ability is useful for computer vision tasks where changing viewpoints should not change the labels of the task. We explore how we can embed 2D visual information of human poses to be consistent across camera views.
Inspired by 2D-to-3D lifting models [26] , we learn view invariant embeddings directly from 2D pose keypoints. As illustrated in Figure 1 , we explore whether view invariance of human bodies can be achieved from 2D poses alone, without predicting 3D pose. Typically, embedding models * This work was done while the author was a research qfischer@example.net. are trained from images using deep metric learning techniques [28, 11, 7] . However, images with similar human poses can appear different because of changing viewpoints, subjects, backgrounds, clothing, etc. As a result, it can be difficult to understand errors in the embedding space from a specific factor of variation. Furthermore, multi-view image datasets for human poses are difficult to capture in the wild with 3D groundtruth annotations. In contrast, our method leverages existing 2D keypoint detectors, allowing the embedding model to focus on learning view invariance. Our 2D keypoint embeddings can be trained using datasets in lab environments, while having the model generalize to in-thewild data. Another advantage is that we can easily augment training data by synthesizing multi-view 2D poses from 3D poses through perspective projection.
Using our method, we explore the question: can we achieve view invariance without predicting absolute 3D pose? This enables us to work with lower dimensional representations in the embedding space and opens up the potential to train these models without 3D pose annotations in the future.
Another aspect that we address is input uncertainty. The input to our embedding model is 2D human pose, which has an inherent ambiguity. Many valid 3D poses can be projected to the same or very similar 2D pose [1] . This input uncertainty is difficult to represent using deterministic map-pings to the embedding space (point embeddings) [29, 19] . Our embedding space consists of probabilistic embeddings based on multivariate Gaussians, as shown in Figure 1b . We show that the learned variance from our method correlates with input 2D ambiguities. We call our approach Pr-VIPE for Probabilistic View-Invariant Pose Embeddings. The non-probabilistic, point embedding formulation will be referred to as VIPE.
Contributions Our main contribution is the method for learning an embedding space where 2D pose embedding distances correspond to their similarities in absolute 3D pose space. We also develop a probabilistic formulation that captures 2D pose ambiguity. The view invariance property of our embeddings can be leveraged for downstream tasks. In this paper, our evaluation focuses on cross-view pose retrieval: given a single view image, we retrieve images of the same pose from different views without using camera parameters. We develop a method to compute retrieval confidence based on [29] and show that higher confidence correlates with smaller retrival error. Our results suggest that 2D poses are sufficient to achieve view-invariance in the absence of more information from images, and we do not have to predict absolute 3D pose to achieve this property. We plan to open-source our code and experiment details.
Related Work
Metric Learning We are working to understand similarity in human poses across views. Most works that aim to capture similarity between inputs generally apply techniques from metric learning. Objectives such as contrastive loss (based on pair matching) [4, 9, 29] and triplet loss (based on tuple ranking) [45, 40, 46, 10] are often used to push together/pull apart similar/dissimilar examples in embedding space.
The number of possible training tuples increases exponentially with respect to the number of samples in the tuple, and not all combinations are equally informative. To find informative training tuples, various mining strategies are proposed [40, 47, 30, 10] . In particular, semi-hard triplet mining has been widely used [40, 47, 33] . This mining method finds negative examples that are fairly hard as to be informative but not too hard for the model. The hardness of a negative sample is based on its embedding distance to the anchor. Commonly, this distance is the Euclidean distance [45, 46, 40, 10] , but any differentiable distance function could be applied [10] . [13, 15] show that alternative distance metrics also work for image and object retrieval.
In our work, we learn a mapping from Euclidean distance in the embedding space to a probabilistic pose similarity score. This probabilistic similarity captures closeness in 3D pose space from 2D poses. Our work is inspired by the mapping used in soft contrastive loss [29] for learning from an occluded N-digit MNIST dataset.
Most of the papers discussed above involve deterministically mapping inputs to point embeddings. There are works that also map inputs to probabilistic embeddings. Probabilistic embeddings have been used to model specificity of word embeddings [44] , uncertainty in graph representations [3] , and input uncertainty due to occlusion [29] . We will apply probabilistic embeddings to address inherent ambiguities in 2D pose due to 3D-to-2D projection.
Human Pose Estimation 3D human poses in a global coordinate frame are view-invariant, since images across views are mapped to the same 3D pose. However, as mentioned by [26] , it is difficult to infer the 3D pose in an arbitrary global frame since any changes to the frame does not change the input data. Many approaches work with poses in the camera coordinate system [26, 5, 34, 37, 50, 41, 38, 42, 6] , where the pose description changes based on viewpoint.
Our approach is similar in setup to existing 2D-to-3D lifting pose estimators [26, 5, 34, 37] in terms of using 2D pose keypoints as input. The difference is that lifting models are trained to regress to 3D pose keypoints, while our model is trained using metric learning and outputs an embedding distribution. Some recent works also use multiview datasets to predict 3D poses in the global coordinate frame [35, 21, 16, 39, 43] . Our work differs from these methods with our goal (view-invariant embeddings), task (cross-view pose retrieval), and approach (metric learning). Another work that focuses on pose retrieval [28] embeds images with similar 2D poses in the same view close together. Our method focuses on learning view-invariance, and we also differ from [28] in method (probabilistic 2D pose embeddings).
View Invariance and Object Retrieval When we capture a 3D scene in 2D as images or videos, changing the viewpoint often does not change other properties of the scene. The ability to recognize visual similarities across viewpoints is helpful for a variety of vision tasks, such as motion analysis [18, 17] , tracking [31] , vehicle and human re-identification [7, 49] , object classification and retrieval [22, 12, 11] , and action recognition [36, 25, 48, 23] .
Some of these works [11] focus on metric learning for object retrieval. Their learned embedding spaces place different views of the same object class close together. Our work on human pose retrieval differs in a few ways. Our labels are continuous 3D poses, whereas in object recognition tasks, each embedding is associated with a discrete class label. Furthermore, we embed 2D poses, while these works embed images. Our approach allows us to investigate the impact of input 2D uncertainty with probabilistic embeddings and explore different methods to measure cross-view pose retrieval confidence. We hope that our work provides a novel perspective on view invariance for human poses. 
Our Approach
Our goal is to embed 2D poses such that distances in the embedding space correspond to similarities of their corresponding absolute 3D poses in Euclidean space. We achieve this view invariance property through our triplet ratio loss (Section 3.2), which pushes together/pull apart 2D poses corresponding to similar/dissimilar 3D poses. The positive pairwise loss (Section 3.3) is applied to increase the matching probability of similar poses. Finally, the Gaussian prior loss (Section 3.4) helps regularize embedding magnitude and variance. The training and inference framework of Pr-VIPE is illustrated in Figure 2 
Matching Definition
The 3D pose space is continuous, and two 3D poses can be trivially different without being identical. We define two 3D poses to be matching if they are visually similar regardless of viewpoint. Given two sets of 3D keypoints (y i , y j ), we define a matching indicator function
where κ controls visual similarity between matching poses.
Here, we use mean per joint position error (MPJPE) [14] between the two sets of 3D pose keypoints as a proxy to quantify their visual similarity. Before computing MPJPE, we normalize the 3D poses and apply Procrustes alignment between them. The reason is that we want our model to be view-invariant and to disregard rotation, translation, or scale differences between 3D poses. We refer to this normalized, Procrustes aligned MPJPE as NP-MPJPE.
Triplet Ratio Loss
The triplet ratio loss aims to embed 2D poses based on the matching indicator function (1) . Let n be the dimension of the input 2D pose keypoints x, and d be the dimension of the output embedding. We would like to learn a mapping f :
For a pair of input 2D poses (x i , x j ), we define p(m|x i , x j ) to be the probability that their corresponding 3D poses (y i , y j ) match, that is, they are visually similar. While it is difficult to define this probability directly, we propose to assign its values by estimating p(m|z i , z j ) via metric learning. We know that if two 3D poses are identical, then p(m|x i , x j ) = 1, and if two 3D poses are sufficiently different, p(m|x i , x j ) should be small. For any given input triplet (
where β > 1 represents the ratio of the matching probability of a similar 3D pose pair to that of a dissimilar pair. Applying negative logarithm to both sides, we have
(3) Notice that the model can be trained to satisfy this with the triplet loss framework [40] . Given batch size N , we define triplet ratio loss L ratio as
we set the anchor x i and positive x i + to be projected from the same 3D pose and perform online semi-hard negative mining [40] to find x i − .
It remains for us to compute matching probability using our embeddings. To compute p(m|z i , z j ), we use the formulation proposed by [29] :
where σ is a sigmoid function, and the trainable scalar parameters a > 0 and b ∈ R calibrate embedding distances to probabilistic similarity.
Positive Pairwise Loss
The positive pairs in our triplets have identical poses. We would like them to have high matching probabilities, which can be encouraged by adding the positive pairwise loss
The combination of L ratio and L positive can be applied to training point embedding models, which we refer to as VIPE in this paper.
Probabilistic Embeddings
In this section, we discuss the extension of VIPE to the probabilistic formulation Pr-VIPE. The inputs to our model, 2D pose keypoints, are inherently ambiguous, and there are many valid 3D poses that can be projected to a similar 2D pose [1] . This input uncertainty can be difficult to model using point embeddings [19, 29] . We investigate representing this uncertainty using distributions in the embedding space by mapping 2D poses to probabilistic embeddings: x → p(z|x). Similar to [29] , we extend the input matching probability (5) to using probabilistic embeddings
which can be approximated using Monte-Carlo sampling with K samples drawn from each distribution as
).
We model p(z|x) as a d-dimensional Gaussian with a diagonal covariance matrix. The model outputs mean µ(x) ∈ R d and covariance Σ(x) ∈ R d with shared base network and different output layers. We use the reparameterization trick [20] during sampling.
In order to prevent variance from collapsing to zero and to regularize embedding mean magnitudes, we place a unit Gaussian prior on our embeddings with KL divergence by adding the Gaussian prior loss
Inference At inference time, our model takes a single 2D pose (either from detection or projection) and outputs the mean and the variance of the embedding Gaussian distribution.
Keypoint Augmentation
Our triplets can be made of detected and/or projected 2D keypoints as shown in Figure 2 . When we train only with detected 2D keypoints, we are constrained to the camera views in training images. To reduce overfitting to these camera views, we perform keypoint augmentation by generating triplets using detected keypoints alongside projected 2D keypoints at random views.
To form triplets using multi-view image pairs, we run 2D keypoint detection and use detected 2D keypoints from different views as anchor-positive pairs. To use projected 2D keypoints, we perform two random rotations to a normalized input 3D pose to generate two 2D poses from different views for anchor/positive. The projection is based on a camera with unit focal length and centered at the origin. For finding negative matches, we perform online semihard mining [40] . Keypoint augmentation is then performed by using a mixture of detected and projected 2D keypoints as anchors, positives, and negatives. We find that training using keypoint augmentation can help our models learn to generalize better to unseen views (Section 4.3).
Implementation Details
Our 3D pose normalization procedure is based on [6] , and we perform instance normalization to our 2D poses. More details are provided in the appendix.
The backbone network architecture for our model is based on [26] for its simplicity. We use two residual blocks, batch normalization, 0.3 dropout, and no weight norm clipping. We use d = 16 as a good trade-off between embedding size and accuracy. To weigh different losses, we use w ratio = 1, w positive = 0.005, and w prior = 0.001. We choose β = 2 for the triplet ratio loss margin and K = 20 for the number of samples. During training, we normalize matching probabilities to within [0.05, 0.95] for numerical stability. The matching NP-MPJPE threshold is κ = 0.1 for all training and evaluation. Our approach does not rely on a particular 2D keypoint detector, and we use PersonLab [32] for our experiments. For random rotation during keypoint augmentation, we uniformly sample azimuth angle between ±180 • , elevation between ±30 • , and roll between ±30 • . We use Adagrad optimizer [8] with fixed learning rate 0.02, and batch size N = 256. Additional ablation studies on hyperparameters can be found in the appendix. Our implementation is in TensorFlow, and all the models are trained with CPUs.
Experiments
We demonstrate the performance of our model through pose retrieval across different camera views. Given a multiview dataset, we query using detected 2D pose keypoints from one camera view and find the nearest neighbors in the embedding space from a different camera view. We iterate through all camera pairs in the dataset as query and index. Results averaged across all cameras pairs are reported.
Datasets
For all experiments, we train on a subset of the Hu-man3.6M [14] dataset. We present quantitative and qualitative results on the Human3.6M hold-out set and another dataset (MPI-INF-3DHP [27] ) unseen during training. We also present qualitative results on MPII Human Pose [2] , for which 3D groundtruth is not available.
Human3.6M (H3.6M) H3.6M is a large human pose dataset containing 3.6 million image frames from 4 chest level cameras with 3D pose groundtruth. We follow the standard protocol [26] : subjects 1, 5, 6, 7, and 8 for training, and subjects 9 and 11 held out for validation. For evaluation, we downsample the frames to 10Hz and further remove near-duplicate 3D poses within 0.02 NP-MPJPE. This process is camera-consistent, meaning if a frame is selected under one camera, it is selected under all cameras, so that the perfect retrieval result is possible. This results in a total of 10910 evaluation frames per camera. We pick the best training checkpoint for all models using this dataset.
MPI-INF-3DHP (3DHP) 3DHP is a more recent human pose dataset containing over 1.3 million images from 14 diverse camera views and scenarios, covering more pose variations than H3.6M [27] . We use 11 cameras from this dataset and exclude the 3 cameras with overhead views. Similar to H3.6M, we downsample frames to 5Hz and remove near-duplicate 3D poses, resulting in 6824 frames per camera.
MPII Human Pose (2DHP) This dataset is commonly used in 2D pose estimation, containing 25K images from YouTube videos. Since groundtruth 3D poses are not available, we only show qualitative results on this dataset.
Evaluation Procedure
Metric We report Hit@k with k = 1, 5, 10, and 20 on pose retrievals. A retrieval is considered accurate if the 3D groundtruth from the retrieved pose satisfies the matching function (1) . All of our results are based on κ = 0.1. The metric Hit@k is the percentage of top-k retrieved poses that have at least one accurate retrieval. k = 1 measures model accuracy while k > 1 measures model recall.
Baseline Approaches We compare Pr-VIPE with 2Dto-3D lifting models [26] and L2-VIPE. L2-VIPE outputs L2-normalized point embeddings, and is trained with the squared L2 distance kernel, similar to [40] .
For fair comparison, we use the same backbone network architecture for all the models. Notably, this architecture [26] has been tuned for the lifting task on H3.6M. Since the estimated 3D poses in the camera coordinate system are not view-invariant, we apply normalization and Procrustes alignment when running pose retrieval to align the estimated 3D poses between index and query. For comparison, our embeddings are used without alignment or other post-processing during retrieval. For Pr-VIPE, we retrieve poses using nearest neighbors in the embedding space with respect to the sampled matching probability (7), which we refer to as retrival confidence. We present the results on the VIPE models with and without keypoint augmentation. We applied similar keypoint augmentation to the lifting model, but did not see improvement in performance. We also show the results of pose retrieval using aligned 2D keypoints only. The poor performance of using input 2D keypoints for retrieval from different views confirms the fact that models must learn view invariance from inputs for this task.
Pose Retrieval Evaluation
Human3.6M Results From Table 1 , we see that Pr-VIPE outperforms all baselines. We reduce retrieval top-1 error by 23.2% compared with 2D-to-3D lifting, and 10.2% compared with L2-VIPE. It is noteworthy that Pr-VIPE is able to achieve better performance with more compact embeddings (16 dimensions) compared with the lifting model output (39 dimensions). We further investigate the effect of embedding dimensions in Section 4.5. Keypoint augmentation reduces performance on H3.6M for both the Pr-VIPE and the L2-VIPE model. This is likely because augmentation reduces overfitting to the training camera views. As we will show on the unseen 3DHP dataset, keypoint augmentation improves model generalization to new views. Our model is robust to the choice of β and the number of samples K, for which detailed analysis is provided in the appendix.
MPI-INF-3DHP Results
We test the ability of our models to generalize to new poses and views using 3DHP. We separate our evaluations using either all 11 cameras or a subset of 5 chest-level cameras. When we use all cameras from 3DHP, we evaluate on camera positions from different elevations, for example, from knee-level. When we evaluate using only the 5 chest-level cameras from 3DHP, the views are more similar to H3.6M, and generalization to new poses becomes more important. Note that all models are trained with H3.6M (4 chest-level cameras) without seeing 3DHP.
As the results show in Table 2 , Pr-VIPE without keypoint augmentation is able to perform better than the baselines for chest-level cameras. These results show that Pr-VIPE is able to generalize as well as other baseline methods to new poses. However, for all cameras in 3DHP, the performance for Pr-VIPE without augmentation is notably worse compared with chest-level cameras. This observation indicates that when trained on chest-level cameras only, Pr-VIPE does not generalize as well to new views. The same results can be observed for L2-VIPE between chest-level and all cameras. In contrast, the 3D lifting models are able to generalize better to new views with the help of additional Procrustes alignment, which requires expensive SVD computation for every index-query pair.
We further apply keypoint augmentation to training the Pr-VIPE and the L2-VIPE models. Note that this step does not require camera parameters or additional groundtruth. The results from Table 2 on Pr-VIPE show that the augmentation improves performance in all-camera retrieval by 6% to 9% for all metrics. This step also increases chest-level camera accuracy slightly. For L2-VIPE, we can observe a similar increase on all views. By performing keypoint augmentation, Pr-VIPE is able to generalize better to new poses and new views. Figure 4 shows qualitative retrieval results using Pr-VIPE. As shown in the first row, the retrieval confidence of the model is generally high for H3.6M. This indicates that the retrieved poses are close to the queries in the embedding space. Errors in 2D keypoint detection can lead to retrieval errors as shown by the rightmost pair. In the second and the third row, the retrieval confidence is lower for 3DHP. This is likely because there are new poses and views unseen during training, which has the nearest neighbor slightly further away in the embedding space. We see that the model can generalize to new views as the images are taken at different camera elevations from H3.6M. Interestingly, the rightmost pair on row 2 shows that the model can retrieve poses with large differences in roll angle, which is not present in the training set. The rightmost pair on row 3 shows an example of a large NP-MPJPE error due to mis-detection of the left leg in the index pose.
Qualitative Results
We show qualitative results using queries from H3.6M hold-out set to retrieve from 2DHP in the last two rows of Figure 4 . The results on these in-the-wild images indicate that as long as the 2D keypoint detector works reliably, our model is able to retrieve poses across views and subjects. More qualitative results are provided in the appendix.
We present more qualitative results for Pr-VIPE on all datasets in Figure 13 . The first 2 rows show results on H3.6M, the next 3 rows are on 3DHP and the last 3 rows shows results using the hold-out set in H3.6M to retrieve from 2DHP. We are able to retrieve across camera views and subjects on all datasets.
On H3.6M, retrieval confidence is generally high and retrievals are visually accurate. NP-MPJPE is in general smaller on H3.6M compared to 3DHP, since 3DHP has more diverse poses and camera views. The model works reasonably well on 3DHP despite additional variations on pose, viewpoints and subjects. For the pairs on the right in rows 4 and 5, the subject is occluded by the chair and the pose inferred by the 2D keypoint detector may not be accurate. Our model is dependent on the result of the 2D keypoint detector. Interestingly, the right pair in row 4 and the middle pair in row 3 shows retrievals with large rolls, which is unseen during training. The results on 3DHP demonstrate the generalization capability of our model to unseen poses and views. To test on in-the-wild images, we use the hold-out set of H3.6M to retrieve from 2DHP. The retrieval results demonstrate that Pr-VIPE embeddings can retrieve visually accurate poses from detected 2D keypoints. The left pair in the last row is particularly interesting, as the retrieval has a large change in viewpoint. For the low confidence pairs in the right on the third to last row and last row, we can see that the arms of the subjects seems to be bent slightly differently. In contrast, the higher confidence retrieval pairs looks visually similar. The results suggest that performance of existing 2D keypoint detectors, such as [32] , is sufficient to train pose embedding models to achieve the view-invariant property in diverse images. 
Ablation Study
Embedding Space Visualization We run Principal Component Analysis (PCA) on the 16-dimensional embeddings using the Pr-VIPE model and visualize the first two principal dimensions in Figure 3 . To visualize more unique poses, we randomly subsample the H3. respectively. Despite the similar retrieval accuracies, Pr-VIPE is generally more accurate and, more importantly, has additional desirable properties in that the variance can model 2D input ambiguity as to be discussed next.
A 2D pose is ambiguous if there are similar 2D poses corresponding to very different poses in 3D. To measure this, we compute the average 2D NP-MPJPE between a 2D pose and its top-10 nearest neighbors in terms of 2D NP-MPJPE. To ensure that the 3D poses are different, we sam- ple 1200 poses from H3.6M hold-out set with a minimum gap of 0.1 3D NP-MPJPE. If a 2D pose has small 2D NP-MPJPE to its neighbors, it means there are many similar 2D poses corresponding to different 3D poses, or, in other words, the 2D pose is ambiguous. Figure 5 shows that the 2D pose with the greatest variance is ambiguous as it has similar 2D poses in H3.6M with different 3D poses. In contrast, we see that the closest 2D poses corresponding to the smallest variance pose on the first row of Figure 5 are clearly different. Figure 6 shows that as the average variance increases, the 2D NP-MPJPE between similar poses generally decreases, which means that 2D poses with larger variances are more ambiguous.
Embedding Dimensions Figure 7 demonstrates the effect of embedding dimensions on H3.6M and 3DHP. The lifting model lifts 13 2D keypoints to 3D, and therefore has a constant output dimension of 39. We see that Pr-VIPE is able to achieve a higher accuracy than lifting on both datasets at 16 embedding dimensions. Additionally, we can increase the number of embedding dimensions to 32, which increases accuracy of Pr-VIPE to 75.5%. Our method can extend to additional dimensions with no additional annotations, but to increase dimension of the predicted 3D pose, additional annotation of more 2D and 3D keypoints are needed.
Retrieval Confidence In order to validate the retrieval confidence values, we randomly sample 100 queries along with their top-5 retrievals (using Pr-VIPE retrieval confidence) from each query-index camera pair. This proce- dure forms 6000 query-retrieval sample pairs for H3.6M (4 views, 12 camera pairs) and 55000 for 3DHP (11 views, 110 camera pairs), which we further bin by their retrieval confidences. Figure 8 shows the matching accuracy under κ = 0.1 for each confidence bin. We can see that the accuracy positively correlates with the confidence values, which suggest our retrieval confidence is a valid indicator to model performance.
What if 2D keypoint detectors were perfect? We repeat our experiments using groundtruth 2D keypoints to simulate a perfect 2D keypoint detector on H3.6M and 3DHP. All experiments use the 4 views from H3.6M for training following the standard protocol. For the baseline lifting model in camera frame, we achieve 55.5% Hit@1 on H3.6M, 30.6% on 3DHP (all), and 25.9% on 3DHP (chest). For Pr-VIPE, we achieve 97.5% Hit@1 on H3.6M, 44.3% on 3DHP (all), and 66.4% on 3DHP (chest). Using perfect 2D keypoints, the Pr-VIPE model is able to perform much better than the lifting model. Comparing the results with using detected keypoints, the large improvement in performance using groundtruth keypoints suggests that a considerable fraction of error in our model is due to imperfect 2D keypoint detections.
Conclusion
We introduce Pr-VIPE, an approach to learning probabilistic view-invariant embeddings from 2D pose keypoints.
Our experiments suggest that input 2D keypoints alone are sufficient to achieve view invariant properties in the embedding space, without having to explicitly predict 3D pose. By working with 2D keypoints, we can use synthetic projection augmentation to improve model generalization to unseen camera views. We also demonstrate that our probabilistic embeddings learn to capture input ambiguity, which can be useful for measuring uncertainty in downstream tasks. Pr-VIPE is compact with a simple architecture, and in addition to cross-view retrieval, our embeddings can be applied to other human pose related tasks. We hope that our work can contribute towards future studies in recognizing human poses and body motions.
Acknowledgement
We are grateful to Yuxiao Wang and Liangzhe Yuan from Google Research and Xiao Zhang from University of Chicago for helpful discussions. We really appreciate the support of Pietro Perona and the Computational Vision Lab at Caltech for making this collaboration possible. The 3D pose space is continuous, and we use the NP-MPJPE as a proxy to quantify visual similarity between pose pairs. Figure 9 shows pairs of 3D pose keypoints with their corresponding NP-MPJPE, where each row depicts a different NP-MPJPE range. This plot demonstrates the effect of choosing different κ, which controls matching threshold between 3D poses. If we choose κ = 0.05, then only the first row in Figure 9 would be considered matching, and the rest of the rows are non-matching. Our current value of κ = 0.10 corresponds to using the first two rows as matching pairs and the rest of the rows as non-matching ones. By loosening κ, poses with greater differences will be considered as matching, as shown by different rows in Figure 9 . We note that pairs in rows 3 and 4 shows significant visual differences compared with the first two rows. We further investigate the effects of different κ during training and evaluation in Section 3.
Additional Implementation Details
Keypoint Definition Figure 10 illustrates the keypoints that we use in our experiments. The 3D poses used in our experiments is the 17 keypoints corresponding to the H3.6M [14] skeleton used in [26] , shown in Figure 10a . We use this keypoint definition to compute NP-MPJPE for 3D poses and evaluate retrieval accuracy. The Pr-VIPE training and inference process do not depend on a particular 2D keypoint detector. Here, we use PersonLab (ResNet152 single-scale) [32] in our experiments. Our 2D keypoints are selected from the keypoints in COCO [24] , which is the set of keypoints detected by PersonLab [32] . We use the 12 body keypoints from COCO and select the "Nose" keypoint as the head, shown in Figure 10b .
Pose Normalization We normalize our 2D and 3D poses such that camera parameters are not needed during training and inference. When we do projection, we assume a centered camera with a unit focal length. We adapt the normalization procedure in [6] to work with more keypoints so that the results are consistent even when the distance between some keypoints are small. For the 3D pose, we translate it so that the hip located at (0, 0, c). We then scale the hip to spine to head distance of the 3D pose to unit scale. When we project this 3D pose, the projected 2D pose has hip to head distance approximately Hyperparameter Value Hit@1 Hit@5 Hit@10 Hit@20 Table 3 : Additional ablation study results of Pr-VIPE on H3.6M with the number of samples K and margin parameter β.
1/c. We center the 2D pose between the LHip and RHip and normalize such that the maximum distance between shoulder and hip joints are 1/c. The maximum distance computed is between Rshoulder, LShoulder, RHip and LHip. We use c = 2 in our experiments.
Additional Ablation Studies
Effect of Number of Samples K and Margin Parameter β Table 3 shows the effect of the number of samples K and the margin parameter β (actual triplet margin α = log β) on Pr-VIPE. The number of samples control how many points we sample from the embedding distribution to compute matching probability and β controls the ratio of matching probability between matching and non-matching pairs. Our model is robust to the choice of β in terms of retrieval accuracy as shown by Table 3 . The main effect of β is on retrieval confidence, as non-matching pairs are scaled to a smaller matching probability for larger β. Pr-VIPE performance with 10 samples is competitive with the baselines in the main paper, but we do better with 20 samples. Increasing the number of samples further has similar performance. For our experiments, we use 20 samples and β = 2.
Effect of Keypoint Augmentation
We explore the effect of different random rotations during keypoint augmentation on pose retrieval results in Table 4 . All models are trained on the 4 chest-level cameras on H3.6M but the models with keypoint augmentation also use projected 2D keypoints from randomly rotated 3D poses. For the random rotation, we always use azimuth range of ±180 • , and we test performance with different angle limits for elevation and roll. We see that the model Effect of NP-MPJPE threshold κ We train and evaluate with different values of the NP-MPJPE threshold κ in Table 5 . κ controls the NP-MPJPE threshold for a matching pose pair and visualizations of pose pairs with different NP-MPJPE are in Figure 9 . Table 5 shows that Pr-VIPE generally achieves the best accuracy for a given NP-MPJPE threshold when the model is trained with the same matching threshold. Additionally, when we train with a tight threshold, i.e. κ = 0.05, we do comparatively well on accuracy at looser thresholds. In contrast, when we train with a loose threshold, i.e. κ = 0.20, we do not do as well given a tighter accuracy threshold. This is because when we push non-matching poses using the triplet ratio loss, κ = 0.20 only push poses that are more than 0.20 NP-MPJPE apart, and does not explicitly push poses less than the NP-MPJPE threshold. The closest retrieved pose will then be within 0.20 NP-MPJPE but it is not guaranteed to be within any threshold < 0.20 NP-MPJPE. But when we use κ = 0.05 for training, poses that are more than 0.05 NP-MPJPE are pushed apart, which also satisfies κ = 0.20 threshold.
In the main paper, we use κ = 0.1. For future applications with other matching definitions, the Pr-VIPE framework is flexible and can be trained with different κ to satisfy different accuracy requirements.
Additional Plots for Ordered Variances Similar to the main paper, we retrieve poses using 2D NP-MPJPE for the top-3 2D poses with smallest and largest variances in Figure 11 . Figure 11a shows that for the poses with the top-3 smallest variances, the nearest 2D pose neighbors are visually distinct, which means that these 2D poses are less ambiguous. On the other hand, the nearest 2D pose neighbors of the poses with the largest variances in Figure 11b are visually similar, which means that these 2D poses are more ambiguous.
PCA Visualization
We run Principal Component Analysis (PCA) on the 16-dimensional embeddings using the Pr-VIPE model. Figure 12 visualizes the first two principal dimensions and this is similar to Figure 3 in the main paper with more poses. To visualize more unique poses, we randomly subsample the H3.6M hold-out set and select 3D poses at least 0.1 NP-MPJPE apart. Figure 12 demonstrates that 2D poses from similar 3D poses are close together, while non-matching poses are further apart. 
Standing

Sitting

Arms Raised

Arms Down


{'paper_id': '208527516', 'title': 'View-Invariant Probabilistic Embedding for Human Pose.', 's2_url': 'https://api.semanticscholar.org/CorpusID:208527516', 'source': 'S2ORC'}