title: Monocular, One-stage, Regression of Multiple 3D People
authors: Yu Sun Qian Bao Wu Liu Yili Fu Michael J. Black Tao Mei
text: 
  
  
  
  
  
  
  
  
  
  
  Monocular, One-stage, Regression of Multiple 3D People
  
      Yu Sun^1This work was done when Yu Sun was an intern at JD AI Research.   Qian Bao^2   Wu Liu^2Corresponding author.   Yili Fu^1†  Michael J. Black^3   Tao Mei^2
  
  
  ^1Harbin Institute of Technology   ^2JD AI Research 
  
  ^3Max Planck Institute for Intelligent Systems, Tübingen, Germany
  
  
  
  
  
  
      December 30, 2023
  =====================================================================================================================================================================================================================================================================================================
  
  empty
  
  
  This paper focuses on the regression of multiple 3D people from a single RGB image. Existing approaches predominantly follow a multi-stage pipeline that first detects people in bounding boxes and then independently regresses their 3D body meshes. In contrast, we propose to Regress all meshes in a One-stage fashion for Multiple 3D People (termed ROMP). The approach is conceptually simple, bounding box-free, and able to learn a per-pixel representation in an end-to-end manner. Our method simultaneously predicts a Body Center heatmap and a Mesh Parameter map, which can jointly describe the 3D body mesh on the pixel level. Through a body-center-guided sampling process, the body mesh parameters of all people in the image are easily extracted from the Mesh Parameter map. Equipped with such a fine-grained representation, our one-stage framework is free of the complex multi-stage process and more robust to occlusion. Compared with state-of-the-art methods, ROMP achieves superior performance on the challenging multi-person benchmarks, including 3DPW and CMU Panoptic. Experiments on crowded/occluded datasets demonstrate the robustness under various types of occlusion. The released code[<https://github.com/Arthur151/ROMP>] is the first real-time implementation of monocular multi-person 3D mesh regression.
  
  
  
  
  
  
  § INTRODUCTION
   
  
  
  
  
  
  Recently, great progress has been made in monocular 3D human pose and shape estimation, particularly in images with a single person <cit.>.
  However, for more general scenes with multiple people, it is crucial to deal with truncation by the image frame, person-person occlusion, and environmental occlusion. 
  Robustness to such occlusions is critical for real-world applications.
  
  Existing approaches <cit.> follow a multi-stage design that equips the single-person method with a 2D person detector to handle multi-person scenes.
  Generally, they first detect regions with people and then extract the bounding-box-level features, which are used to regress each single 3D human mesh <cit.>. 
  However, as shown in Fig. <ref>, this strategy is prone to fail in cases of multi-person occlusion and truncation. 
  Specifically, as shown in Fig. <ref>(b), when two people overlap, it is hard for the multi-stage method to estimate diverse body meshes from similar image patches.
  The ambiguity of the implicit bounding-box-level representation results in failure for such inseparable multi-person cases. 
  
  
  For multi-person 2D pose estimation, this problem has been tackled via a subtle and effective bottom-up framework. 
  The paradigm first detects all body joints and then assigns them to different people by grouping body joints. 
  This pixel-level body-joint representation enables their impressive performance in crowded scenes <cit.>. 
  However, it is non-trivial to extend this bottom-up one-stage process beyond joints <cit.>.
  Unlike 2D pose estimation, which predicts dozens of body joints, we need to regress a human body mesh with thousands of vertices, making it hard to follow the paradigm of body joint detection and grouping.
  
  
  Instead, we introduce ROMP, a one-stage network for regressing multiple 3D people in a per-pixel prediction fashion.
  It directly estimates multiple differentiable maps from the whole image, from which we can easily parse out the 3D meshes of all people.
  Specifically, as shown in Fig. <ref>(c), ROMP predicts a Body Center heatmap and a Mesh Parameter map, representing the 2D position of the body center and the parameter vectors of the corresponding 3D body mesh, respectively.
  Via a simple parameter sampling process, we extract 3D body mesh parameter vectors of all people from the Mesh Parameter map at the body center locations described by the heatmap. 
  Then we put the sampled mesh parameter vectors into the SMPL body model <cit.> to derive multi-person 3D meshes.
  
  Following the guidance of body centers during training, the ambiguity of the regression target is greatly alleviated in crowded multi-person scenes. 
  Additionally, in contrast to the local, bounding-box-level, features learned by traditional methods, end-to-end learning from the whole image forces the model to learn appropriate features from the holistic scene to predict bodies with occlusion.
  This holistic approach captures the complexity of real-world scenes, enabling the generalization and robustness to complex multi-person cases.
  
  Moreover, since the body centers of severely overlapping people may collide at the same 2D position, we further develop an advanced collision-aware representation (CAR). 
  The key idea is to construct a repulsion field of body centers, where close body centers are analogous to positive charges that are pushed apart by mutual repulsion. In this way, the body centers of the overlapping people are more distinguishable. Especially in the case of severe overlap, most part of the human body is invisible. Mutual repulsion will push the center to the visible body area, meaning that the model tends to sample 3D mesh parameters estimated from the position centered on the visible body parts. It improves the robustness under heavy person-person occlusion. 
  
  Compared with previous state-of-the-art methods for multi-person <cit.> and single-person <cit.> 3D mesh regression, ROMP 
  achieves superior performance on challenging benchmarks, including 3DPW <cit.> and CMU Panoptic <cit.>. 
  Experiments on person-person occlusion datasets (Crowdpose <cit.> and 3DPW-PC, a person-occluded subset of 3DPW <cit.>) demonstrate the effectiveness of the proposed CAR under person-person occlusion.  
  To further evaluate it in general cases, we test ROMP on images from the Internet and web camera videos. 
  With the same backbone as the multi-stage counterparts, ROMP runs in real-time (over 30 FPS) on a 1070Ti GPU.
  
  In summary, the contributions are:
  (1) To the best of our knowledge, ROMP is the first one-stage method for monocular multi-person 3D mesh regression, along with an open-source real-time implementation. Its simple yet effective framework leads to superior accuracy and efficiency.
  (2) The proposed explicit body-center-guided representation facilitates the pixel-level human mesh regression in an end-to-end manner.
  (3) We develop a collision-aware representation to deal with cases of severe overlap.
  
  
  
  
  
  § RELATED WORK
  
  
  Single-person 3D mesh regression. 
  Parametric human body models, e.g., SMPL <cit.>, have been widely adopted to encode the complex 3D human mesh into a low-dimensional parameter vector, which can be regressed from images <cit.>.
  Impressive performance has been achieved using various weak supervision signals, such as 2D pose <cit.>, semantic segmentation <cit.>, 
  geometric priors and discriminative training <cit.>, 
  motion dynamics <cit.>, temporal coherence <cit.>, texture consistency <cit.>, optimization <cit.> in the loop <cit.>, etc.
  However, all these methods adopt a bounding-box-level representation, which is implicit and ambiguous for multi-person/occlusion cases.
  To handle occluding objects, Zhang et al. <cit.> use a 2D UV map to represent a 3D human mesh. 
  Considering the object-occluded body parts as blank areas in the partial UV map, they in-paint the partial UV map to make up the occluded information.
  However, in the case of person-person occlusion where one person's body parts are occluded by those of another, it is hard to generate the partial UV map.
  
  Multi-person 3D pose estimation. 
  Mainstream methods can be divided into two categories: the one-stage and the multi-stage.
  Many multi-stage methods follow the top-down design of Faster R-CNN <cit.>, such as LCR-Net++ <cit.> and 3DMPPE <cit.>. 
  From anchor-based feature proposals, they estimate the target via regression.
  Other works explore a one-stage solution that reasons about all people in a single forward pass. They estimate all the body joint positions and then group them into individuals.
  Mehta et al. <cit.> propose  occlusion-robust pose-maps and exploit the body part association to avoid the bounding box prediction.
  PandaNet <cit.> is an anchor-based one-stage model that relies on a huge number of pre-defined anchor predictions and the positive anchor selection. 
  To handle person-person occlusion, Zhen et al. <cit.> extend part affinity fields <cit.> to be depth-aware.
  ROMP extends the end-to-end one-stage process beyond joints through a concise body-center-guided pixel-level representation and does not require the part association or an enormous number of anchor predictions.
  
  Multi-person 3D mesh regression. 
  There are only a few approaches for multi-person 3D mesh regression. 
  Zanfir et al. <cit.> estimate the 3D mesh of each person from its intermediate 3D pose estimation.
  Zanfir et al. <cit.> further employ multiple scene constraints to optimize the multi-person 3D mesh results. 
  Jiang et al. <cit.> propose a network for Coherent Reconstruction of Multiple Humans (CRMH).
  Built on Faster-RCNN <cit.>, they use the RoI-aligned feature of each person to predict the SMPL parameters. 
  Additionally, they learn the relative positions between multiple people via an interpenetration and depth ordering loss.
  All these methods follow a multi-stage design.
  The complex multi-step process requires a repeated feature extraction, which is computationally expensive. 
  Moreover, since they rely on detected bounding boxes, the ambiguity and the limited local view of the bounding-box-level features make it hard to effectively learn from person-person occlusion and truncation.
  Instead, our proposed one-stage method learns an explicit pixel-level representation with a holistic view, which significantly improves both accuracy and efficiency in multi-person in-the-wild scenes. 
      
   Pixel-level representations have proven useful in anchor-free detection, such as CornerNet <cit.>, CenterNet <cit.>, and ExtremeNet <cit.>.
  They directly estimate the corner or center point of the bounding box in a heatmap manner, which avoids the dense anchor-based proposal.
  Inspired by these, we develop a pixel-level fine-grained representation for multi-person 3D meshes.
  Different from the bounding box center used in <cit.>, our body center is determined by the body joints, as introduced in Sec. <ref>. 
  A recent work, BMP <cit.>, uses a multi-scale grid-level representation for multi-person 3D mesh recovery, which locates a target person at the center of the grid cell.[The arXiv version of ROMP, called CenterHMR <cit.>, predates BMP.]
  In contrast to these methods, ROMP adopts a concise body-center-based representation and further evolves it to a collision-aware version to deal with the inherent center collision problem. 
  
  
  Disambiguation is a key goal of ROMP, enabling it to  deal with the crowded multi-person scenes. Related techniques have been studied in many other fields. For instance segmentation, Adaptis <cit.> separately learns the segmentation mask of each instance selected by the guide point. 
  To alleviate the ambiguity between embeddings of similar samples, associate embedding <cit.>, triplet loss <cit.>, and pose-guided association <cit.> are developed for pose estimation, face recognition, and tracking respectively. 
  In this paper, a robust and distinguishable representation is developed to help the model explicitly learn from the crowded scenes.
  
  
  
  
  
  
  § OUR APPROACH
  
  
  
  
  
  
   §.§ Overview
  
  
  The overall framework is illustrated in Fig. <ref>.
  It adopts a simple multi-head design with a backbone and three head networks.
  Given a single RGB image as input, it outputs a Body Center heatmap, Camera map, and SMPL map, describing the detailed information of the estimated 3D human meshes. 
  In the Body Center heatmap, we predict the probability of each position being a human body center. 
  At each position of the Camera/SMPL map, we predict the camera/SMPL parameters of the person that takes the position as the center. 
  For simplicity, we combine the Camera map and SMPL map into the Mesh Parameter map. 
  During inference, we sample the 3D body mesh parameter results from the Mesh Parameter map at the 2D body center locations parsed from the Body Center heatmap. 
  Finally, we put the sampled parameters into the SMPL model to generate the 3D body meshes. 
  
  
  
   §.§ Basic Representations 
  
  
  In this section, we introduce the detailed representation of each map. Each output map is of size n × H × W, where n is the number of channels. Here, we set H=W=64. 
  
  Body Center heatmap: C_m∈ℝ^1 × H × W is a heatmap representing the 2D human body center in the image. 
  Each body center is represented as a Gaussian distribution in the Body Center heatmap. 
  For better representation learning, the Body Center heatmap also integrates the scale information of the body in the 2D image.
  Specifically, we calculate the Gaussian kernel size k of each person center in terms of its 2D body scale in the image. 
  Given the diagonal length d_bb of the person bounding box and the width W of the Body Center heatmap, k is derived as
  
      k = k_l+( d_bb/√(2)W)^2 k_r ,
  
  where k_l is the minimum kernel size and k_r is the variation range of k. We set k_l=2 and  k_r=5 by default.
  
  Mesh Parameter map: P_m∈ℝ^145 × H × W consists of two parts, the Camera map and SMPL map.
  Assuming that each location of these maps is the center of a human body, we estimate its corresponding 3D body mesh parameters.
  Following <cit.>, we employ a weak-perspective camera model to project K 3D body joints J=(x_k,y_k,z_k), k=1⋯ K of the estimated 3D mesh back to the 2D joints J=(x_k,y_k) on the image plane. 
  This facilitates training the model with in-the-wild 2D pose datasets, which helps with robustness and generalization. 
   
  Camera map: A_m∈ℝ^3 × H × W contains the 3-dim camera parameters (s, t_x, t_y) that describe the 2D scale s and translation t=(t_x, t_y) of the person in the image. 
  The scale s reflects the body size and the depth to some extent.
  t_x and t_y, ranging in (-1,1), reflect the normalized translation of the human body relative to the image center on the x and y axis, respectively.
  The 2D projection J of  3D body joints J can be derived as  x_k = s x_k+t_x, y_k = s y_k+t_y.
  The translation parameters allow more accurate position estimates than the Body Center heatmap. 
  
  SMPL map: S_m∈ℝ^142 × H × W  contains the 142-dim SMPL parameters, which describe the 3D pose and shape of the body mesh. 
  SMPL establishes an efficient mapping from the pose  θ  and shape β parameters to the human 3D body mesh M∈ℝ^6890 × 3.
  The shape parameter β∈ℝ^10 is the top-10 PCA coefficients of the SMPL statistical shape space. 
  The pose parameters θ∈ℝ^6 × 22 contain the 3D rotation of the 22 body joints in a 6D representation <cit.>.  
  Instead of using the full 24 joints of the original SMPL model, we drop the last two hand joints. 
  The 3D rotation of the first joint denotes the body 3D orientation in the camera coordinate system, 
  while the remainder are the relative 3D orientations of each body part with respect to its parent in a kinematic chain. 
  3D joints J are derived via PM, where P∈ℝ^K × 6890 is a sparse weight matrix that describes the linear mapping from 6890 vertices of the body mesh M to the K body joints.
  
  
  
   §.§ CAR: Collision-Aware Representation 
  
  
  
  
  The entire framework is based on a concise body-center-guided representation. 
  It is crucial to define an explicit and robust body center so that the model can easily estimate the center location in various cases. 
  Here we introduce the basic definition of the body center for the general case and its advanced version for severe occlusion.
  
  Basic definition of the body center. Existing center-based methods <cit.> define the center of the bounding box as the target center. 
  This works well for general objects (e.g., a ball or bottle) that lack semantically meaningful keypoints.
  However, a bounding box center is not a meaningful point on the human body and often falls outside the body area.
  For stable parameter sampling, we need an explicit body center.
  Therefore, we calculate each body center from the ground truth 2D pose. 
  Considering that any body joint may be occluded in general cases, we define the body center as the center of visible torso joints (neck, left/right shoulders, pelvis, and left/right hips). 
  When all torso joints are invisible, the center is simply determined by the average of the visible joints.
  In this way, the model is encouraged to predict the body location from the visible parts. 
  
  However, in cases of severely overlapping people, the body center of the people might be very close or even at the same location on C_m.
  This center collision problem makes the center ambiguous and hard to identify in crowded cases.
  To tackle this, we develop a more robust representation to deal with person-person occlusion.
  To alleviate the ambiguity, the center points of overlapping people should be kept at a minimum distance to ensure that they can be well distinguished. 
  Additionally, to avoid sampling multiple parameters for the same person, the network should assign a unique and explicit center for each person.
  
  Based on these principles, we develop a novel Collision-Aware Representation (CAR). 
  To ensure that the body centers are far enough from each other, we construct a repulsion field. 
  In this field, each body center is treated as a positive charge, whose radius of repulsion is equal to its Gaussian kernel size derived by Eq. (<ref>).
  In this way, the closer the body centers are, the greater the mutual repulsion and the further they will be pushed apart. 
  Fig. <ref> illustrates the principle of CAR.  
  Suppose that c_1∈ℝ^2, c_2∈ℝ^2 are the  body centers of two overlapping people. 
  If  their Euclidean distance d_cm and Gaussian kernel sizes k_1, k_2 satisfy  d_cm<k_1+k_2+1, the repulsion is triggered to push the close centers apart  via
  
      ĉ_1 = c_1 + γd_p, ĉ_2 = c_2 - γd_p,
         d_p = k_1+k_2+1 - d_cm/d_cm  (c_1-c_2),
  
  where d_p∈ℝ^2 is the repulsion vector from c_2 to c_1 and γ is an intensity coefficient to adjust the strength. When there are  multiple overlapped people, we take Eq. (<ref>) to generate the mutual repulsion vectors d^i_p for the i-th pair of centers. For the center that is affected by N  repulsive forces, we calculate the composition of these forces as the numerical summation ∑_i=1^Nd^i_p.
  
  
  During training, we use CAR to push apart close body centers and use them to supervise the Body Center heatmap. 
  In this way, the model is encouraged to estimate the  centers that maintain a distinguishable distance.
  For the Body Center heatmap, it helps the model to effectively locate the occluded person.
  For the Mesh Parameter map, sampling the parameters from these shifted locations enables the model to extract diverse and individual features for each person.
  The model trained with CAR is more suitable for crowded scenes with significant person-person occlusion such as train stations, canteens, etc.
  
  
  
  
   §.§ Parameter Sampling 
  
  
  
  To parse the 3D body meshes from the estimated maps, we need to first parse the 2D body center coordinates c∈ℝ^K × 2 from C_m, where K is the number of the detected people, and then use them to sample P_m for the SMPL parameters. 
  In this section, we introduce the process of the center parsing, matching and sampling. 
  
  C_m is a probability map whose local maxima are regarded as the body centers. 
  The local maxima are derived via Mp(C_m) C_m where Mp is the max pooling operation and  is the logical conjunction operation. 
  Let c be the 2D coordinates of a local maximum with confidence score larger than a threshold t_c. 
  We rank the confidence score at each c and take the top N as the final centers.
  During inference, we directly sample the parameters from P_m at c. 
  During training, the estimated c are matched with the nearest ground truth body center according to the L_2 distance. 
  
  Additionally, we approximate the depth order between multiple people by using the center confidence from C_m and the 2D body scale s of the camera parameters from A_m.
  For people of different s, we regard the one with the larger s as located in the front.
  For people of similar s, the person with a higher center confidence is considered to be in the front. Please refer to the SuppMat for the details.
  
  
  
  
   §.§ Loss Functions
  
  
  To supervise ROMP, we develop individual loss functions for different maps.
  In total, ROMP is supervised by the weighted sum of the body center loss L_c and mesh parameter loss L_p.
  
  Body Center loss. L_c encourages a high confidence value at the body center c of the Body Center heatmap C_m and low confidence elsewhere.
  To deal with the imbalance between the center location and the non-center locations in C_m, we train the Body Center heatmap based on the focal loss <cit.>. 
  Given the predicted Body Center heatmap C_m^p and the ground truth C_m^gt,  L_c is defined as 
  
      L_c = - L_pos+L_neg/∑I_pos w_c,
         L_neg = log(1-C_m^p)(C_m^p)^2 (1-C_m^gt)^4 (1-I_pos),
         L_pos = log(C_m^p)(1-C_m^p)^2 I_pos,  I_pos=C_m^gt≥1,
  
  where I_pos  is a binary matrix with a positive value at the body center location, and w_c is the loss weight.
  
  Mesh Parameter loss. 
  
  As we introduced in Sec. <ref>, the parameter sampling process matches each ground truth body with a predicted parameter result for supervision.
  The mesh parameter loss is derived as 
  
      L_p = w_poseL_pose+w_shapeL_shape+w_j3dL_j3d
      +w_paj3dL_paj3d+w_pj2dL_pj2d+w_priorL_prior.
  
  L_pose is the L_2 loss of the pose parameters in the 3 × 3 rotation matrix format. 
  L_shape is the L_2 loss of the shape parameters. 
  L_j3d is the L_2 loss of the 3D joints J regressed from the body mesh M.
  L_paj3d is the L_2 loss of the 3D joints J after Procrustes alignment.
  L_pj2d is the L_2 loss of the projected 2D joints J. 
  L_prior is the Mixture Gaussian prior loss of the SMPL parameters adopted in <cit.> for supervising the plausibility of 3D joint rotation and body shape. Lastly, w_(.) denotes the corresponding loss weights.
  
  
  
  
  
  
  
  
  
  
  
  § EXPERIMENTS
  
  
  
  
  
   §.§ Implementation Details
  
  
  
  Network Architecture. 
  For a fair comparison with other approaches, we use ResNet-50 <cit.> as the default backbone.
  Since our method is not limited to a specific backbone, we also test HRNet-32 <cit.> in the experiments.
  Through the backbone, a feature vector f_b∈ℝ^32 × H_b × W_b is extracted from a single RGB image. 
  Also, we adopt the CoordConv <cit.> to enhance the spatial information.
  Therefore, the backbone feature f∈ℝ^34 × H_b × W_b is the combination of a coordinate index map ci∈ℝ^2 × H_b × W_b and f_b.
  Next, from f, three head networks are developed to estimate the Body Center, Camera, and SMPL maps. 
  More details of the architecture are in the SuppMat.
  
  Setting Details. 
  The input images are resized to 512 × 512, keeping the same aspect ratio and padding with zeros. 
  The size of the backbone feature is H_b=W_b=128. 
  The maximum number of detections, N=64, which is set manually. 
  The loss weights are set to w_c=160,w_j3d=360,w_paj3d=400,w_pj2d=420,w_pose=80,w_shape=1, and w_prior=1.6 to ensure that the weighted loss items are of the same magnitude.
  The threshold t_c of the Body Center heatmap is 0.2. The repulsion coefficient γ of CAR is 0.2. 
  
  Training Datasets.
  For a fair comparison with previous methods <cit.>, the basic training datasets we used in the experiments include two 3D pose datasets (Human3.6M <cit.> and MPI-INF-3DHP <cit.>), one pseudo-label 3D dataset (UP <cit.>) and four in-the-wild 2D pose datasets (MS COCO <cit.>, MPII <cit.>, LSP <cit.> and AICH <cit.>). 
  We also use the pseudo 3D annotations from <cit.>. 
  To further explore the upper limit of performance, we also use additional training datasets, including two 3D pose datasets (MuCo-3DHP <cit.> and OH <cit.>), the pseudo 3D labels of 2D pose datasets provided by <cit.>, and two 2D pose datasets (PoseTrack <cit.> and Crowdpose <cit.>), to train an advanced model.
  
  Evaluation Benchmarks. 
  3DPW <cit.> is employed as the main benchmark for evaluating 3D mesh/joint error since it contains in-the-wild multi-person videos with abundant 2D/3D annotations.
  Specially, we divide 3DPW into 3 subsets, including 3DPW-PC for person-person occlusion, 3DPW-OC for object occlusion, and 3DPW-NC for the non-occluded/truncated cases, to evaluate the performance in different scenarios.
  Additionally, we also evaluate on a indoor multi-person 3D pose benchmark, CMU Panoptic <cit.>. 
  Furthermore, we evaluate the stability under occlusion on  Crowdpose <cit.>, a crowded-person in-the-wild 2D pose benchmark.
  
  Evaluation Metrics.
  We adopt per-vertex error (PVE) to evaluate the 3D surface error.
  To evaluate the 3D pose accuracy, we employ mean per joint position error (MPJPE), Procrustes-aligned MPJPE (PMPJPE), percentage of correct keypoints (PCK), and area under the PCK-threshold curve (AUC).
  We also adopt mean per joint angle error (MPJAE), and Procrustes-aligned MPJAE (PA-MPJAE) to evaluate the 3D joint rotation accuracy. 
  Also, to evaluate the pose accuracy in crowded scenes, we calculate the average precision (AP^0.5) between the 2D-projection J and the ground truth 2D poses on Crowdpose.
  
  
  
  
  
  
  
  
  
  
  
  
   §.§ Comparisons to the State-of-the-Art 
  
  
  3DPW. 
  We adopt three evaluation protocols, which reveal different properties.
  To validate the performance in actual scenes, we follow Protocol 1 from the 3DPW Challenge to evaluate on the entire 3DPW dataset without using any ground truth, especially the bounding box. 
  With the whole image as input, we equip each single-person method <cit.> with a human detector (OpenPose <cit.> or YOLO <cit.>).
  For a fair comparison, ROMP uses the same backbone (ResNet-50) and training datasets as the competing method <cit.>.
  We obtain the results of  OpenPose + SPIN from <cit.>. 
  The results of YOLO + VIBE are obtained using the officially released code, which already contains the YOLO part for human detection.
  The results of BMP, which adopts a multi-scale grid-level representation, are obtained from <cit.>.
  In Tab. <ref>, ROMP significantly outperforms all these methods, particularly in MPJPE, PMPJPE, and MPJAE.
  These results validate that learning a robust pixel-level representation with a holistic view is helpful for improving the robustness and generalization in actual scenes.
  Training with extra datasets (*) shows that the accuracy of ROMP can be further improved.
  
  As a sanity check, we also compare ROMP with the single-person approaches in the evaluation protocols that allow them to use the cropped single-person image as input, while ROMP still takes the whole image as input.
  Following VIBE <cit.>, Protocol 2 uses the 3DPW test set for evaluation without fine-tuning on the training set, while Protocol 3 fine-tunes the model on the 3DPW training set and uses the test set for evaluation.
  In Tab. <ref>, ROMP outperforms these multi-stage approaches on Protocol 2, further demonstrating the advantage of our one-stage design.
  In Tab. <ref>, ROMP achieves comparable results with the state-of-the-art methods. 
  If we use HRNet-32 as the backbone, the accuracy improves significantly after fine-tuning. 
  
  
  
  
  
  
  
  
  
  CMU Panoptic. 
  Following the evaluation protocol of CRMH <cit.>, we evaluate ROMP on the multi-person benchmark, CMU Panoptic, without any fine-tuning.
  For a fair comparison, we use the same backbone and similar training set as CRMH.
  As shown in Tab. <ref> , ROMP outperforms the existing multi-stage  methods <cit.> in all activities by a large margin.
  These results further demonstrate that learning pixel-level representation with a holistic view improves the performance on multi-person scenes.
  
  
  
  Occlusion benchmarks.
  To validate the stability under occlusion, we evaluate ROMP on multiple occlusion benchmarks.
  Firstly, on the person-occluded 3DPW-PC and Crowdpose <cit.>, results in Tab. <ref> and <ref> show that ROMP significantly outperforms previous state-of-the-art methods <cit.>.
  Additionally, in Fig. <ref>, some qualitative comparisons to 
  CRMH also demonstrate ROMP's robustness to person-person occlusion.
  These results suggest that the pixel-level representation is important for improving the performance under person-person occlusion.
  Finally, on the object-occluded 3DPW-OC, ROMP also achieves promising performance.
  These results demonstrate that the fine-grained pixel-level representation is beneficial for dealing with various occlusion cases.
  
  Runtime comparisons. 
  We compare ROMP with the state-of-the-art methods in processing videos captured by a web camera.
  All runtime comparisons are performed on a desktop with a GTX 1070Ti GPU, Intel i7-8700K CPU, and 8 GB RAM.
  As shown in Tab. <ref>, ROMP achieves real-time performance, significantly faster than the competing methods.
  Additionally, as shown in Fig. <ref>, compared with the multi-stage methods <cit.>, ROMP's processing time is roughly constant regardless of the number of people. 
  
  
  
  
  
   §.§ Ablation Study of the CAR
  
  
  
  As shown in Tab. <ref> and <ref>, CAR improve the PMPJPE metric on the 3DPW-PC and the Crowdpose datasets by 4.8% and 10.3%, respectively.
  Additionally, Fig. <ref> shows, qualitatively, the impact of ablating CAR.
  Adding CAR improves  performance in  crowded scenes, which demonstrates that CAR effectively alleviates the center collision problem.
  
  Intensity coefficient γ of the CAR. 
  To set γ, we conduct an ablation study on 3DPW-PC.
  In Tab. <ref>, setting γ=0.2 performs much better on the crowded scenes (3DPW-PC) and its performance on the normal cases (3DPW-NC/3DPW-OC) is comparable to the best. 
  For general in-the-wild cases of Crowdpose, setting γ=0.2 improves the performance by 10%  over γ=0 in Tab. <ref>.
  Therefore, we suggest training the model with γ=0.2 for all cases.
  The reason performance degrades in the normal cases is probably that pushing apart the body centers affects the consistency of the body-center-guided representation.
  
  
  
  
  
  
   §.§ Discussion
  
  
  
  To understand the source of our performance gains, we conduct an ablation study on different subsets of 3DPW subsets. 
  Results in Tab. <ref> show that our main gains come from the person-occluded and the non-occluded/truncated cases.
  It demonstrates the effectiveness of the proposed pixel-level representation in improving the disambiguation in the crowded scenes.
  Our experiments suggest that the difference between ROMP and the state-of-the-arts <cit.> is the method of representation learning. 
  ROMP learns the pixel-level representation from a holistic view, while the multi-stage methods learn a bounding-box-level representation in a local view.
  Our one-stage framework enables ROMP to learn more discriminative features that are robust to rich disturbances outside the bounding box, helping generalization. 
  
  
  
  
  
  § CONCLUSION
  
  
  
  We introduce a novel one-stage network, ROMP, for monocular multi-person 3D mesh regression. 
  For pixel-level estimation, we propose an explicit body-center-guided representation and further develop it as a collision-aware version, CAR, enabling robust prediction under person-person occlusion. 
  ROMP is the first open-source one-stage method that achieves state-of-the-art performance on multiple benchmarks as well as real-time inference speed.
  ROMP can serve as a simple yet effective foundation for related multi-person 3D tasks, such as depth estimation, tracking, and interaction modeling. 
  
  Acknowledgements: This work was supported by the National Key R&D Program of China under Grand No. 2020AAA0103800. 
  We thank Peng Cheng for discussing the Center map training.
  
  
  Disclosure: MJB has received research funds
  from Adobe, Intel, Nvidia, Facebook, and Amazon. While MJB
  is a part-time employee of Amazon, his research was performed
  solely at Max Planck. MJB has financial interests in Amazon, Datagen Technologies, and Meshcapade
  GmbH.
  
  packages/ieee_fullname
  
  
  
  
  [pages=1]SuppMat.pdf
  [pages=2]SuppMat.pdf
  [pages=3]SuppMat.pdf
  [pages=4]SuppMat.pdf
  [pages=5]SuppMat.pdf
  [pages=6]SuppMat.pdf
  [pages=7]SuppMat.pdf
  
  