SMPLer: Taming Transformers for Monocular 3D Human Shape and Pose Estimation
Xiangyu Xu Lijuan Liu Shuicheng Yan

	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	IEEE Transactions on Pattern Analysis and Machine Intelligence
	
	
	
	
	
	
	
	
		
			
			Existing Transformers for monocular 3D human shape and pose estimation typically have a quadratic computation and memory complexity with respect to the feature length, which hinders the exploitation of fine-grained information in high-resolution features that is beneficial for accurate reconstruction.
			
			
			In this work, we propose an SMPL-based Transformer framework (SMPLer) to address this issue.
			
			SMPLer incorporates two key ingredients: a decoupled attention operation and an SMPL-based target representation, which allow effective utilization of high-resolution features in the Transformer.
			In addition, based on these two designs, we  also introduce several novel modules including a multi-scale attention and a joint-aware attention to further boost  the reconstruction performance.
			
			
			Extensive experiments demonstrate the effectiveness of SMPLer against existing 3D human shape and pose estimation methods both quantitatively and qualitatively.
			Notably, the proposed algorithm achieves an MPJPE of 45.2 mm on the Human3.6M dataset, improving upon Mesh Graphormer by more than 10% with fewer than one-third of the parameters.
			Code and pretrained models are available at <https://github.com/xuxy09/SMPLer>.
		
		
		
		
			3D human shape and pose, Transformer, attention, multi-scale, SMPL, joint-aware
	
	
	
	
	SMPLer: Taming Transformers for Monocular 3D Human Shape and Pose Estimation
    Xiangyu Xu,  
		Lijuan Liu,  Shuicheng Yan
		
			X. Xu is with Xi'an Jiaotong University, Xi'an, China. E-mail: xuxiangyu2014@gmail.com.
			L. Liu is with Sea AI Lab, Singapore. E-mail: liulj@sea.com.
			S. Yan is with Skywork AI, Singapore. E-mail: shuicheng.yan@gmail.com.
			
		
		
	
    Received xx; accepted xx
==================================================================================================================================================================================================================================================================================================


	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
		

§ INTRODUCTION

	
	
	
	
	
	
	Monocular 3D human shape and pose estimation is a fundamental task in computer vision, which aims to recover the unclothed human body shape as well as its 3D pose from a single input image <cit.>. 
	
	It has been widely used in many applications including visual tracking <cit.>, virtual/augmented reality <cit.>, motion generation <cit.>, image manipulation <cit.>, and neural radiance field <cit.>.
	Different from multi-view 3D reconstruction where the solution is well restricted by geometrical constraints <cit.>,  
	this task is particularly challenging due to the inherent depth ambiguity of a single 2D image, and usually requires strong prior knowledge learned from large amounts of data to generate plausible results.
	
	Thus, the state-of-the-art algorithms <cit.> use Transformers <cit.> for this task due to their powerful capabilities of grasping knowledge and learning representations from data <cit.>. 
	
	
	
	Existing Transformers for monocular 3D human shape and pose estimation <cit.> generally follow the ViT style <cit.> to design the network.
	As shown in Figure <ref>(a), the target embeddings are first concatenated with the input features and then processed by a full attention layer that models all pairwise dependencies including target-target, target-feature, feature-target, and feature-feature.
	
	While this design has achieved impressive results, it leads to quadratic computation and memory complexity with respect to the length of the image feature, 𝒪((l_F+l_T)^2),
	
	where l_F and l_T are the numbers of feature and target tokens, respectively (or simply the lengths of the image feature F and the target embedding T). 
	
	Such complexity is prohibitive for a large l_F, hindering the existing methods from employing high-resolution image features in Transformers, which possess abundant fine-grained features <cit.> that are beneficial for accurate 3D human shape and pose recovery.
	
	
	
	
	
	
	In this work, we propose two strategies to improve the Transformer framework to better exploit higher-resolution image features for high-quality 3D body shape and pose reconstruction.
	
	One of them is attention decoupling.
	We notice that different from the original ViT <cit.> where the image features are learned by attention operations, the 3D human Transformers <cit.> usually rely on Convolutional Neural Networks (CNNs) to extract these features, and the attention operations are mainly used to aggregate the image features to improve the target embeddings. 
	
	Thus, it is less important to model the feature-feature and feature-target correlations in Figure <ref>(a), as they do not have direct effects on the target. 
	We therefore propose to decouple the full attention operation into a target-feature attention and a target-target attention, which are cascaded together to avoid the quadratic complexity.
	
	As shown in Figure <ref>(b), the decoupled attention only has a computation and memory complexity of 𝒪(l_F l_T+l_T^2) which is linear with respect to the feature length l_F.
	
	
	The other strategy we propose for improving the previous 3D human Transformer is SMPL-based target representation.
	
	Existing Transformers for 3D human shape and pose estimation mostly adopt a vertex-based target representation <cit.>, where the embedding length l_T is often quite large, equaling the number of vertices on a body mesh. 
	Even with the proposed attention decoupling strategy, a large l_T can still bring a considerable cost of computation and memory, which hinders the usage of high-resolution features.
	
	
	To address this issue, we introduce a new target representation based on the parametric body model SMPL <cit.>, with which we only need to learn a small number of target embeddings that account for the human body shape as well as 3D body part rotations.
	
	As illustrated in Figure <ref>(c), the SMPL-based representation 𝒯 further lessens the computation and memory burden compared to the vertex-based representation with l_𝒯≪ l_T.
	
	
	
	
	Combining the above two strategies leads to a much more concise attention learning process, which not only allows the utilization of high-resolution features in the Transformer, but also motivates us to explore more new designs for better 3D human shape and pose estimation.
	
	In particular, enabled by the lifted efficiency of our model, we introduce a multi-scale attention operation that effectively exploits the multi-scale information in a simple and  unified framework.
	
	Further, as the proposed target representation explicitly describes 3D relative rotations between body parts which are mostly local, we further propose a joint-aware attention module that emphasizes local regions around body joints to better infer the articulation status of the 3D human.
	
	
	
	
	
	
	To summarize, we make the following contributions: 
	
		
		
  ∙ By introducing the attention decoupling and SMPL-based target representation, we propose a new Transformer framework, SMPLer, that can exploit a large number of feature tokens for accurate and efficient 3D human shape and pose estimation. 
		
		
		
		
  ∙  
		
		
		Based on the two key designs above, we further develop multi-scale attention and joint-aware attention modules which significantly improve the reconstruction results. 
		
		
		
		
  ∙ Extensive experiments demonstrate that SMPLer performs favorably against the baseline methods  with better efficiency.
		In particular, compared to the state-of-the-art method <cit.>, the proposed algorithm lowers the MPJPE error by more than 10% on Human3.6M <cit.> with fewer than one-third of its parameters, showing the effectiveness of the proposed algorithm.
	
	
	
	
	
	

§ RELATED WORK

	We first review the previous 
	methods for 3D human pose and shape estimation and then discuss recent advances in vision Transformers.
	
	

 §.§ 3D Human Shape and Pose Estimation

	Recent years have witnessed significant progress in the field of monocular 3D human shape and pose estimation <cit.>.
	
	Due to the intrinsic ambiguity of 2D-to-3D mapping, this problem 
	
	is highly ill-posed and requires strong prior knowledge learned from large datasets to regularize the solutions.
	
	
	Indeed, the progress of 3D human shape and pose estimation largely relies on the development of powerful data-driven models.
	
	
	As a pioneer work, SMPLify <cit.> learns a Gaussian Mixture model from CMU marker data <cit.> to encourage plausible 3D poses, which however needs to conduct inference in a time-consuming optimization process.
	
	To address this low efficiency issue, more recent methods use deep learning models to directly regress the 3D human mesh in an end-to-end fashion, which have shown powerful capabilities in absorbing prior knowledge and discovering informative patterns from a large amount of data.
	
	Typically, GraphCMR <cit.> trains Graph Neural Networks (GNNs) for 3D human shape and pose estimation, which directly learns the vertex location on human mesh and is prone to outliers and large errors under challenging poses and cluttered backgrounds.
	
	In contrast, some other methods, such as SPIN <cit.> and RSC-Net <cit.>, circumvent this issue by training deep CNNs to estimate the underlying SMPL parameters, which have demonstrated robust performance for in-the-wild data, inspiring numerous new applications <cit.>.
	
	
	Nevertheless, recent research in the machine learning community <cit.> reveals shortcomings of CNNs that they have difficulties in modeling long-range dependencies, which limits their capabilities for higher-quality representation learning.
	
	To overcome the above issue, METRO <cit.> and Mesh Graphormer <cit.> introduce Transformers to this task, significantly improving the reconstruction accuracy.
	
	However, these Transformer-based methods generally follow the architecture of ViT <cit.> without considering the characteristics of 3D human shape and pose, and adopt a straightforward vertex representation, which hinders the network from exploiting high-resolution features for better performance.
	
	In contrast, we propose a decoupled attention formulation that is more suitable for accurate 3D human reconstruction with a large number of feature tokens.
	To better realize the concept of attention decoupling, we develop a new Transformer framework that can effectively exploit multi-scale (high and low) and multi-scope (global and local) information for better results.
	Moreover, we show that the parametric SMPL model can be combined with Transformers by introducing an SMPL-based target representation, which naturally addresses issues of vertex regression.
	
	
	

 §.§ Transformers

	
	With its remarkable capability of representation learning, the Transformer has become a dominant architecture in natural language processing <cit.>.  
	Recently, it has been introduced in computer vision, and the applications include object detection <cit.>, image classification <cit.>, image restoration <cit.>, video processing <cit.>, and generative models <cit.>. 
	Most of these works employ an attention operation that has quadratic complexity regarding the feature length and thus is not suitable for our goal of exploiting higher-resolution features for more accurate 3D human shape and pose estimation.
	
	
	In this work, we demonstrate the effectiveness of a decoupled attention mechanism that has linear complexity with respect to the feature length. 
	We also introduce a compact target representation based on a parametric human model.
	Furthermore, we provide new insights in designing multi-scale and joint-aware attention operations for 3D human reconstruction, and the proposed network achieves better performance than the state-of-the-art Transformers with improved efficiency. 
	
	
	
	
	
		
	
	
	

§ METHODOLOGY

	
	
	In this work, we propose a new 
	
	SMPL-based Transformer framework (SMPLer) for 3D human shape and pose estimation,
	which can exploit a large number of image feature tokens
	
	based on an efficient decoupled attention and a compact target representation.
	
	An overview is shown in Figure <ref>.
	
	

 §.§ Efficient Attention Formulation

	
	The attention operation <cit.> is the central component of Transformers, which can be formulated as:
	
    h(Q,K,V) = f_soft( (Q  W_q)  (K   W_k)^⊤/√(d)) (V   W_v),

	where f_soft is the softmax function along the row dimension. Q∈ℝ^l_Q × d, and K, V∈ℝ^l_K × d are the input of this operation, representing query, key, and value, respectively.
	
	l_Q is the length of the query Q, and l_K is the length of K and V.
	d corresponds to the channel dimension.
	
	W_q,W_k,W_v∈ℝ^d × d represent learnable linear projection weights.
	
	We use a multi-head attention <cit.> in our  implementation, where each head follows the formulation in Eq. <ref>, and different heads employ different linear projection weights.
	
	Similar to prior work <cit.>,  we also use layer normalization <cit.> and MLP in the attention operation, which are  omitted in Eq. <ref> for brevity.
	
	
	Essentially, Eq. <ref> models dependencies between pairs of tokens from Q and K with dot product. 
	The output h(Q,K,V) ∈ℝ^l_Q × d can be seen as a new query embedding enhanced by aggregating information from V, and the aggregation weights are decided by the dependencies between Q and K.
	
	Noticeably, when query, key and value are the same, Eq. <ref> is called self-attention which we denote as h_self(Q)=h(Q,Q,Q).
	When only key and value are the same while query is different, the operation becomes cross-attention, denoted as h_cross(Q,K)=h(Q,K,K).
	
	
	
	
	

  §.§.§ Full Attention

	Existing Transformers for 3D human reconstruction <cit.> basically follow a ViT-style structure <cit.>, which adopts the full attention formulation as shown in Figure <ref>(a).
	Mathematically, the full attention can be written as:
	
    h_self(T ‖ F),

	where the self-attention h_self(Q) is used, and the Q is  a concatenation of the target embedding T and the image feature F along the token dimension, denoted by T‖ F ∈ℝ^(l_T+l_F)× d. 
	
	The target embedding represents the variable of interest, which corresponds to the class token in ViT and the  3D human representation in this work (see more explanations in Section <ref>).
	
	As the image feature F in Eq. <ref> is involved in both query and key[This can be seen by replacing Q,K,V in Eq. <ref> with T‖ F.],  the full attention leads to a quadratic computation and memory cost with respect to the length of the image feature l_F, 𝒪((l_F+l_T)^2),. 
	
	Consequently, previous works <cit.> only use low-resolution features in the Transformer where l_F is small.
	
	In particular, suppose different resolution features from a CNN backbone are denoted as ℱ={F_1, ⋯, F_S} (see Figure <ref>) where S is the number of scales[The resolution decreases from scale 1 to S.], 
	Mesh Graphormer <cit.> only uses F_S in the attention operation, and straightforwardly including higher-resolution features in Eq. <ref>, , F_1, would be computationally prohibitive.
	
	
	
	
	

  §.§.§ Decoupled Attention

	
	
	We notice that the attention operation serves different roles in ViT <cit.> and the Transformers for 3D human shape and pose estimation <cit.>:
	
	the original ViT relies solely on attention to learn expressive image features, while the 3D human Transformer uses a deep CNN for feature extraction (Figure <ref>), and the attention is mainly used to aggregate the image features for improving the target embeddings.
	
	
	Therefore, modeling feature-feature dependencies is less important here as it does not have a direct effect on the target, implying that it is possible to avoid the quadratic complexity by pruning the full attention.
	
	
	Motivated by this observation, we propose a decoupled attention that bypasses the feature-feature computations as shown in Figure <ref>(b). 
	It is composed of a target-feature cross-attention and a target-target self-attention, which can be written as:
	
    h_self(h_cross(T, F)).

	Notably, Eq. <ref> has a computation and memory cost of 𝒪(l_F l_T + l_T^2), which is linear with respect to the feature length l_F.
	
	
	

 §.§ Compact Target Representation
 
	While the attention decoupling strategy effectively relaxes the computation burden, a large l_T may still hinder the utilization of high-resolution features in Eq. <ref>. 
	
	Existing 3D human Transformers <cit.> usually recover the 3D body mesh by regressing the vertex locations Y ∈ℝ^N × 3, which leads to a redundant target representation T ∈ℝ^N × d, where the i-th row of T is the embedding of the i-th vertex. 
	
	The length of this target embedding is often quite large (N=6890 by default for the SMPL mesh), resulting in heavy computation and memory cost in attention operations even after mesh downsampling.
	
	To address this issue, we devise a more compact target representation based on a parametric human body model SMPL.
	
















	
	
	
	

  §.§.§ Parametric Human Model
 
	
	SMPL <cit.> is a flexible and expressive human body model that has been widely used for 3D human shape and pose modeling. 
	
	It is parameterized by a set of pose parameters θ∈ℝ^H × 3 and a compact shape vector β∈ℝ^1×10.
	
	
	The body pose θ is defined by a skeleton rig with H=24 joints including the body root. 
	The i-th row of θ (denoted by θ_i) is the rotation of the i-th body part in the axis-angle form whose skew symmetric matrix lies in the Lie algebra space 𝔰𝔬(3) <cit.>.
	
	The body shape is represented by a low-dimensional space learned with principal component analysis <cit.> from a training set of 3D human scans, and β is the coefficients of the principal basis vectors.
	
	With θ and β, we can obtain the 3D body mesh:
	
	
    Y ∈ℝ^N × 3 = f_SMPL(θ, β),

	
	
	where f_SMPL is the SMPL function <cit.> that gives the vertices Y on a pre-defined triangle mesh.
	
	The 3D body joints J can be obtained from the vertices via linear mapping using a pretrained  matrix ℳ∈ℝ^H × N:
	
    J ∈ℝ^H × 3 = ℳY.

	
	
	
	With the 3D human joints, we can further obtain the 2D joints using weak perspective projection. 
	Denoting the camera parameters as C ∈ℝ^3 that represents the scaling factor and the 2D principal-point translation in the projection process, the 2D joints can be obtained by:
	
	
    𝒥∈ℝ^H × 2 = Π_C(J),

	
	where Π_C is the weak perspective projection function <cit.>. 
	














	
	
	
	

  §.§.§ SMPL-Based Target Representation
 
	Inspired by the compactness and expressiveness of SMPL, we replace the original vertex-based representation with an SMPL-based representation.
	Since the variables of interest are {θ_i}_i=1^H, β, C as introduced in Eq. <ref> and <ref>, 
	we design the new target representation as
	𝒯∈ℝ^(H+2) × d, where the first H rows of 𝒯  correspond to the H body part rotations of θ, and the remaining two rows describe the body shape β and camera parameter C. 
	
	
	This new target representation conveys several advantages.
	
	First, the length of this representation is much shorter than the vertex-based one (H+2 ≪ N), thereby facilitating the efficient design of our model as shown in Figure <ref>(c).
	
	Second, our target representation is able to restrict the solution to the SMPL body space, which naturally ensures smooth meshes, whereas the vertex-based representation is prone to outliers and may lead to spiky body surfaces as shown in Figure <ref>.
	
	Third, the SMPL-based representation explicitly models the 3D rotation of body parts and thus is more readily usable in many applications, , driving a virtual avatar. 
	In contrast, the vertices cannot be directly used  and have to be converted into rotations first (often in an iterative optimization manner), which is sub-optimal in terms of both efficiency and accuracy (see more details in Section <ref>).
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	

 §.§ Multi-Scale Attention
 
	The above strategies, attention decoupling and SMPL-based target representation, enable us to explore different resolution features in the Transformer framework, which inspires a multi-scale attention design for high-quality 3D human shape and pose estimation.
	
	
	
	








































	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	

  §.§.§ Combining Multi-Scale Features


Our insight is that different resolution features are complementary to each other and should be collaboratively used for 3D human shape and pose estimation.
	
	A straightforward way to combine those features is to take each scale as a subset of tokens and concatenate all the tokens into a single feature embedding.
	Then the concatenated feature can be used as the K of the cross-attention h_cross(Q,K) in Eq. <ref>. 
	
	The resulting multi-scale attention can be written as[We focus on improving the cross-attention part of the decoupled attention (Eq. <ref>). The self-attention part is kept unchanged and  omitted  in the following sections for brevity.]:
	
	
    h̃_ms(𝒯, ℱ) =  h_cross(𝒯,  F_1  F_2 ⋯ F_S).

	
	
	However, this strategy uses the same projection weights for different scale features and thereby models target-feature dependencies in the same subspace <cit.>, which ignores the characteristics of different scales and is less flexible for fully exploiting the abundant multi-scale information.
	
	To address this issue, we introduce an improved multi-scale strategy that  can be written as
	
	
    h_ms (𝒯, ℱ) = 1/S∑_i=1^S h_cross(𝒯, F_i),

	
	where we employ different projection weights for each scale, and the output is an average of all scales as illustrated in Figure <ref>.
	
	Whereas conceptually simple, Eq. <ref> effectively incorporates multi-scale image features into the attention computation, which differs sharply from existing works <cit.> that only rely on single-scale low-resolution features.
	
	
	
	

  §.§.§ Multi-Scale Feature Positional Encoding
 
	
	
	As the attention operation itself in Eq. <ref> is position agnostic, positional encoding is often used in Transformers to inject the position information of the input.
	
	Similar to ViT <cit.>, we use the learnable positional encoding for the target and feature, which generally takes the form of x+ϕ, where ϕ is a set of learnable parameters representing the position information of a token x. 
	
	
	
	
	In particular, the positional encoding of image features can be written as ϕ_i ∈ℝ^l_F_i× d (i=1,2,⋯, S), which is directly added to feature F_i.
	
	
	Straightforwardly, we can learn positional encoding ϕ_i for all scales, which not only results in excessive parameters, but also ignores the relationship between the locations across different scales, , similar spatial locations at different scales should have similar positional embeddings. 
	
	To address this issue, we adjust our strategy to only learn the positional embedding for the highest scale, , ϕ_1, and the embeddings for the other scales are produced by aggregating ϕ_1: 
	
    ϕ_i = f_pool^(2^i-1)(ϕ_1),

	where f_pool^(2^i-1) is the average pooling with a stride and window size of 2^i-1.
	
	In real implementation, we iteratively apply a stride-2 pooling layer to ϕ_1 (see Figure <ref>):
	
    ϕ_i = f_pool^(2)(ϕ_i-1), i=2,⋯,S,
  
	which is equivalent to Eq. <ref> but requires slightly fewer computations.
	
	
	
	
	
	
	

 §.§ Joint-Aware Attention
 
	In addition to the multi-scale approach, the SMPL-based target representation 𝒯 also motivates the design of a joint-aware attention.
	
	Recall that the first H rows of 𝒯 describe the relative rotations of the H body parts (Section <ref>).
	
	As shown in Figure <ref>, the local articulation status around human joints strongly implies the relative rotation between neighboring body parts.
	Thus, it can be beneficial to adequately focus on the local image features around joints in the attention operation to improve the first H rows of 𝒯 for better estimation of human pose.
	
	To this end, we devise a joint-aware attention operation, which modifies the cross attention h_cross(Q,K) by restricting K to a local neighborhood of the human joints.
	
	This operation can be written as:
	
	
    h_ja(𝒯_i, ℱ) = f_soft((𝒯_iW_q)  (F^𝒩(𝒥_i)_1 W_k)^⊤/√(d)  + η) (F^𝒩(𝒥_i)_1 W_v),
 
	where 𝒯_i is the i-th row of 𝒯, i=1,⋯, H. 
	𝒩(𝒥_i) denotes an image patch around the i-th joint 𝒥_i with size r× r as shown in Figure <ref>, 
	
	and F^𝒩(𝒥_i)_1  ∈ℝ^r^2 × d represents the local features sampled at 𝒩(𝒥_i) from F_1.
	
	Eq. <ref> has a similar form to Eq. <ref>, where 𝒯_i and F^𝒩(𝒥_i)_1 serve as the Q and K, respectively.
	
	It is noted that we only use the highest-resolution feature F_1 for the local attention here, as 𝒩(𝒥_i) can cover a large area on smaller feature maps such that the joint-aware attention on lower-resolution features becomes similar to the global attention in Eq. <ref>.
	
	
	Similar to the Swin Transformer <cit.>, we incorporate a relative positional encoding η∈ℝ^1× r^2 in the softmax function, which is bilinearly sampled from a learnable tensor  η̃∈ℝ^(r+1)× (r+1) according to the distance between 𝒥_i and pixels in 𝒩(𝒥_i).
	
	The local attention module has a computation and memory cost of 𝒪(Hr^2), which is almost negligible compared to the normal attention that attends to the image features globally.
	
	
	Eventually, we combine the multi-scale attention (Eq. <ref>) and the joint-aware attention (Eq. <ref>) by simply taking the average as shown in Figure <ref>. 
	Denoting the combined attention as h_co(𝒯, ℱ) ∈ℝ^(H+2)× d, the i-th row of  the output  can be written as:
	
    h_co(𝒯_i, ℱ) = {   1/2 (h_ja(𝒯_i, ℱ) + h_ms (𝒯_i, ℱ)),    i ≤ H 
        h_ms (𝒯_i, ℱ),  i > H 
    		. .

	Note that h_ja(𝒯_i, ℱ) is only defined for the H body parts, , i=1,⋯, H, and thereby we directly use the multi-scale attention for the body shape β and camera C without averaging, which  correspond to i=H+1, H+2 in Eq. <ref>.
	
	
	Following the attention decoupling in Eq. <ref>, the final formulation of our attention module can  be written as 
	
    h_final(𝒯, ℱ) = h_self(h_co(𝒯, ℱ)),

	which is briefly illustrated in Figure <ref>. 
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	

	
	

 §.§ Hierarchical Architecture

	As shown in Eq. <ref>, an important issue of our current design is that the joint-aware attention relies on the 2D joints 𝒥, which is supposed to be an output of our algorithm (see Eq. <ref>).
	In other words, we need 𝒥 to reconstruct the 3D human and meanwhile need the 3D human to regress 𝒥, which essentially leads to a chicken-and-egg problem.
	To circumvent this problem, we propose a hierarchical architecture (Figure <ref>) to iteratively refine the 2D joint estimation and 3D reconstruction results.
	
	We denote the output of the b-th stage in Figure <ref>(a) as P^b={R_θ_1^b, ⋯, R_θ_H^b, β^b, C^b}, where R_θ_i^b ∈SO(3) is the rotation matrix of the i-th body part, corresponding to θ_i (the i-th row of θ).
	Then the refinement process can be written as:
	
    𝒯^b   = f_TB^b (𝒯^b-1, P^b-1, ℱ), 
    P^b    = f_fusion(𝒯^b, P^b-1),     b = 1,2,⋯, B,

	where f_TB^b is the b-th Transformer Block, which improves the target embedding  𝒯^b-1 with image features ℱ and the current 3D estimation P^b-1.
	f_fusion is a fusion layer that generates the refined estimation P^b based on the improved target embedding 𝒯^b.
	As illustrated on the rightmost of Figure <ref>, in the fusion layer we first use a linear layer to map 𝒯^b into a set of residuals: 
	
    Δ P^b = {Δ R_θ_1^b, ⋯,  Δ R_θ_H^b, Δβ^b, Δ C^b},

	and then add the residuals to the current estimation P^b-1.
	
	Similar to <cit.>, we apply the Gram–Schmidt orthogonalization to the rotation residuals such that Δ R_θ_i^b ∈SO(3). We use matrix multiplication (MatMul in Figure <ref>) instead of addition to adjust the rotation parameters.
	
	As shown in Figure <ref>(a), the hierarchical architecture is composed of B Transformer Blocks, and each Transformer Block is composed of U Transformer Units in Figure <ref>(b).
	The Transformer Unit represents the aforementioned decoupled attention h_final illustrated in Figure <ref>.
	
	To bootstrap this refinement process, we apply a global pooling layer and an MLP to the CNN features to obtain an initial coarse estimation P^0 similar to HMR <cit.>.
	
	For the initial target embedding 𝒯^0, a straightforward choice is to use a learned feature embedding as in DETR <cit.>.
	Nevertheless, we empirically find this choice makes the training less stable. Instead, we employ a heuristic strategy by combining the globally-pooled image feature and linearly-transformed P^0, , 𝒯^0 = f_global(F_S) + f_linear(P^0), where f_global is the global average pooling function.
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	

 §.§ Loss Function

	
	
	For training the proposed Transformer, we adopt the loss functions of Mesh Graphormer <cit.> which restrict the reconstructed human in terms of vertices and body joints:
	
	
    ℓ_basic =     w_Y · Y - Ŷ_1 + w_J ·J - Ĵ^2_2 + w_𝒥·𝒥 - 𝒥̂^2_2,

	where Y, J, 𝒥 are the predicted vertices, 3D joints, and 2D joints that are computed with Eq. <ref>-<ref> using the final output of our Transformer, , P^B in Figure <ref>(a).
	Ŷ, Ĵ, 𝒥̂ indicate the corresponding ground truth.
	w_Y, w_J, w_𝒥 are hyper-parameters for balancing different terms.
	
	Following <cit.>, we use L_1 loss for vertices and MSE loss for human joints.
	
	
	
	
	
	
	
	In addition, we include a rotation regularization term:
	
    ℓ_rotation = w_R ·1/H∑_i=1^HR_θ_i - R̂_θ_i_1,

	where we encourage the predicted rotation R_θ_i  to be close to the ground-truth rotation matrix R̂_θ_i.
	
	w_R is the weight of the loss.
	
	
	
	Eventually, our training loss is a combination of ℓ_basic and ℓ_rotation.
	
	We do not restrict the body shape β as we empirically find no benefits from it.
	Note that ℓ_rotation can only be used in our Transformer which uses an SMPL-based target representation; it is in contrast to existing 3D human reconstruction Transformers that do not directly consider rotations.
	
	
	
	
	
	
	
	
		
	


 §.§ Discussions

	Decoupled attention.
	While a similar idea to our attention decoupling has been touched upon in <cit.> for video classification, our work marks the pioneering exploration of it in monocular 3D human shape and pose estimation, addressing the intrincs limitations of the full attention formation of the existing works. 
	
	While we present the idea in a simplified manner in Figure <ref> for clarity, the innovation of our decoupled attention goes beyond a basic conceptual introduction. 
	
	Unlike the approach in <cit.> that relies solely on single-scale low-resolution features, we propose a multi-scale decoupled attention framework (Section <ref>). 
	This unique design allows the model to exploit both coarse and granular visual information, substantially improving the performance of 3D human shape and pose estimation.
	
	Notably, this multi-scale approach is enabled by the enhanced efficiency of our attention decoupling strategy, which allows incorporating higher-resolution features without prohibitive computational costs.
	
	In addition, we emphasize that effectively combining multi-scale features is not a straightforward endeavor, which requires dedicated algorithmic design and meticulous engineering efforts. 
	
	Particularly, instead of directly concatenating all scale features, our approach assigns different projection weights for each scale to model target-feature dependency in a scale-aware manner (Section <ref>). 
	Moreover, we devise a pooling-based multi-scale positional encoding to better represent spatial information across varying scales, as detailed in Section <ref>.
	
	
	

	SMPL-based target representation.
	While SMPL has been used as output in prior methods <cit.>, our work marks the first time that the SMPL-based target representation is used in a Transformer framework.
	
	
	We emphasize the contribution of our SMPL-based target representation is rooted in its distinct advantages.
	
	First, it considerably reduces the computational cost as illustrated in Figure <ref>.
	Second, it ensures consistent, smooth SMPL mesh, avoiding the spiky outliers on human surfaces produced by vertex-based representations (Figure <ref>).
	
	Third, it explicitly models body part rotations, facilitating its applications in driving virtual avatars (Figure <ref>).


	Furthermore, the SPML-based target representation allows the development of the joint-aware attention and motivates the hierarchical architecture of our Transformer, both of which are novel designs absent from previous works <cit.> and the concurrent work <cit.>, significantly improving the reconstruction results.

		
	
	

§ EXPERIMENTS


	
	

 §.§ Implementation Details

	For the network structure, we set the number of Transformer Blocks as B=3  and the number of Transformer Units in each block as U=2 by default.
	We use four attention heads for the Transformer  and set the feature sampling range of the joint-aware attention as r=8.
	It corresponds to a 32×32 region in the input image, which  is adequately large to encompass the vicinity of the joints.
	We set the number of feature scales S=4.
	
	
	We use HRNet <cit.> as the CNN backbone.
	We present two variants of the proposed Transformer: a base model SMPLer and a larger one SMPLer-L. 
	The two models use the same architecture, and the only difference is the channel dimension of the backbone, where we increase the channels by half for SMPLer-L.
	
	
	We use a batch size of 200 and train the model for 160 epochs.
	The loss weights in ℓ_basic and ℓ_rotation are empirically set as w_Y=100, w_J=1000, w_𝒥=100, w_R=50.
	
	
	
	
	
	
	
	

	Dataset.
	Similar to <cit.>, we extensively train our model by combining several human datasets, including Human3.6M <cit.>, MuCo-3DHP <cit.>, UP-3D <cit.>, COCO <cit.>, MPII <cit.>. 
	Similar to <cit.>, we use the pseudo 3D mesh training data from <cit.> for Human3.6M.
	
	We follow the general setting where subjects S1, S5, S6, S7, and S8 are used for training, and subjects S9 and S11 for testing. We present all results using the P2 protocol <cit.>. 
	We fine-tune the models with the 3DPW <cit.> training data for 3DPW experiments.
	
	

	Metrics.
	We mainly use mean per-joint position error (MPJPE) and Procrustes-aligned mean per-joint position error (PA-MPJPE) for evaluation.
	
	MPJPE is defined as:
	
    1/H∑_i=1^HJ_i - Ĵ_i _2,

	where J_i is the i-th row of J, , the coordinate of the i-th human joint,
	and Ĵ represents the ground truth.
	
	Eq. <ref> directly measures the joint-to-joint error, which can be influenced by global factors, including scaling, global rotation and translation. 
	
	PA-MPJPE  is defined as:
	
    min_s,R,t1/H∑_i=1^HsJ_i R+t - Ĵ_i _2, 
       s.t.   s∈ℝ,  R ∈SO(3), t ∈ℝ^1× 3,

	
	where s, R, t represent scaling, global rotation and translation, respectively.
	
	Instead of directly computing the joint-to-joint error, Eq. <ref> first aligns the prediction J to the ground truth Ĵ with a scaled rigid transformation (a closed-form solution for the alignment is given by Procrustes analysis <cit.>).
	
	Thus, PA-MPJPE is able to exclude the global factors and focus on the intrinsic human body shape and pose, emphasizing the measurement of relative position between adjacent body parts. 
	
	

	

	
	
	
	

	
		
	
	

	
	

 §.§ Comparison with the State of the Arts
 
	Quantitative evaluation.
	We compare against the state-of-the-art 3D human shape and pose estimation methods, including CNN-based <cit.>, GNN-based <cit.>), and Transformer-based <cit.>. 
	
	As shown in Table <ref>, the proposed Transformer performs favorably against the baseline approaches on both the Human3.6M and 3DPW datasets.
	
	Notably, compared to Mesh Graphormer <cit.>, the proposed SMPLer and SMPLer-L lowers the MPJPE error on Human3.6M by 8.2% and 11.7% respectively, while using only 16.5% and 32.5% of its parameters,  clearly demonstrating the effectiveness of our algorithm.
	
	

	Qualitative evaluation.
	For a more intuitive understanding of the results, we also provide visual comparisons in Figure <ref>. 
	As a typical example, in the first row of Figure <ref>, existing approaches cannot well pose the human legs as they only use low-resolution features in the Transformer and thus are prone to errors under self-occlusions and challenging poses.
	In contrast, the proposed SMPLer is able to better exploit the abundant image features in a multi-scale (coarse and fine) and multi-scope (global and local) manner, which effectively improves the estimation performance in complex scenarios.
	
	On the other hand, since existing Transformers mostly rely on a vertex-based target representation, their results do not always lie on the SMPL mesh manifold as illustrated in Figure <ref>. 
	This leads to spiky outliers on the human surface as shown in the second row of Figure <ref> or even distorted meshes in the fourth row of Figure <ref>.
	
	By contrast, our method introduces an SMPL-based target representation, which naturally guarantees the solutions to lie on the smooth human mesh space (Figure <ref>) and thereby achieves higher-quality results  as shown in Figure <ref>.
	
	
	
	
	
	
	In addition, we show more qualitative results of SMPLer with alternative views in Figure <ref>, where the proposed network performs robustly under diverse poses and complex backgrounds.
	In particular, the results show that our model is able to accurately recover global rotation relative to the camera coordinate frame, which is also validated by the improvements over MPJPE and MPVE metrics.
	
		
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	

	Controlling virtual avatars.
	An important application of 3D human shape and pose estimation is to control virtual avatars, , in Metaverse.
	As our SMPL-based target representation explicitly models the 3D rotations of body parts, the proposed SMPLer can be easily used to drive virtual humans. An example is shown in Figure <ref>.
	
	In contrast, the previous Transformers take vertices as the output which are not directly applicable here and have to be converted into rotations first for this task.
	More specifically, the predicted vertices need to be fitted to the SMPL model (Eq. <ref>) in an iterative optimization manner, which corresponds to projecting the results onto the SMPL manifold in Figure <ref>.
	Compared to the proposed solution which is essentially one-step, the two-step approach (vertices → rotations) not only leads to inefficiency issues due to the time-consuming fitting process, but also suffers from low accuracy caused by accumulative errors.
	
	To quantitatively evaluate the rotation accuracy, we introduce a new metric, called mean per-body-part rotation error (MPRE), which is defined as:
	
    MPRE = 180/π H∑_i=1^Harccos(trace(R_θ_iR̂_θ_i^⊤) - 1/2),

	where RR̂^⊤ represents the rotation matrix between the predicted rotation R and the ground truth R̂ (this can be seen from R = (RR̂^⊤) R̂).
	Essentially, Eq. <ref> describes the rotation angle ω (in degree) between the prediction and ground truth, as the three eigenvalues of a rotation matrix are 1 and e^± iω, and thus  trace(RR̂^⊤)=1+e^iω+e^ -iω=1+2cos(ω). 
	
	As the ground-truth rotations of Human3.6M are not available, we evaluate the rotation accuracy on 3DPW. 
	The proposed SMPLer achieves an MPRE of 9.9, significantly outperforming the 57.0 of Mesh Graphormer, which demonstrates the advantage of the proposed method in controlling virtual avatars.
	The visual comparison in Figure <ref> further verifies the improvement. 
	
	
	
	
	
	
	
	
	
	
	

 §.§ Analysis and Discussions

	We conduct a comprehensive ablation study on Human3.6M to investigate the capability of our method.
	We also provide more analysis and discussions about the model efficiency as well as the attention mechanism.
	We use SMPLer as our default model in the following sections.
	
	
	Decoupled attention and SMPL-based representation.
	
	We propose a new Transformer framework that is able to exploit high-resolution features for accurate 3D human reconstruction.
	At the core of this framework are a decoupled attention module and an SMPL-based target representation as introduced in Section <ref> and <ref>.
	To analyze the effectiveness of these designs, we study different choices of the attention operation (full decoupled) and the target representation (vertex-based SMPL-based) in Table <ref>.
	
	
	As shown by Table <ref>(b), the straightforward approach of using a full attention and a vertex-based representation leads to prohibitive memory and computation cost for exploiting multi-resolution image features ℱ.
	
	With limited computational resources, we cannot train this model with a proper batch size.
	
	
	While the decoupled attention can well reduce the model complexity (Table <ref>(c)), the memory footprint is still  large due to the high dimensionality of the vertex-based target representation.
	
	In contrast, the proposed Transformer combines the decoupled attention and SMPL-based representation (Table <ref>(d)), which significantly lessens the memory and computational burden, and thereby achieves more effective utilization of multi-scale features.
	
	
	
	
	
	
	Note that the feature dimensionality is not a computational bottleneck for our network anymore, as it only leads to a marginal computation overhead for employing high-resolution features as shown by Table <ref>(d) and (e).
	
	
	
	
	
	
	
	
	
	

	Effectiveness of the multi-scale attention.
	As introduced in Section <ref>, we propose an attention operation h_ms for better exploiting multi-scale image information.
	As shown in Table <ref>, using single-scale feature for 3D human shape and pose estimation, either the low-resolution (only scale S) or the high-resolution one (only scale 1), leads to inferior results compared to our full model.
	
	
	On the other hand, unifying multi-scale features in Transformer is not a trivial task. 
	As introduced in Section <ref>, instead of simply concatenating different resolution features into a single feature vector (h̃_ms in Eq. <ref>), we separately process each scale with different projection weights, , h_ms in Eq. <ref>.
	As shown in Table <ref>, the straightforward concatenation method cannot fully exploit the multi-scale information due to the undesirable restriction of using the same projection weights for different scales, resulting in less accurate 3D human estimation compared to the proposed approach.
	
	In addition, we apply multi-scale feature positional encoding  to supplement position information for the attention operation.
	As shown in Table <ref>, the model without feature positional encoding (w/o FPE) suffers from a major performance drop, especially for MPJPE. 
	
	
	Furthermore, instead of directly learning the feature positional embedding for all scales (all-scale FPE), we propose a pooling-based strategy Eq. <ref> as illustrated in Figure <ref> to better handle the spatial relation across scales.
	
	Compared to “all-scale FPE” in Table <ref>, this strategy further improves the  3D human shape and pose estimation results.
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	

	Effectiveness of the joint-aware attention.
	Motivated by the SMPL-based target representation, we propose a joint-aware attention h_ja in Eq. <ref> to better exploit local features around human joints.
	
	As shown in Table <ref>, the model without h_ja suffers from a significant performance drop, showing the importance of this module in inferring relative body part rotations.
	
	
	Further, we include a relative positional encoding in Eq. <ref> to model the spatial relation between target embedding and image features, which slightly improves the performance as  shown in Table <ref> (“w/o η”).
	
	
	
	
	
	
	

	Analysis of the hierarchical architecture.
	We apply a hierarchical architecture in Figure <ref> that is composed of  multiple Transformer Blocks and multiple Transformer Units to
	progressively refine the 3D human estimation results. 
	We visualize the refinement process in Figure <ref>, where the reconstruction becomes more accurate after more Blocks.
	
	
	Moreover, we investigate the effect of different settings of the hierarchical architecture. 
	As shown in Table <ref>, the performance gain of adding more Blocks and Units is significant at the start but converges quickly after  B>1 and U>1. 
	Therefore, we refrain from adding even more blocks and take B=3, U=2 as the default setting for a better tradeoff
	between performance and computation cost.
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	

	Attention visualization.
	For a more comprehensive study of the proposed attention modules, we visualize the learned attention maps in Figure <ref>. 
	
	First, we show the multi-scale attention h_ms in Figure <ref>(e)-(h), where the different scales correspond to the cross attention modules in Figure <ref>.
	
	The attention maps for higher resolutions, , in Figure <ref>(e), tend to be more concentrated on fine-grained features of specific body parts.
	In particular, the proposed Transformer relies on the foot and head poses to help infer body part rotation around the knee. 
	
	By contrast, for lower resolutions, , in Figure <ref>(h), the attention gets more diffused in a wider space, indicating that the multi-scale attention can exploit global information, including the background, to decide the global rotation and scale of the body.
	
	To further analyze the attention of different scales, we also provide quantitative results in Figure <ref>.
	We use the KL divergence between the attention map and a uniform distribution to measure how much the attention distribution diffuses in space. 
	As shown in Figure <ref>, the higher-resolution attention modules have higher divergence from the uniform distribution, indicating that their attention is less diffused and more concentrated at fine-grained features.
	This verifies the complementarity of low- and high-resolution features as discussed in Section <ref>.
	
	In addition, we provide a visualization of the learned joint-aware attention in Figure <ref>(d), which shows intriguing local patterns where the information around the target human joint is emphasized for better 3D human reconstruction.
	Specifically, it has higher weights on the upper and lower sides of the knee, which captures the kinetic status around the joint and helps infer the relative rotation between the thigh and calf.
	
	
	
	Lastly, we employ a self-attention layer h_self for attention decoupling (Figure <ref>), which models the target-target dependencies in Figure <ref>(c).
	We visualize this layer by showing the interactions between one query joint and all other joints, where the h_self focuses on neighboring joints on the kinetic tree <cit.> to produce more reasonable predictions.
	
	
	
	
	
	
	
	
	
	
	
	
	

	Running speed.
	As shown in Table <ref>, the inference of SMPLer and SMPLer-L runs at 96.0 and 73.9 frames per second (fps) respectively, which is effectively real-time.
	While exploiting high-dimensional  multi-resolution features, SMPLer has a nearly three-times running speed and one-fifth GFlops of the baseline Transformers <cit.> that are solely based on low-dimensional features, showing the efficiency of the proposed algorithm.
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	

	Relationship with prior works.
	The proposed SMPLer is based on two fundamental designs: the attention decoupling and the SMPL-based target representation. 
	These designs substantially distinguish our algorithm from existing Transformers <cit.>, which are based on full attention and vertex-based representation and thereby can only use low-resolution features in the attention operation.
	Moreover, the proposed framework also clearly differs from the existing SMPL-based approaches, such as HMR <cit.>, SPIN <cit.>, and RSC-Net <cit.> which are solely based on CNNs and cannot exploit the powerful learning capabilities of Transformers for 3D human shape and pose estimation.
	As a result, they suffer from a large performance gap compared to the state-of-the-arts as evidenced in Table <ref>.
	
	
	The two basic designs naturally lead to the development of several novel modules, including the multi-scale attention and the joint-aware attention.
	While these modules are conceptually simple, to have them work properly in our framework is by no means trivial and requires meticulous algorithmic designs, such as the approach to fuse multi-scale features, the pooling-based positional encoding, the relative positional encoding in the joint-aware attention, and the hierarchical architecture.
	
	Eventually, with the above designs, SMPLer achieves significant improvement over the state-of-the-art methods <cit.> with better efficiency.
	
	
	
	
	
	
	
	
	
	
	
	

§ CONCLUSIONS

	
	We have developed a new Transformer framework for high-quality 3D human shape and pose estimation from a single image.  
	At the core of this work are the decoupled attention and the SMPL-based target representation that allow efficient utilization of high-dimensional features. 
	These two strategies also motivate the design of the multi-scale attention and joint-aware attention modules,
	
	which can be potentially extended to other areas where multi-scale and multi-scope information is valuable, such as motion estimation <cit.> and image restoration <cit.>.
	
	Meanwhile, the proposed algorithm can also be applied to other 3D reconstruction problems, such as 3D animal reconstruction <cit.> and 3D hand reconstruction <cit.> by replacing SMPL with other parametric models, , SMAL <cit.> and MANO <cit.>. 
	Nevertheless, this is beyond the scope of this work and will be an interesting direction for future research.
	
	Similar to existing Transformers, the proposed SMPLer is still a hybrid structure that consists of CNN layers in the backbone. To incorporate attention-based backbones in SMPLer such as <cit.> will be another direction worth exploring in future work.
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	










	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	
	IEEEtran
	
	
	
	
	
	
	
	
	
	

















	







	
	










	
	
	
	
	
	
	
	
	
	
	
	
	
