SEAL-Pose: Enhancing Pose Estimation through Trainable Loss Function
Structured Energy network;Energy-based models;Trainable Loss-function;Dynamic loss function;Pose Estimation
Yes
No ethics review needed.
The contributions do not reach the bar of ICLR therefore I recommend rejection. The further improvement can be better presentation of paper and more experimental validation of the proposed loss.
1. This paper addresses an interesting problem of improving the plausibility of output pose. The SEAL loss is new and effective.
2. This paper is easy to understand.
This paper presents SEAL-Pose, a Structured Energy As Loss framework to improve the 3D pose estimation from 2D keypoints. The SEAL loss is previously applied in probabilistic models whereas the paper modifies it to deterministic models. This loss improves the dependencies among the keypoints which produces more plausible poses empirically.
1. The writing of this paper can be improved, e.g., the names of task-net and loss-net are confusing since the task here is only pose estimation and loss-net only refers to energy loss. Also, it would be clearer if authors can give an overview of what each net functions in the beginning of Sec.3.1.
2. The motivation is not clearly explained and verified. It’s not clear why SEAL-Pose can improve the plausibility of estimated poses and why it can perform better in the proposed metrics, i.e., LSE, BSLE, and LLE.
3. The method should be evaluated on more advanced frameworks. It’s necessary for this method to compete against state-of-the-art methods and on more challenging benchmarks, for example, 3DPW, MPI-INF-3DHP. In Table 2, the performance is very similar to the baseline method VideoPose.
4. Minor issues:
    1. Inconsistency between P-MPJPE and PA-MPJPE in H36M and H3WB
    2. The quality of figures can be improved
Yes
No ethics review needed.
* Is GBI used in combination with SEAL? Fig. 2 (left) seems to indicate so but the text does not make this clear.
* As far as I understand, the method is only tested with ground-truth 2D keypoints (L200). This is not a realistic setup, and 2D keypoints from 2D pose estimators would yield a more realistic evaluation. Is this correct?
* Are the hyperparameters tuned for the test set?
* The problem of human pose lifting is of interest to the research community.
* Applying the SEAL framework to this task is novel.
* The method, when applied to SimpleBaseline and VideoPose3D, improves results on Human3.6M, both in terms of the standard metrics and in terms of bone symmetry and bone length error.
* The ablation model (Section 3.2) is useful in demonstrating the value in training the loss-net together with the task-net.
This work addresses 2D-to-3D human pose lifting, by applying the "Structured Energy as Loss" (SEAL) framework. That is, besides training a regressor for the task (here called task-net), a second network (loss-net) is also trained that scores prediction-input pairs for their plausibility. At inference time, the output can be taken as is, or gradient-based inference (GBI) can be applied to modify the prediction towards driving down the loss-net output (to make the prediction seem more plausible to the loss-net). This GBI can be applied by tuning either the prediction itself or the model weights.
The motivation is to better take into account dependencies in the output variables, such as bone lengths in the case of human pose estimation.
The model is evaluated on Human3.6M and its whole-body extension and is shown to improve the scores.
* The most serious problem I see is using a hyperparameter sweep tool (from WandB) to tune hyperparameters directly for the Human3.6M/H3WB test set. If this was indeed done so, it invalidates the seen improvements, as the gap is not so large and tuning hyperparameters for a particular test set can aways achieve significantly stronger results.
* While the method is motivated from the prior SEAL method, I am not convinced that the terminology of structured energy is helpful for understanding rather than obscuring what actually happens in the approach. In essence, the final model is a fairly standard conditional GAN. The terminology of "structured" learning/energy/outputs comes from the pre-deep-learning era when multi-output model were generally less standard and more difficult to train. Today, talking about a learned loss function or indeed simply a discriminator may be much better understood by the community, since models that make "structured" predictions are commonplace today. GANs have been used in a similar fashion for 2D-to-3D lifting (e.g. [3]), diminishing the novelty of the proposed paper.

* The work only uses Human3.6M (and its wholebody extension). While Human3.6M has been very valuable for research over the last decade, today we now have many more datasets and it is now possible to give stronger evidence for a method than improvements on two specific subjects. The particular bone-length structures of subject 9 and subject 11 of H36M may not generalize. Other possible training datasets would include the following (of course I do not expect using all of these, it is just to give some ideas): MPI-INF-3DHP, CMU-Panoptic, AMASS, HuMMan, AIST-dance++, AGORA, BEDLAM, GeneBody, DNA-Rendering, RICH etc. And other evaluation datasets include 3DPW and EMDB. Again, I do not expect using all, but using at least something further beyond the Human3.6M data would make the evidence much stronger.
* The baseline methods (SimpleBaseline, VideoPose) are fairly old by the standards of this field (2017 and 2019). It would be important to try newer lifting methods as well. The results are also not compared to the current SOTA methods or those from the last five years.
* The writing is quite verbose, for example an full page (page 5) is spent on describing limb length losses, which could be expressed in a briefer way. Such bone-based losses and metrics have been used in many prior works, for example [2,3]
* It is not clear why the SemGCN model has spikes in the training curve (Fig. 3). These spikes can appear for many practical reasons in a particular implementation and do not generally mean a fundamental problem.
Yes
No ethics review needed.
1. The technical part is not clearly stated, thus making the readers hard to follow. For example, 
in eqn(1),(3), how is B_t defined? How is \eta_{\phi} defined? Is “x” the input image or 2D pose for lifting? Is L the joint number? 

2. There are many previous methods that estimate a probabilistic distribution of output 3D skeleton from 2D input (e.g. Diffusion-based 3d human pose estimation with multi-hypothesis aggregation, ICCV 2023). It may be better for SEAL pose to combine with such framework instead of a deterministic model. 

3. Why not use MPI-INF-3DHP for evaluation? The experiments have been limited to the camera settings of H36M, which may not convince the readers for the generalization ability of SEAL-pose. 

4. The authors propose LSE and BSLE. However, they only report these two metrics on H3WB in Table.3. Why not report on H36M for body only poses? 

5. Is there any visualized comparison to demonstrate the 3d lifting quality of SEAL-pose? Without visualized results, I could not fully understand the improvement brought by lower LSE and BSLE. As H3WB contains 133 keypoints with distinct scale, which part is  improved the most (face, or hand, or body)?
The incorporation of SEAL framework for 3D human pose lifting is a novel idea. The experiments could support the effectiveness of SEAL-psoe to some degree.
This paper presents SEAL-pose to handle 2D-to-3D lifting problem in human pose estimation. The core is training a learnable loss function to encode the structure of output body limbs. To evaluate the performance, the authors propose two new metrics: Limb symmetry error (LSE) and Body segment length error (BSLE). Experiments are performed on H36M and H3WB.
Description of methods should be improved. Experiments are not enough. See Questions below.
Yes
No ethics review needed.
Please see the Weaknesses.
The proposed method is simple but effective.
The paper introduces a Structured Energy As Loss (SEAL) framework aimed at improving 3D human pose estimation. However, the paper is incomplete (less than eight pages) and lacks a description of the method’s motivation, a comparison with prior approaches, and an evaluation against state-of-the-art methods.
1. The diagram in Figure 1 is overly simplistic and does not effectively convey the motivation behind the proposed method.
2. The paper is incomplete (less than eight pages), lacks a thorough description of the method’s motivation, a discussion on advantages over prior methods, and comparisons with state-of-the-art approaches.
@inproceedings{
anonymous2024sealpose,
title={{SEAL}-Pose: Enhancing Pose Estimation through Trainable Loss Function},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=KRqMfdwQaP},
note={under review}
}
Accurately predicting 3D human pose is a fundamental task in computer vision, where capturing the dependencies between multiple output variables remains a significant challenge. In this paper, we propose SEAL-Pose, a novel application of the Structured Energy As Loss (SEAL) framework to enhance 3D human pose estimation. SEAL, which was previously applied only to probabilistic models, has been adapted in this work for deterministic models, specifically focusing on 3D human pose estimation from 2D keypoints. Our method uses a structured energy network as a dynamic, trainable loss function, and shows promising results in enhancing pose estimation. We also introduce new evaluation metrics—Limb Symmetry Error (LSE), Body Segment Length Error (BSLE)—to assess structural consistency, with results indicating improvements in this area. Through experiments on the Human3.6M (H36M) and Human3.6M WholeBody (H3WB) datasets, SEAL-Pose shows reductions in pose estimation errors and outperforms existing baselines. This work highlights the potential of applying structured energy networks to tasks requiring complex output structures, offering a promising direction for future research.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Structured Energy network Energy-based models Trainable Loss-function Dynamic loss function Pose Estimation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/3ec76edda82f2ec602aa65e220e90747468bc7aa.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
SEAL-Pose: Enhancing Pose Estimation through Trainable Loss Function
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission