Previous work on predicting or generating 3D human pose sequences regresses either joint rotations or joint positions. The former strategy is prone to error accumulation along the kinematic chain, as well as discontinuities when using Euler angles or exponential maps as parameterizations. The latter requires re-projection onto skeleton constraints to avoid bone stretching and invalid configurations. This work addresses both limitations. QuaterNet represents rotations with quaternions and our loss function performs forward kinematics on a skeleton to penalize absolute position errors instead of angle errors. We investigate both recurrent and convolutional architectures and evaluate on short-term prediction and long-term generation. For the latter, our approach is qualitatively judged as realistic as recent neural strategies from the graphics literature. Our experiments compare quaternions to Euler angles as well as exponential maps and show that only a very short context is required to make reliable future predictions. Finally, we show that the standard evaluation protocol for Human3.6M produces high variance results and we propose a simple solution.
@cite_15: This paper presents the first study on forecasting human dynamics from static images. The problem is to input a single RGB image and generate a sequence of upcoming human body poses in 3D. To address the problem, we propose the 3D Pose Forecasting Network (3D-PFNet). Our 3D-PFNet integrates recent advances on single-image human pose estimation and sequence prediction, and converts the 2D predictions into 3D space. We train our 3D-PFNet using a three-step training strategy to leverage a diverse source of training data, including image and video based human pose datasets and 3D motion capture (MoCap) data. We demonstrate competitive performance of our 3D-PFNet on 2D pose forecasting and 3D structure recovery through quantitative and qualitative results.
@cite_16: 1. Introduction and Historical Background 1.1 Why make human figure models? 1.2 Historical roots 1.3 What is currently possible? 1.4 Manipulation, animation, and simulation 1.5 What did we leave out? 2. Mody Modeling 2.1 Geometric body modeling 2.2 Representing articulated figures 2.3 A flexible torso model 2.4 Shoulder complex 2.5 Clothing models 2.6 The anthropometry database 2.7 The anthropometry spreadsheet 2.8 Strength and torque display 3. Spatial Interaction 3.1 Direct manipulation 3.2 Manipulation with constraints 3.3 Inverse kinematic positioning 4. Behavioral Control of Articulated Figures 4.1 An interactive system for postural control 4.2 Interactive manipulation with behaviors 4.3 The animation interface 4.4 Human figure motions 4.5 Virtual human control 5. Simulation with Societies of Behaviors 5.1 Forward simulation with behaviors 5.2 Locomotion 5.3 Strength guided motion 5.4 Collision-free path and motion planning 5.5 Posture planning 6. Task-level Specifications 6.1 Performance simulation with simple commands 6.2 Natural language expressions of kinematics and space 6.3 Task-level simulation 6.4 A framework for instruction understanding 7. Epilogue 7.1 A road map toward the future 7.2 Conclusion
@cite_11: We introduce SE3-Nets, which are deep networks designed to model rigid body motion from raw point cloud data. Based only on pairs of depth images along with an action vector and point wise data associations, SE3-Nets learn to segment effected object parts and predict their motion resulting from the applied force. Rather than learning point wise flow vectors, SE3-Nets predict SE3 transformations for different parts of the scene. Using simulated depth data of a table top scene and a robot manipulator, we show that the structure underlying SE3-Nets enables them to generate a far more consistent prediction of object motion than traditional flow based networks.
Forecasting is an active topic of research beyond the prediction of human pose sequences. Pixel-level prediction using human pose as an intermediate variable has been explored . Related work also includes the forecasting of locomotion trajectories , human instance segmentation , or future actions . Other types of conditioning have also been explored for predicting poses: for instance, @cite_16 explore generating skeleton pose sequences of music players from audio, @cite_15 aim at predicting future pose sequences from static images. Also relevant is the prediction of 3D poses from images or 2D joint positions . The prediction of rigid object motion for robotic applications is also relevant, e.g. @cite_11 model object dynamics using a neural network that performs spatial transformations on point clouds.