HDM-Net: Monocular Non-Rigid 3D Reconstruction with Learned Deformation Model
Vladislav Golyanik Soshi Shimada Kiran Varanasi Didier Stricker


HDM-Net: Monocular Non-Rigid 3D Reconstruction

V. Golyanik 

^♭Augmented Vision, DFKI (<https://av.dfki.de>)  ^♯University of Kaiserslautern 

HDM-Net: Monocular Non-Rigid 3D Reconstruction 
with Learned Deformation Model
    Vladislav Golyanik^♭,♯Soshi Shimada^♭,♯ Kiran Varanasi^♭Didier Stricker^♭,♯
    December 30, 2023
===============================================================================



Monocular dense 3D reconstruction of deformable objects is a hard ill-posed problem in computer vision. 
Current techniques either require dense correspondences and rely on motion and deformation cues, or assume a highly accurate 
reconstruction (referred to as a template) of at least a single frame given in advance and operate in the manner of non-rigid tracking. 
Accurate computation of dense point tracks often requires multiple frames and might be computationally expensive.
Availability of a template is a very strong prior which restricts system operation to a pre-defined environment and scenarios. 
In this work, we propose a new hybrid approach for monocular non-rigid reconstruction which we call 
Hybrid Deformation Model Network (HDM-Net). 
In our approach, a deformation model is learned by a deep neural network, with a combination 
of domain-specific loss functions. We train the network with multiple states of a non-rigidly deforming structure with a known shape 
at rest. HDM-Net learns different reconstruction cues including texture-dependent surface deformations, 
shading and contours. 
We show generalisability of HDM-Net to states not presented in the training dataset, with unseen textures and under 
new illumination conditions. 
Experiments with noisy data and a comparison with other methods demonstrate the robustness and accuracy of the proposed approach 
and suggest possible application scenarios of the new technique in interventional diagnostics and augmented reality. 


 





§ INTRODUCTION



The objective of monocular non-rigid 3D reconstruction (MNR) is the recovery of a time-varying geometry observed 
by a single moving camera. In the general case, none of the states is observed from multiple views, 
and at the same time, both the object and the camera move rigidly. This problem is highly ill-posed in the sense 
of Hadamard since multiple states can cause similar 2D observations. To obtain a reasonable solution, multiple 
additional priors about the scene, types of motions and deformations as well as camera trajectory are required. 
Application domains of MNR are numerous and include robotics, medical applications and visual communication systems. 
MNR also has a long history in augmented reality (AR), and multiple applications 
have been proposed over the last twenty years ranging from medical systems to communication and entertainment <cit.>. 

All approaches to MNR can be divided into two main model-based classes — non-rigid structure from motion (NRSfM) and template-based 
reconstruction (TBR). NRSfM relies on motion and deformation cues and requires dense point correspondences over multiple frames 
<cit.>. Most accurate methods for dense correspondences operate on multiple frames and are prohibitively slow 
for real-time applications <cit.>. Moreover, their accuracy is volatile and influenced by changing illumination and shading effects 
in the scene. TBR, per definition, assumes a known template of the scene or an object, a highly accurate reconstruction 
for at least one frame of the scene <cit.>. Sometimes, the template also needs to be accurately positioned, with a minimal initial 
reprojection error to the reference frame. In this context, TBR can also be comprehended as non-rigid tracking <cit.>. 
Obtaining a template is beyond the scope of TBR, though joint solutions were demonstrated in the literature. 
In some cases, a template is obtained under the rigidity assumption, which might not always be fulfiled 
in practical applications <cit.>. 

Apart from the main classes, methods for monocular scene flow (MSF) and hybrid NRSfM can be named. 
MSF jointly reconstructs non-rigid geometry and 3D displacement fields <cit.>. 
In some cases, it relies on a known camera trajectory or proxy geometry (an initial coarse geometry estimate) <cit.>. 
In hybrid NRSfM, a scene-specific shape prior is obtained on-the-fly under non-rigidity, and the input is 
a sequence of point tracks <cit.>. Geometry estimation is then conditioned upon the shape prior. 

MNR has only recently entered the realm of dense reconstructions <cit.>. 
The dense setting brings additional challenges for augmented reality applications such as scalability with 
the number of points and increased computational and memory complexity. 



 §.§ Contributions


The scope of this paper is general-purpose MNR, the reconstruction scenarios are not known in advance. 
We propose deep neural network (DNN) based deformation model for MNR. 
We train DNN with a new synthetically generated dataset
covering the variety of smooth and isometric deformations occurring in the real world 
(clothes deformations, waving flags, bending paper and, to some extent, biological soft tissues). 
The proposed DNN architecture combines supervised learning with domain-specific loss functions. 
Our approach with a learned deformation model — Hybrid Deformation Model Network (HDM-Net) — surpasses performances 
of the evaluated state-of-the-art NRSfM and template-based methods by a considerable margin. We do not require dense point tracks or 
a well-positioned template. Our initialisation-free solution supports large deformations and copes well with several textures and illuminations. 
At the same time, it is robust to self-occlusions and noise. In contrast to existing DNN architectures for 3D,
we directly regress 3D point clouds (surfaces) and depart from depth maps or volumetric representations. 

In the context of MNR methods, our solution can be seen as a TBR with considerably relaxed initial conditions and 
a broader applicability range per single learned deformation model. Thus, it constitutes a new class of methods — instead of a template, 
we rather work with a weak shape prior and a shape at rest for a known scenario class. 

We generate a new dataset which fills a gap for training DNNs for non-rigid scenes[the dataset is 
available upon request.] and perform series 
of extensive tests and comparisons with state-of-the-art MNR methods. Fig. <ref> 
provides an overview of the proposed approach — after training the network, we accurately infer 3D geometry 
of a deforming surface. Fig. <ref> provides a high-level overview of the proposed architecture. 

The rest of the paper is partitioned in
Related Work (Sec. <ref>), 
Architecture of HDM-Net (Sec. <ref>), 
Geometry Regression and Comparisons (Sec. <ref>) 
and Concluding Remarks (Sec. <ref>) Sections. 




§ RELATED WORK

 

In this section, we review several algorithm classes and position the proposed HDM-Net among them. 



 §.§ Non-Rigid Structure from Motion

NRSfM requires coordinates of tracked points throughout an image sequence. 
The seminal work of Bregler <cit.> marks the origin of batch NRSfM. 
It constrained surfaces to lie in a linear subspace of several unknown 
basis shapes. This idea was pursued by several successor methods <cit.>. 
Since the basis shapes, as well as their number, are unknown, this subclass is sensitive to
noise and parameter choice. Furthermore, an optimal number of basis shapes allowing to express all 
observed deformation modes does not necessarily always exist <cit.>. 
Along with that, multiple further priors were proposed for NRSfM including temporal smoothness <cit.>, 
basis <cit.>, inextensibility <cit.> and 
shape prior <cit.>, among others. 
The inextensibility constraint penalises deviations from configurations increasing the total surface area. 
In other words, non-dilatable states are preferred. Several methods investigate a dual trajectory basis and
considerably reduce the number of unknowns <cit.>, whereas the other ones explicitly
model deformations using physical laws <cit.>. 
Multiple general-purpose unsupervised learning techniques were successfully 
applied to NRSfM including non-linear dimensionality reduction <cit.> (diffusion maps), 
<cit.> (kernel trick) and expectation-maximisation <cit.>. 
A milestone in NRSfM was accompanied by a further decrease in the number of unknowns and required prior 
knowledge for reconstruction. Thus, some of the methods perform a low-rank approximation of a stacked 
shape matrix <cit.>. A further milestone is associated with the ability to perform dense reconstructions 
<cit.>.

Several methods allow sequential processing <cit.>. Starting from an initial estimate 
obtained on several first frames of a sequence, they perform reconstructions upon arrival of every new frame 
in an incremental manner. The accuracy of sequential methods is consistently lower than those of the batch counterparts. 
While still relying on point tracks, they can enable lowest latencies in real-time and interactive applications. 
Several methods learn and update an elastic model of the observed scene on-the-fly <cit.> 
(similarly to the sequential methods, point tracks over the complete sequence are not required). 
Solving the underlying equations might be slow, and the solution was demonstrated only for sparse settings. 



    
    < g r a p h i c s >
 
  An overview of the architecture of the proposed HDM-Net with encoder and decoder. The input of HDM-Net is an image of 
  dimensions 224 × 224 with three channels, and the output is a dense reconstructed 3D surface 
  of dimensions 73 × 73 × 3 (a point cloud with 73^2 points).  
   
      

    
    < g r a p h i c s >
 
  Architecture of the proposed HDM-Net: detailed clarification about the structures of the encoder and decoder. 
   








 §.§ Template-Based Reconstruction


Approaches of this class assume a known template, 
an accurate reconstruction of at least one frame of the sequence. Most methods operate 
on a short window of frames or single frames. Some TBR methods are known as
non-rigid trackers <cit.>. Early physics-based techniques formalised 3D reconstruction 
with elastic models and modal analysis <cit.>. They assumed that some material 
properties (such as the elastic modulus) of the surface are known and could handle small non-linear deformations. 

Multiple priors developed for NRSfM proved their effectiveness for TBR including isometry 
<cit.>, 
statistical priors <cit.>, 
temporal smoothness <cit.>,  
inextensibility priors <cit.> and mechanical priors in an improved form <cit.>. 
Moreover, modelling image formation process by decomposing observed intensities into lighting, shading and albedo components 
was also shown to improve tracking accuracy <cit.>. 



 §.§ Monocular Scene Flow

A somewhat exotic class of approaches developed in parallel to NRSfM and TBR is monocular scene flow (MSF). 
Birkbeck 's approach can handle non-rigid scenes relying on a known constant camera motion <cit.>. 
While camera trajectory can be sometimes available in AR systems, there is no guarantee of its linearity. 
In <cit.>, a variational solution to rigid multi-body scenes was proposed. Recently, Xiao proposed 
an energy-based method for rigid MSF in the context of automotive scenarios. Their approach is based on 
a temporal velocity constancy constraint <cit.>. 

In general, MSF methods are restricted in the handling of non-rigid surfaces. One exception 
— NRSfM-Flow of Golyanik <cit.> — 
takes advantage of known 2D-3D correspondences and relies 
on batch NRSfM techniques for an accurate scene flow estimation of non-rigid scenes. 
It inherits the properties of NRSfM and does not assume a known camera trajectory or proxy geometry. 



 §.§ Specialised Models for Faces and Bodies


For completeness, we provide a concise overview of specialised approaches. 
Compared to TBR, they are dedicated to the reconstruction of single object classes like 
human faces <cit.> or 
human bodies <cit.>. They do not use a single prior state (a template), 
but a whole space of states with feasible deformations and variations. The models et al.
are learned from extensive data collections showing a wide variety of forms, expressions (poses) and textures. 
In almost all cases, reconstruction with these methods means projection into the space of 
known shapes.
To obtain accurate results, post-processing steps are required (for transferring subtle details to 
the initial coarse estimates). In many applications, solutions with predefined models might be a right choice, 
and their accuracy and speed may be sufficient. 





 §.§ DNN-based 3D Reconstruction

In the recent three years, several promising approaches for inferring 2.5D and 3D geometry have been developed. 
Most of them regress depth maps <cit.> 
or use volumetric representations <cit.> akin to sign distance fields <cit.>. 
Currently, the balance of DNN-based methods for 3D reconstruction is perhaps in favour of 
face regressors <cit.>. 
The alternatives to sparse NRSfM of Tome and Zhou work exclusively for human poses <cit.>. 
The 3D-R2N2 network generates 3D reconstructions from single and multiple views and requires large 
data collections for training <cit.>. 
In contrast to several other methods, it does not require image annotations. 
Point set generation netwoet al.rk of Fan <cit.> is trained for a single view reconstruction of 
rigid objects and directly outputs point sets. 
More and more methods combine supervised learning and model-based losses thus imposing
additional problem-specific constraints <cit.>. 
Also, this has often the side effect of decreasing the volume requirements on the datasets <cit.>. 
The work of Pumarola <cit.> is most closely related to ours. The architecture is separated into three sub-networks which have different roles — creating heat-map of 2D images, depth estimation and 3D geometry inference. Those sub-networks are jointly trained. Our architecture is relatively simple. Encoder and decoder are employed and the output is penalized with three kinds of losses which have different geometrical properties — 3D geometry, smooth surface and contour information after projection onto a 2D plane.





 §.§ Attributes of HDM-Net


In this section, we position the proposed approach among the vast body of the literature on MNR. 
HDM-Net bears a resemblance to DNN-based regressors which use encoder-decoder architecture <cit.>. 
In contrast to many DNN-based 3D regressors <cit.>, our network does not include fully connected layers 
as they impede generalisability (lead to overfitting) as applied to MNR. As most 3D reconstruction approaches, 
it contains a 3D loss. 

In many cases, isometry is an effective and realistic constraint for TBR, as shown in <cit.>. 
In HDM-Net, isometry is imposed through training data. 
The network learns the notion of isometry from the opposite, by not observing other deformation modes. 
Another strong constraint in TBR is contour information which, however, has not found wide use in MSR, 
with only a few exceptions <cit.>. In HDM-Net, we explicitly impose contour constraints 
by comparing projections of the learned and ground truth surfaces.

Under isometry, the solution space for a given contour is much better constrained compared to the extensible cases. 
The combined isometry and contour cues enable efficient occlusion handling in HDM-Net. 
Moreover, contours enable texture invariance up to a certain degree, as a contour remains unchanged irrespective of the texture. 
Next, through variation of light source positions, we train the network for the notion of shading. 
Since for every light source configuration, the underlying geometry is the same, HDM-Net
acquires awareness of varying illumination.
Besides, contours and shading in combination enable reconstruction of texture-less surfaces. 
To summarise, our framework has unique properties among MSR methods
which are rarely found in other MNR techniques, especially when combined. 

 



§ ARCHITECTURE OF HDM-NET
 

We propose a DNN architecture with encoder and decoder depicted in Fig. <ref> (a general overview). 
The network takes as an input an image of dimensions 224 × 224 with three channels. 
Initially, the encoder extracts contour, shading and texture deformation cues and generates a compact latent space representation of 
dimensions 28 × 28 × 128. Next, the decoder applies a series of deconvolutions and outputs a 3D surface 
of dimensions 73 × 73 × 3 (a point cloud). It lifts the dimensionality of the latent space until the dimensionality 
of activation becomes identical to the dimensionality of ground truth.  The transition from the implicit representation into 3D 
occurs on the later stage of decoder through a deconvolution. Fig. <ref> provides a detailed clarification about 
the structures of encoder and decoder. 

As can be seen in Fig. <ref> and <ref>, we skip some connections in HDM-Net 
to avoid vanishing gradients, similar to resnet <cit.>. Due to the nature of convolutions, 
our deep network might potentially lose some important information in the forward path which might be advantageous 
in the deeper layers. 
Thus, connection skipping compensates for this side effect — for each convolution layer — which results 
in the increased performance. Moreover, in the backward path, shortcut connections help to overcome 
the vanishing gradient problem, a series of numerically unstable gradient multiplications leading 
to vanishing gradients. Thus, the gradients are successfully passed to the shallow layers. 

Fully connected (FC) layers are often used in classification tasks <cit.>. They have more
parameters than convolution layers and are known as a frequent cause of overfitting. We have tried FC layers 
in HDM-Net and observed overfitting on the training dataset. Thus, FC layers reduce generalisation ability 
of our network. Furthermore, spatial information is destroyed as the data in the decoder is concatenated
before being passed to the FC layer. In our task, needless to say, spatial cues are essential for 3D regression. 
In the end, we omit FC layers and successfully show generalisation ability of 3D reconstruction on the 
test data. 




 §.§ Loss Functions


Let = {_f}, f ∈{1, , F} denote predicted 3D states, and ^GT = {_f^GT} is 
the ground truth geometry; 
F is the total number of frames and N is the number of points in the 3D surface. 
In HDM-Net, contour similarity and the isometry constraint are the key innovations
and we apply three types of loss functions summarised into the loss energy: 

    (, ^GT) = _3D(, ^GT) + _iso() + _cont.(, ^GT).



3D error: The 3D loss is the main loss in 3D regression. 
It penalises the differences between predicted and ground truth 3D states and is 
common in training for 3D data: 

    _3D(, ^GT)=1/F∑_f=1^F‖_f^GT - _f‖_ℱ^2,

where ‖·‖_ℱ denotes the Frobenius norm. 
Note that we take an average of the squared Frobenius norms of the differences between the learned and ground truth geometries. 

Isometry prior: To additionally constrain the regression space, we embed 
isometry loss which enforces the neighbouring vertices to be located close to each other. 
Several versions of inextensibility and isometry constraints can be found in MSR — a common one is based on 
differences between Euclidean and geodesic distances. For our DNN architecture, we choose a differentiable loss 
which performs Gaussian smoothing of _f and penalises the difference between the unembellished and smoothed 
version _i: 

    _iso() = 1/F∑_f=1^F‖_f - _f‖_ℱ,

with 

    _f=1/2πσ^2exp ( -x^2+y^2/2σ^2)∗_f,

where ∗ denotes a convolution operator and σ^2 is the variance of Gaussian. 

 


Contour loss: If the output of the network and the ground truth coordinates are similar, 
the contour shapes after projection onto a 2D plane have to be similar as well. The main idea of the reprojection loss 
is visualised in Fig. <ref>-(a). 
After the inference of the 3D coordinates by the network, we project them onto the 2D plane and compute the difference 
between the two projected contours. 
If focal lengths f_x, f_y as well as the principal point (c_x, c_y) of the camera are known 
(the  used for the dataset generation is provided in Sec. <ref>), 
observed 3D points = (p_x, p_y, p_z) are projected to the 
image plane by the projection operator π : ℝ^3 →ℝ^2: 

    '(u, v) = π() = ( f_x p_x/p_z + c_x, f_y p_y/p_z + c_y )^𝖳,
 
where ' is the 2D projection of  with 2D coordinates u and v. Otherwise, we apply an orthographic camera model. 

A naïve shadow casting of a 3D point cloud onto a 2D plane is not differentiable, the network cannot backpropagate gradients 
to update the network parameters. The reason is twofold. 
In particular, the cause for indifferentiability is 
the transition from point intensities to binary shadow indicators with an ordinary step function (the numerical reason) 
using point coordinates as indexes on the image grid (the framework-related reason). 

Fig. <ref>-(b) shows how we circumvent this problem. 
The first step of the procedure is the projection of 3D coordinates onto a 2D plane using either a perspective 
or an orthographic projection. As a result of this step, we obtain a set of 2D points. 
We generate K = 73^2 translation matrices _j = [ 1 0 u; 0 1 v ] 
using 2D points and a flow field tensor of dimension K × 99 × 99 × 2
(the size of each binary image is 99 × 99). 
Next, we apply bilinear interpolation <cit.> with generated flow fields on the replicated basis matrix  K times 
and obtain K translation indicators. 
_99 × 99 is a sparse matrix with only a single central non-zero element which equals to 1. 
Finally, we sum up all translation indicators and softly threshold positive values in the sums to ≈ 1, our shadow indicator. 
Note that to avoid indifferentiability in the last step, the thresholding is performed by a combination of a rectified 
linear unit (ReLU) and tanh function (see Fig. <ref>-(b)): 

    τ( ℐ(_f(n))) = max(tanh(2  _f(n)), 0),

where n ∈{1, , N} denotes the point index, _f(n) denotes a reprojected point _f(n) in frame f, 
and ℐ(·) fetches intensity of a given point. We denote the differentiable projection operator and differentiable soft thresholding 
operator by the symbols π^†(·) and τ(·) respectively. 
Finally, the contour loss reads 

    _cont.(, ^GT) = 1/F∑_f = 1^F‖τ(π^†(_f)) - 
      τ( π^†(_f^GT)) ‖_ℱ^2.

Note that object contours correspond to 0-1 transitions. 






§ DATASET AND TRAINING
 

For our study, we generated a dataset with a non-rigidly deforming object using Blender <cit.>. 
In total, there are 4648 different temporally smooth 3D deformation states with structure bendings, smooth foldings 
and wavings, rendered under Cook-Torrance illumination model <cit.> 
(see Fig. <ref> for the exemplary frames from our dataset). 
We have applied five different camera poses, 
five different light source positions and four different textures corresponding 
to the scenarios we are interested in — endoscopy, graffiti (it resembles a waving flag)
clothes and carpet (an example of an arbitrary texture). 
The endoscopic texture is taken from <cit.>. 
 
Illuminations are generated based on the scheme in Fig. <ref>-(a), the textures and illuminations are shown in Fig. <ref>-(b). 
We project the generated 3D scene by a virtual camera onto a 2D plane upon Eq. (<ref>), with 
=
[   280     0   128;     0 497.7   128;     0     0     1 ]
. The background in every image is of the same opaque colour. 
We split the data into training and test subsets in a repetitive manner, see Fig. <ref> for the pattern. 
We train HDM-Net jointly on several textures and illuminations, with the purpose of illumination-invariant and texture-invariant 
regression. One illumination and one texture are reserved for the test dataset exclusively. 
Our images are of the dimensions 256 × 256. They reside in 15.2 Gb of memory, and the ground truth geometry 
requires 1.2 Gb (in total, 16.4 Gb). The hardware configuration consists of two six-core processors 
Intel(R) Xeon(R) CPU E5-1650 v4 running at 3.60GHz, 16 GB RAM and a GEFORCE GTX 1080Ti GPU with 11GB of global memory. 
In total, we train for 95 epochs, and the training takes two days in pytorch <cit.>. 
The evolution of the loss energy is visualised in Fig. <ref>-(a). 
The inference of one state takes ca. 5 ms.






§ GEOMETRY REGRESSION AND COMPARISONS




 
    < g r a p h i c s >
 
   Selected reconstruction results on endoscopically textured surfaces for HDM-Net (our method) and Yu <cit.>.  
   



We compare our method with the template-based reconstruction of Yu <cit.>, variational NRSfM approach (VA) of Garg <cit.> 
and NRSfM method of Golyanik <cit.> — Accelerated Metric Projections (AMP). 
We use an optimised heterogeneous CPU-GPU version of VA written in C++ and CUDA C <cit.>. 
AMP is a C++ CPU version which relies on an efficient solution of a semi-definite programming 
problem and is currently one of the fastest batch NRSfM methods. 
For VA and AMP, we compute required dense point tracks.
Following the standard praxis in NRSfM, we project the ground truth shapes onto a virtual 
image plane by a slowly moving virtual camera. Camera rotations are parametrised 
by Euler angles around the x-, y- and z-axes. 
We rotate for up to 20 degrees around each axis, with five degrees per frame.
This variety in motion yields minimal depth changes required for an accurate initialisation in NRSfM. 
We report runtimes, 3D error

    e_3D = 1/F∑_f = 1^F‖_f^GT - _f‖_ℱ/‖_f^GT‖_ℱ

and standard deviation σ of e_3D. 
Before computing e_3D, we align _f and the corresponding _f^GT with Procrustes analysis. 

Runtimes, e_3D and σ for all three methods are summarised in Table <ref>. 
AMP achieves around 30 fps and can execute only for 100 frames per batch at a time. 
However, this estimate does not include often prohibitive 
computation time of dense correspondences with multi-frame optical flow methods such as <cit.>. 
Note that runtime of batch NRSfM depends on the batch size, and the batch size 
influences the accuracy and ability to reconstruct.
VA takes advantage of a GPU and executes with 2.5 fps. 
Yu <cit.> achieves around 0.3 fps. In contrast, HDM-Net processes one frame 
in only 5 ms. This is by far faster than the compared methods. 
Thus, HDM-Net can compete in runtime with rigid structure from motion 
<cit.>. The runtime of the latter method is still considered as the lower runtime bound for NRSfM[when executed 
in a batch of 100 frames with 73^2 points each, a C++ version of <cit.> takes 
1.47 ms per frame on our hardware; for 400 frames long batch, it requires 5.27 ms per frame. ]. 

At the same time, the accuracy of HDM-Net is the highest among all tested methods. 
Selected results with complex deformations are shown in Fig. <ref>. 
We see that Yu <cit.> copes well with rather small 
deformations, and our approach accurately resolves even challenging cases not exposed during the training.
In the case of Yu <cit.>, 
the high e_3D is explained by a weak handling of self-occlusions and large deformations.
In the case of NRSfM methods, the reason for the high e_3D is an inaccurate initialisation. Moreover, VA 
does not handle foldings and large deformations well. 




 


Table <ref> summarises e_3D for our method under different illumination conditions. 
We notice that our network copes well with all generated illuminations —
the difference in e_3D is under 3%. Table <ref> shows e_3D comparison for different 
textures. Here, the accuracy of HDM-Net drops on the previously unseen texture by the factor of three, which still corresponds 
to reasonable reconstructions with the captured main deformation mode. 
Another quantitative comparison is shown in Fig. <ref>. In this example, all methods execute on the first 100 frames of the sequence. 
AMP <cit.> captures the main deformation mode with e_3D = 0.1564 but struggles to perform a fine-grained distinction 
(in Table <ref>, e_3D is reported over the sequence of 400 frames, hence the differing metrics). 
VA suffers under an inaccurate initialisation under rigidity assumption and Yu <cit.>, by contrast,
does not recognise the variations in the structure.
All in all, HDM-Net copes well with self-occlusions. Graphs of e_3D as functions of the state index under varying illuminations and textures 
can be found in Fig. <ref>-(b,c). Table <ref> shows the comparison of e_3D using networks trained with various combinations of loss functions. 3D + Con. shows the lowest e_3D and applying isometry prior increases e_3D. Since isometry prior is smoothing loss, the 3D grid becomes smaller in comparison to the outputs without isometry prior hence higher e_3D.  However, as shown in Fig. <ref>, isometry prior allows the network to generate smoother 3D geometries preserving deformation states. 


 
Next, we evaluate the performance of HDM-Net on noisy input images. Therefore, we augment the dataset with increasing amounts of uniform 
salt-pepper noise. Fig. <ref>-(d) shows the evolution of the e_3D as a function of the amount of noise, 
for several exemplary frames corresponding to different input difficulties for the network. We observe that HDM-Net is well-posed w.r.t 
noise — starting from the respective values obtained for the noiseless images, the e_3D increases gradually. 



We tested HDM-Net on several challenging real images. 
Fig. <ref> shows the tested images and our reconstructions. We recorded a music note image for an evaluation of our network in real-world scenario.
Despite different origin of the inputs (music notes, a fabric <cit.>, an endoscopic view during a surgery <cit.> and an air balloon <cit.>), HDM-Net produces realistic and plausible results.
Note how different are the regressed geometries which suggests the generalisation ability of the proposed solution. 

In many real-world cases, HDM-Net produces acceptable results. 
However, if the observed states differ a lot from the states in the training data, HDM-Net might fail to recognise and regress the state. This can be addressed by an extension or tailoring of the data set for specific cases. Adding training data originating 
from motion and geometry capture of real objects might also be an option. 



 






§ CONCLUDING REMARKS
 

We have presented a new monocular surface recovery method with a deformation model replaced by a DNN — HDM-Net. 
The new method reconstructs time-varying geometry from a single image and is robust to self-occlusions, changing 
illumination and varying texture. Our DNN architecture consists of an encoder, a latent space and a decoder, and is furnished with 
three domain-specific losses. Apart from the conventional 3D data loss, we propose isometry and reprojection losses. 
We train HDM-Net with a newly generated dataset with ca. four an a half thousands states, four different illuminations, five different camera poses 
and three different textures. Experimental results show the validity of our approach and its suitability for 
reconstruction of small and moderate isometric deformations under self-occlusions. Comparisons with one template-based and two template-free 
methods have demonstrated a higher accuracy in favour of HDM-Net. 
Since HDM-Net is one of the first approach of the new kind, there are multiple avenues for investigations and improvements. One apparent
direction is the further augmentation of the test dataset with different backgrounds, textures and illuminations. Next, 
we are going to test more advanced architectures such as generative adversarial networks and 
recurrent connections for the enhanced temporal smoothness. Currently, we are also investigating the relevance of HDM-Net for medical applications 
with augmentation of soft biological tissues. 





§ ACKNOWLEDGEMENT

Development of HDM-Net was supported by the project DYMANICS (01IW15003) of the German Federal Ministry of Education and Research (BMBF). The authors thank NVIDIA Corporation for the hardware donations.





splncs04

