LIMP: Learning Latent Shape Representations with Metric Preservation Priors
Luca Cosmo Antonio Norelli Oshri Halimi Ron Kimmel Emanuele Rodolà
cs.LG cs.CG cs.GR stat.ML


headings
4866  




LIMP: Latent Interpolation with Metric Priors



L. Cosmo et al.

Sapienza University of Rome, Italy
University of Lugano, Switzerland
Technion - Israel Institute of Technology, Israel

LIMP: Learning Latent Shape Representations with Metric Preservation Priors
    Luca Cosmo1,2, Antonio Norelli1, Oshri Halimi3, Ron Kimmel3,
 Emanuele Rodolà1
    December 30, 2023
==================================================================================





In this paper, we advocate the adoption of metric preservation as a powerful prior for learning latent representations of deformable 3D shapes. 
Key to our construction is the introduction of a geometric distortion criterion, defined directly on the decoded shapes, translating the preservation of the metric on the decoding to the formation of linear paths in the underlying latent space. Our rationale lies in the observation that training samples alone are often insufficient to endow generative models with high fidelity, motivating the need for large training datasets. In contrast, metric preservation provides a rigorous way to control the amount of geometric distortion incurring in the construction of the latent space, leading in turn to synthetic samples of higher quality. We further demonstrate, for the first time, the adoption of differentiable intrinsic distances in the backpropagation of a geodesic loss. Our geometric priors are particularly relevant in the presence of scarce training data, where learning any meaningful latent structure can be especially challenging. The effectiveness and potential of our generative model is showcased in applications of style transfer, content generation, and shape completion.







§ INTRODUCTION


Constructing high-fidelity generative models for 3D shapes is a challenging problem that has met with increasing interest in recent years. 
Generative models are applicable in many practical domains, ranging from content creation to shape exploration, as well as in 3D reconstruction.
As a new generation of methods, they come to face a number of difficulties.
 
Most existing approaches address the case of static or rigid geometry, for example, man-made objects like chairs and airplanes, with potentially high intra-class variability; see the ShapeNet <cit.> repository for such examples. 
In this setting, the main focus has been on the abstraction capabilities of the encoder and the generator, describing complex 3D models in terms of their core geometric features via parsimonious part-based representations. 
Shapes generated with these techniques are usually designed to have valid part semantics that are easy to parse. Concurrently, several recent efforts have concentrated on the definition of convenient representations for the 3D output; these methods find broader application in multiple tasks, where they enable more efficient and high-quality synthesis, and can be often plugged into existing generative models.

To date, relatively fewer approaches have targeted the deformable setting, where the generated shapes are related by continuous, non-rigid deformations. 
These model a range of natural phenomena, such as changes in pose and facial expressions of human subjects, articulations, garment folding, and molecular flexibility to name but a few. 
The extra difficulties brought by such non-rigid deformations can be tackled, in some cases, by designing mathematical or parametric models for the deformation at hand; however, these models are often violated in practice, and can be very hard to devise for general deformations – hence the need for learning from examples. 




The framework we propose is motivated by the observation that existing data-driven approaches for learning deformable 3D shapes, and autoencoders (AE) in particular, do not make use of any geometric prior to drive the construction of the latent space, whereas they rely almost completely on the expressivity of the training dataset. 
This imposes a heavy burden on the learning process, and further requires large annotated datasets that can be costly or even impossible to acquire.

In the absence of additional regularization, limited training data leads to limited generalization capability, which is manifested in the generated 3D shapes exhibiting unnatural distortions.

Variational autoencoders (VAE) provide a partial remedy by modeling a distributional prior on the data via a parametrized density on the latent space. 
This induces additional regularization, but is still insufficient to guarantee the preservation of geometric properties in the output 3D models.


In this paper, we introduce Latent Interpolation with Metric Priors (LIMP).
We propose to explicitly model the local metric properties of the latent space by enforcing metric constraints on the decoded output. 
We do this by phrasing a metric distortion penalty that has the effect to promote naturally looking deformations, and in turn to significantly reduce the need for large datasets at training time. 

In particular, we show that by coupling the Euclidean distances among latent codes (hence, along linear paths in the latent space) to the metric distortion among decoded shapes, we obtain a strong regularizing effect in the construction of the latent space.

Another novel ingredient of the proposed approach is the backpropagation of intrinsic (namely, geodesic) distances during training, which is made possible by a recent geodesic computation technique. 
Using geodesics makes our approach more flexible, and enables the successful application of our generative model to style and pose transfer applications.

See Figure <ref> for an example of novel samples synthesized with our generative model.



§ RELATED WORK


Our method falls within the class of AE-based generative models for 3D shapes. In this Section we cover methods from this family that are more closely related to ours, and refer to the recent survey <cit.> for a broader coverage.

In the 3D computer vision and graphics realms, generative models for part-compositional 3D objects play the lion's share. Such approaches directly exploit the hierarchical, structural nature of 3D man-made objects to drive the construction of encoder and generator <cit.>. These methods leverage on the insight that objects can be understood through their components <cit.>, making an interpretable representation close to human parsing possible. In this setting, a continuous exploration of the generated latent spaces is not always meaningful; the mechanism underlying typical operations like sampling and interpolation happen instead in discrete steps in order to generate plausible intermediate shape configurations (e.g., for transitioning from a 4-legged chair to a 3-legged stool). 

For this reason, with rigid geometry one usually deals with “structural blending” rather than continuous deformations.

Structural blending has been realized, for instance, by learning abstractions of symmetry hierarchies via spatial arrangements of oriented bounding boxes <cit.>, or by explicitly modeling part-to-part relationships <cit.>; generative-adversarial modeling has been applied on volumetric object representations <cit.>; structural hierarchies have been applied for the generation of composite 3D scenes <cit.> and building typologies <cit.> as well. Contributing to their success, is the fact that all these methods train on ShapeNet-scale annotated datasets with >50K unique 3D models, and the recent publication of dedicated benchmarks like PartNet <cit.> testify to the increasing interest of data-driven models for structure-aware geometry processing.

In this paper, we address a different setting; we do not assume part-compositionality of the 3D models since we deal with deformable shapes, where continuous deformations are well-defined, and where annotated datasets are not as prominent.

A second thread of research revolves around the definition of a meaningful representation for the generated 3D output. While many approaches mostly use polygonal meshes with predefined topology or directly synthesize point clouds <cit.>, the focus has been recently shifting towards more effective representations in terms of overall quality, fidelity, and flexibility. These include approaches that predict implicit shape representations at the output, requiring an ex-post isosurface extraction step to generate a mesh at the desired resolution <cit.>; isosurfacing has been replaced by binary space partitioning in <cit.>; while in <cit.>, shapes are represented by a set of parametric surface elements. 

In this work, we focus on learning a better latent representation for deformable shapes, rather than on constructing a better representation for the output.

More closely related to ours are some recent methods from the area of geometric deep learning. A graph-convolutional VAE with dynamic filtering convolutional layers <cit.> was introduced in <cit.> for the task of deformable shape completion of human shapes. The method is trained on ∼7000 shapes from the DFAUST dataset of real human scans <cit.>; due to the lack of any geometric prior, the learned generator introduces large distortions around points in the latent space that are not well represented in the training set. Geometric regularization was injected in <cit.> in the form of a template that parametrizes the surface. The method shows excellent performance in shape matching, however, it crucially relies on a large and representative dataset of 230,000 shapes, and performance drops significantly with smaller training sets or bad initialization. More recently, a geometric disentanglement model for deformable point clouds was introduced in <cit.>. The proposed method uses Laplacian eigenvalues as a weak geometric prior to promote the separation of intrinsic and extrinsic shape information, together with several other de-correlation penalties, and a training set of >40K shapes. In the absence of enough training examples, the approach tends to produce a “morphing” effect between point clouds that does not correspond to a natural motion; a similar phenomenon was observed in <cit.>.

Finally, in <cit.>, a time-dependent physical prior was used to regularize interpolations in the latent space with the goal of obtaining a convincing simulation of moving tissues.

 In particular, our approach bears some analogies with the theory of shape spaces <cit.>, in that we seek to synthesize geometry that minimizes a deformation energy. For example, in <cit.> it was shown how to axiomatically modify a noisy shape such that its intrinsic measures would fit a given prior in a different pose. Differentiating the geodesic distances was done by fixing the update order in the fast marching scheme <cit.>. Our energy is not minimized over a fixed shape space, but rather, it drives the construction of a novel shape space in a data-driven fashion.

In this paper, we leverage classical ideas from shape analysis and metric geometry to ensure that shapes on the learned latent space correspond to plausible (i.e., low-distortion) deformations of the shapes seen at training time, even when only few training samples are available. We do this by modeling a geometric prior that promotes deformations with bounded distortion, and show that this model provides a powerful regularization for shapes within as well as across different classes, e.g., when transitioning between different human subjects.



§ LEARNING WITH METRIC PRIORS


Our goal is to learn a latent representation for deformable 3D shapes. We do this by training a VAE on a training set 𝒮={𝐗_i} of |𝒮| shapes, under a purely geometric loss:


    ℓ(𝒮) = ℓ_recon(𝒮) + ℓ_interp(𝒮) + ℓ_disent(𝒮) .


The loss is composed of three terms. The first is a geometric reconstruction loss on the individual training shapes, as in classical AE's; the second one is a pairwise interpolation term for points in the latent space; the third one is a disentanglement term to separate intrinsic from extrinsic information.

The main novelty lies in (1) the interpolation loss, and (2) the disentanglement loss not relying upon corresponding poses in the training set. The interpolation term provides control over the encoding of each shape in relation to the others. This induces a notion of proximity between latent codes that is explicitly linked, in the definition of the loss, to a notion of metric distortion between the decoded shapes. As we show in the following, this induces a strong regularization on the latent space and rules out highly distorted reconstructions.

The disentanglement loss promotes the factorization of the latent space into two orthogonal components: One that spans the space of isometries (e.g., change in pose), and another that spans the space of non-isometric deformations (e.g., change in identity). As in the interpolation loss, for the disentanglement we also exploit the metric properties of the decoded shapes.



 §.§ Losses


We define 𝐳:=enc(𝐗) to be the latent code for shape 𝐗, and 𝐗':=dec(𝐳) to be the corresponding decoding. During training, the decoder (dec) and encoder (enc) are updated so as to minimize the overall loss of Eq. (<ref>); see Section <ref> for the implementation details.



  
*Geometric reconstruction.

The reconstruction loss is defined as follows:


    ℓ_recon(𝒮) = ∑_i=1^|𝒮|𝐃_ℝ^3( 𝐗'_i)  - 𝐃_ℝ^3(𝐗_i) _F^2 ,


where 𝐃_ℝ^3(𝐗) is the matrix of pairwise Euclidean distances between all points in 𝐗, and ·_F denotes the Frobenius norm. Eq. (<ref>) measures the cumulative reconstruction error (up to a global rotation) over the training shapes.



  
*Metric interpolation.

This loss is defined over all possible pairs of shapes (𝐗_i,𝐗_j):


    ℓ_interp(𝒮) = ∑_i≠ j^|𝒮|𝐃(dec(
        (1-α)𝐳_i  + α𝐳_j_interpolation of
    latent codes
        )) - 
        (
        (1-α)𝐃( 𝐗_i' ) + α𝐃( 𝐗_j')_interpolation of
    geodesic or local distances
        ) _F^2 ,


where α∼𝒰(0,1) is a uniformly sampled scalar in (0,1), different for each pair of shapes. In the equation above, the matrix 𝐃(𝐗) encodes the pairwise distances between points in 𝐗. We use two different definitions of distance, giving rise to two different losses which we sum up together. In one loss, 𝐃 contains geodesic distances between all pairs of points. In the second loss, we consider local Euclidean distances from each point to points within a small neighborhood (set to 10% of the shape diameter); the rationale is that local Euclidean distances capture local detail and tend to be resilient to non-rigid deformations, as observed for instance in <cit.>.

All distances are computed on the fly, on the decoded shapes, at each forward step. 



Since the error criterion in Eq. (<ref>) encodes the discrepancy between pairwise distance matrices, we refer to it as a metric preservation prior. We refer to Section <ref> for a more in-depth discussion from a continuous perspective.



  
*Disentanglement.

We split the latent codes into an intrinsic and an extrinsic part, 𝐳:=(𝐳^int | 𝐳^ext). The former is used to encode “style”, i.e., the space of non-isometric deformations; the latter is responsible for changes in pose, and is therefore constrained to model the space of possible isometries.

The loss is composed of two terms:


    ℓ_disent(𝒮)    = ℓ_int (𝒮) + ℓ_ext (𝒮)  ,   with
    ℓ_int(𝒮)    = ∑_i≠ j
    iso^|𝒮|𝐃_ℝ^3 ( dec (  (1-α)𝐳_i^int + α𝐳_j^int_interpolation of
    style  | 𝐳_i^ext)) - 𝐃_ℝ^3(𝐗_i) _F^2 
    ℓ_ext(𝒮)    = ∑_i≠ j
    non-iso^|𝒮|𝐃_g ( dec ( 𝐳_i^int |  (1-α)𝐳_i^ext + α𝐳_j^ext_interpolation of
    pose )) - 𝐃_g(𝐗_i) _F^2


The ℓ_int term is evaluated only on isometric pairs (i.e., just a change in pose), for which we expect 𝐳_i^int=𝐳_j^int. For a pair (𝐗_i,𝐗_j), it requires that 𝐗_i can be reconstructed exactly even when its intrinsic part 𝐳_i^int is interpolated with that of 𝐗_j. This enforces 𝐳_i^int=𝐳_j^int, thus all the pose-related information is forced to move to 𝐳^ext.

The ℓ_ext term is instead evaluated on non-isometric pairs. Here we require that the geodesic distances of 𝐗_i are left untouched when we interpolate its pose with that of 𝐗_j. This way, we force all the style-related information to be moved to 𝐳^int. We see that by having direct access to the metric on the decoded shapes, we can phrase the disentanglement easily in terms of distances.



The assumption that the metric is nearly preserved under pose changes is widely used in many shape analysis applications such as shape retrieval <cit.>, matching <cit.> and reconstruction <cit.>




  
*Relative error.

In practice, we always measure the error on the Euclidean distances (appearing in Eqs. (<ref>),(<ref>),(<ref>)) in a relative sense. Let 𝐀 be the “ground truth” Euclidean distance matrix computed on the input shape, and let 𝐁 be its predicted reconstruction. Instead of taking  𝐀-𝐁_F^2=∑_ij (𝐀_ij-𝐁_ij)^2, we compute the relative error ∑_ij(𝐀_ij-𝐁_ij)^2/𝐀_ij^2.

In our experiments, this resulted in better reconstruction of local details than by using the simple Frobenius norm.








 §.§ Continuous interpretation


In the continuous setting, we regard shapes as metric spaces (,d_), each equipped with a distance function d_:×→ℝ_+. Given two shapes (,d_) and (,d_), a map ϕ:→ is an isometry if it is surjective and preserves distances, d_(x,x')=d_(ϕ(x),ϕ(x')) for all x,x'∈. Isometries play a fundamental role in 3D shape analysis, since they provide a mathematical model for natural deformations like changes in pose. In practice, however, isometry is rarely satisfied exactly. 



  
*Why interpolation?

Our approach is based on the insight that non-isometric shapes are related by sequences of near-isometric deformations, which, in turn, have a well defined mathematical model.

In our setting, we do not require the training shapes to be near-isometric. Instead, we allow for maps ϕ with bounded metric distortion, i.e., for which there exists a constant K>0 such that:


    |d_(x,x')-d_(ϕ(x),ϕ(x')) | ≤ K


for all x, x'∈. 

For K→ 0 the map ϕ is a near-isometry, while for general K>0 we get a much wider class of deformations, going well beyond simple changes in pose. We therefore assume that there exists a map with bounded distortion between all shape pairs in the training set.

At training time, we are given a map ϕ:→ between two training shapes (,d_) and (,d_). We then assume there exists an abstract metric space (ℒ, d_ℒ) where each point is a shape; this “shape space” is the latent space that we seek to represent when training our generative model.

Over the latent space we construct a parametric sequence of shapes 𝒵_α = (,d_α), parametrized by α∈(0,1), connecting (,d_) to (,d_). By modeling the intermediate shapes as (,d_α), we regard each 𝒵_α as a continuously deformed version of , with a different metric defined by the interpolation:


    d_α(x,x') = (1-α)d_(x,x') + α d_(ϕ(x),ϕ(x')) ,


for all x,x'∈. Each 𝒵_α in the sequence has the same points as , but the shape is different since distances are measured differently.

It is easy to see that if the training shapes  and  are isometric, then d_α(x,x')=d_(x,x') for all x,x'∈ and the entire sequence is isometric, i.e., we are modeling a change in pose. However, if ϕ:→ has bounded distortion without being an isometry, each intermediate shape (,d_α) also has bounded distortion with respect to (,d_), with K_α<K in Eq. (<ref>); in particular, for α→ 0 one gets K_α→ 0 and therefore a near-isometry. In other words, by using the metric interpolation loss of Eq. (<ref>), as α grows from 0 to 1 we are modeling a general non-isometric deformation as a sequence of approximate isometries.




[5]r0.5


  
		[trim=0cm 0cm 0cm 0cm,clip,width=0.9]./figures/arithmetic.pdf
		(16,15)-  
		(45,15)+  
		(74,15)=  
		




  
*Flattening of the latent space.

Taking a linear convex combination of latent vectors as in Eq. (<ref>) implies that distances between codes should be measured using the Euclidean metric ·_2. 

This enables algebraic manipulation of the codes and the formation of “shape analogies”, as shown in the inset (real example based on our trained model). 
 
By the connection of Euclidean distances in the latent space with intrinsic distances on the decoder's output, our learning model performs a “flattening” operation, in the sense that it requires the latent space to be as Euclidean as possible, while absorbing any embedding error in the decoder. A similar line of thought was followed, in a different context, in the purely axiomatic model of <cit.>.



 §.§ Implementation


We design our deep generative model as a VAE (Figure <ref>). The input data is a set of triangle meshes; each mesh is encoded as a matrix of vertex positions 𝐗∈ℝ^n× 3, together with connectivity encoded as a n× n adjacency matrix. 

We anticipate here that mesh connectivity is never accessed directly by the network.



  
*Architecture.

The encoder takes vertex positions 𝐗 as input, and outputs a d-dimensional code 𝐳=enc(𝐗). Similarly, the decoder outputs vertex positions 𝐘 = dec(𝐳) ∈ℝ^n× 3.

In order to clarify the role of our priors versus the sophisticacy of the architecture, we keep the latter as simple as possible.

In particular, we adopt a similar architecture as in <cit.>; we use PointNet <cit.> with spatial transform as the encoder, and a simple MLP as the decoder. We reserve 25% of the latent code for the extrinsic part and the remaining 75% for the intrinsic representation, while the latent space and layer dimensions vary depending on the dataset size. A detailed description of the network is deferred to the Supplementary Material.

We implemented our model in PyTorch using Adam as optimizer with learning rate of 1e-4. To avoid local minima and numerical errors in gradient computation, we start the training by optimizing just the reconstruction loss for 10^4 iterations, and add the remaining terms for the remaining epochs. 



  
*Geodesic distance computation.

A crucial ingredient to our model is the computation of geodesic distances 𝐃_g(dec(𝐳)) during training, see Eq. (<ref>). We use the heat method of <cit.> to compute these distances, based on the realization that its pipeline is fully differentiable. It consists, in particular, of two linear solves and one normalization step, and all the quantities involved in the three steps depend smoothly on the vertex positions given by the decoder (we refer to the Supplementary Material for additional details).

To our knowledge, this is the first time that on-the-fly computation of geodesic distances appears in a deep learning pipeline.
Previous approaches using geodesic distances, such as <cit.>, do so by taking them as pre-computed input data, and leave them untouched for the entire training procedure. 









  
*Supervision. 

We train on a collection of shapes with known pointwise correspondences; these are needed in  Eq. (<ref>), where we assume that the distance matrices have compatible rows and columns. From a continuous perspective, we need maps for the interpolated metric of Eq. (<ref>) to be well defined.

Known correspondences are also needed by other approaches dealing with deformable data <cit.>. In practice, we only need few such examples (we use <100 training shapes), since we rely for the most part on the regularization power of our geometric priors.

Differently from <cit.> we do not assume the training shapes to have the same mesh, since the latter is only used as an auxiliary structure for computing geodesics in the loss; the network only ever accesses vertex positions. Further, we do not require training shapes with similar poses across different subjects.











§ RESULTS




 §.§ Data




To validate our method, we performed experiments using 5 different datasets (3 are obtained from real-world scans, 2 are fully synthetic). 

FAUST <cit.> is composed of 10 different human subjects, each captured in 10 different poses. We train our network on 8 subjects (thus, 80 meshes in total) and leave out the other 2 subjects for testing.

DFAUST <cit.> is a 4D dataset capturing the motion of 10 human subjects performing 14 different activities, spanning hundreds of frames each. As training data we only use 4 representative frames from each subject/sequence pair.

COMA <cit.> is another 4D dataset of human faces; it is composed of 13 subjects, each performing 13 different facial expressions represented as a sequence of 3D meshes. As opposed to the test split proposed in <cit.>, where 90% of the data is used for training, we only select 14 frames for each subject (one representative for each of the 13 expressions, plus one in a neutral pose), thus training with less than 1% of the dataset. 
TOSCA <cit.> is a synthetic dataset containing both animals and human bodies. In our experiments we use only the cat class, containing 11 shapes in different poses.
The last dataset, which we refer to as HANDS, is also completely synthetic and consists of 5 meshes depicting one hand in 5 different poses.

For all the datasets, we subsample the meshes to 2500 vertices by iterative edge collapse <cit.>.



 §.§ Interpolation


We first perform a classical interpolation experiment. Given two shapes 𝐗 and 𝐘, we visualize the decoded interpolation of their latent codes, given by dec((1-α)enc(𝐗)+αenc(𝐘)) for a few choices of α∈ (0,1). 

We measure the interpolation quality via the interpolation error, defined as the average (over all surface points) geodesic distortion of the interpolated shapes.



Two examples of interpolation are shown in Figures <ref> and <ref>. In these examples, the training sets consist of just 5 and 11 shapes respectively, meaning that the intermediate poses have never been seen before. In this few-shot setting, proper regularization is crucial to get meaningful results.

In the experiment in Figure <ref>, we also conduct an ablation study. We disable all the interpolation terms from our complete loss, resulting in a baseline VAE; then we disable the geodesic regularization only; finally we keep the entire loss intact, showing best results.

Quantitative results on 4 different datasets are reported in Table <ref> (first 3 columns), showing that best results are obtained when our full loss is used.





As an additional qualitative experiment, in Figure <ref> we show the decoded shapes in-between two keyframes of a 4D sequence from DFAUST. We remark that none of the intermediate shapes were seen at training time, nor was any similar-looking shape present in the training set. We then compare our reconstructed sequence with the original sequence of real-world scans. The purpose of this experiment is to show that our geometric priors are essential for the generation of realistic motion; apart from a perceptual evaluation, any quantitative comparison here would not be meaningful – there is not a unique “true” way to transition between two given poses.



 §.§ Disentanglement


Our second set of experiments is aimed at demonstrating the effectiveness of our geometric priors for the disentanglement of intrinsic from extrinsic information. We illustrate this in different ways.

In Figure <ref>, we show disentanglement for a generator trained on the FAUST dataset. For visualization purposes, for each vector 𝐳:=(𝐳^int|𝐳^ext) in the latent space (here comprising both training and test shapes), we embed the 𝐳^int and 𝐳^ext parts separately onto the plane (via multidimensional scaling), and attribute different colors to different gender and poses. We then randomly sample two new 𝐳^int and two new 𝐳^ext, and compose them into four latent codes by taking all the combinations. The figure illustrates the four decoded shapes.



In Figure <ref> we show the simultaneous action of disentanglement and interpolation. Given a source and a target shape, we show the interpolation of pose while fixing the style, and the interpolation of style while fixing the pose. We do so with different combinations of source and target. In all cases, our generative model is able to synthesize realistic shapes with the correct semantics, suggesting high potential in style and pose transfer applications.

As we did with the case of interpolation, we also provide a notion of disentanglement error, defined as follows. Given shapes 𝐗_i and 𝐗_j with latent codes (𝐳^int_i | 𝐳^ext_i) and (𝐳^int_j | 𝐳^ext_j), we swap 𝐳^ext_i with 𝐳^ext_j and then measure the average point-to-point distance between dec(𝐳^int_i | 𝐳^ext_j) and the corresponding ground-truth shape from the dataset.

In Table <ref> (last 3 columns) we report the disentanglement error on all 4 datasets, together with the ablation study.

Finally, in Figure <ref> we show a qualitative comparison with the recent state-of-the-art method <cit.> (using public code provided by the authors), which uses Laplacian eigenvalues as a prior to drive the disentanglement, together with multiple other de-correlation terms. Similarly to other approaches like <cit.>, the quality of the interpolation of <cit.> mostly depends on the smoothness properties of the VAE, on the complexity of the deep net, or on the availability of vast training data. For this comparison, both generative models were trained on the same 80 FAUST shapes.






§ CONCLUSIONS


We introduced a new deep generative model for deformable 3D shapes. Our model is based on the intuition that by directly connecting the Euclidean distortion of latent codes to the metric distortion of the decoded shapes, one gets a powerful regularizer that induces a well-behaved structure on the latent space. Our idea finds a theoretical interpretation in modeling deformations with bounded metric distortion as sequences of approximate isometries.

Under the manifold hypothesis, our metric preservation priors explicitly promote a flattening of the true data manifold onto a lower-dimensional Euclidean representation. 

We demonstrated how having access to the metric of the decoded shapes during training enables high-quality synthesis of novel samples, with practical implications in tasks of content creation and style transfer.

Perhaps the main limitation of our method, which we share with other geometric deep learning approaches, lies in the requirement of labeled pointwise correspondences between the training shapes. These can be hard to obtain in certain settings, for example, when dealing with shapes from the same semantic class but with high intra-class variability. 

Few interesting directions of future work may consist in a self-supervised variant of our model, where dense correspondences are not needed for the training, but are estimated during the learning process 

or in the exploitation of spectral properties of the reconstructed shape, that has been shown <cit.> to contain important information of the embedding geometry.

Finally, while in this paper we showed that even a simple prior such as metric distortion can have a significant effect, we foresee that bringing techniques from the areas of shape optimization and analysis closer to deep generative models will enable a fruitful line of stimulating research. 



  §.§.§ Acknowledgments.


LC, AN and ER are supported by the ERC Starting Grant No. 802554 (SPECGEO) and the MIUR under grant “Dipartimenti di eccellenza 2018-2022” of the Department of Computer Science of Sapienza University. OH and RK are supported by the Israel Ministry of Science and Technology grant number 3-14719, the Technion Hiroshi Fujiwara Cyber Security Research Center and the Israel Cyber Directorate.


splncs04


