Multi-View Silhouette and Depth Decomposition for High Resolution 3D Object Representation
Edward Smith Scott Fujimoto David Meger


*
    Saman Bashbaghi, Eric Granger, Robert Sabourin and Mostafa Parchami
    
=======================================================================




We consider the problem of scaling deep generative shape models to high-resolution. Drawing motivation from the canonical view representation of objects, we introduce a novel method for the fast up-sampling of 3D objects in voxel space through networks that perform super-resolution on the six orthographic depth projections. This allows us to generate high-resolution objects with more efficient scaling than methods which work directly in 3D. We decompose the problem of 2D depth super-resolution into silhouette and depth prediction to capture both structure and fine detail. This allows our method to generate sharp edges more easily than an individual network. We evaluate our work on multiple experiments concerning high-resolution 3D objects, and show our system is capable of accurately predicting novel objects at resolutions as large as 512×512×512 – the highest resolution reported for this task. We achieve state-of-the-art performance on 3D object reconstruction from RGB images on the ShapeNet dataset, and further demonstrate the first effective 3D super-resolution method. 




§ INTRODUCTION

The 3D shape of an object is a combination of countless physical elements that range in scale from gross structure and topology to minute textures endowed by the material of each surface. Intelligent systems require representations capable of modeling this complex shape efficiently, in order to perceive and interact with the physical world in detail (e.g., object grasping, 3D perception, motion prediction and path planning). Deep generative models have recently achieved strong performance in hallucinating diverse 3D object shapes, capturing their overall structure and rough texture <cit.>. The first generation of these models utilized voxel representations which scale cubically with resolution, limiting training to only 64^3 shapes on typical hardware. Numerous recent papers have begun to propose high resolution 3D shape representations with better scaling, such as those based on meshes, point clouds or octrees but these often require more difficult training procedures and customized network architectures. 

Our 3D shape model is motivated by a foundational concept in 3D perception: that of canonical views. The shape of a 3D object can be completely captured by a set of 2D images from multiple viewpoints (see <cit.> for an analysis of selecting the location and number of viewpoints). Deep learning approaches for 2D image recognition and generation <cit.> scale easily to high resolutions. This motivates the primary question in this paper: can a multi-view representation be used efficiently with modern deep learning methods? 

We propose a novel approach for deep shape interpretation which captures the structure of an object via modeling of its canonical views in 2D as depth maps, in a framework we refer to as Multi-View Decomposition (MVD). By utilizing many 2D orthographic projections to capture shape, a model represented in this fashion can be up-scaled to high resolution by performing semantic super-resolution in 2D space, which leverages efficient, well-studied network structures and training procedures. The higher resolution depth maps are finally merged into a detailed 3D object using model carving.

Our method has several key components that allow effective and efficient training. We leverage two synergistic deep networks that decompose the task of representing an object's depth: one that outputs the silhouette – capturing the gross structure; and a second that produces the local variations in depth – capturing the fine detail. This decomposition addresses the blurred images that often occur when minimizing reconstruction error by allowing the silhouette prediction to form sharp edges. Our method utilizes the low-resolution input shape as a rough template which simply needs carving and refinement to form the high resolution product.
Learning the residual errors between this template and the desired high resolution shape simplifies the generation task and allows for constrained output scaling, which leads to significant performance improvements.  

We evaluate our method's ability to perform 3D object reconstruction on the the ShapeNet dataset <cit.>. This standard evaluation task requires generating high resolution 3D objects from single 2D RGB images. Furthermore, due to the nature of our pipeline we present the first results for 3D object super-resolution – generating high resolution 3D objects directly from low resolution 3D objects. Our method achieves state-of-the-art quantitative performance, when compared to a variety of other 3D representations such as octrees, mesh-models and point clouds. 
Furthermore, our system is the first to produce 3D objects at 512^3 resolution. We demonstrate these objects are visually impressive in isolation, and when compared to the ground truth objects. We additionally demonstrate that objects reconstructed from images can be placed in scenes to create realistic environments, as shown in figure <ref>. In order to ensure reproducible experimental comparison, code for our system has been made publicly available on a GitHub repository[https://github.com/EdwardSmith1884/Multi-View-Silhouette-and-Depth-Decomposition-for-High-Resolution-3D-Object-Representation]. Given the efficiency of our method, each experiment was run on a single NVIDIA Titan X GPU in the order of hours. 



§ RELATED WORK




Deep Learning with 3D Data   Recent advances with 3D data have leveraged deep learning, beginning with architectures such as 3D convolutions for object classification <cit.>. 
When adapted to 3D generation, these methods typically use an autoencoder network, with a decoder composed of 3D deconvolutional layers <cit.>. This decoder receives a latent representation of the 3D shape and produces a probability for occupancy at each discrete position in 3D voxel space. This approach has been combined with generative adversarial approaches <cit.> to generate novel 3D objects <cit.>, but only at a limited resolution.

2D Super-Resolution   Super-resolution of 2D images is a well-studied problem <cit.>. 
Traditionally, image super-resolution has used dictionary-style methods <cit.>, matching patches of images to higher-resolution counterparts. This research also extends to depth map super-resolution <cit.>. Modern approaches to super-resolution are built on deep convolutional networks <cit.> as well as generative adversarial networks <cit.> which use an adversarial loss to imagine high-resolution details in RGB images.

Multi-View Representation   Our work connects to multi-view representations which capture the characteristics of a 3D object from multiple viewpoints in 2D <cit.>, such as decomposing image silhouettes <cit.>, Light Field Descriptors <cit.>, and 2D panoramic mapping <cit.>. 
Other representations aim to use orientation <cit.>, rotational invariance <cit.> or 3D-SURF features <cit.>. While many of these representations are effective for 3D classification,
they have not previously been utilized to recover 3D shape in high resolution.


Efficient 3D Representations   Given that naïve representations of 3D data require cubic computational costs with respect to resolution, many alternate representations have been proposed. Octree methods <cit.> use non-uniform discretization of the voxel space to efficiently capture 3D objects by adapting the discretization level locally based on shape. Hierarchical surface prediction (HSP) <cit.> is an octree-style method which divides the voxel space into free, occupied and boundary space. The object is generated at different scales of resolution, where occupied space is generated at a very coarse resolution and the boundary space is generated at a very fine resolution. Octree generating networks (OGN) <cit.> use a convolutional network that operates directly on octrees, rather than in voxel space. These methods have only shown novel generation results up to 256^3 resolution. Our method achieves higher accuracy at this resolution and can efficiently produce novel objects as large as 512^3.

A recent trend is the use of unstructured representations such as mesh models <cit.> and point clouds <cit.> which represent the data by an unordered set with a fixed number of points. MarrNet <cit.>, which resembles our work, models 3D objects through the use of 2.5D sketches, which capture depth maps from a single viewpoint. This approach requires working in voxel space when translating 2.5D sketches to high resolution, while our method can work directly in 2D space, leveraging 2D super-resolution technology within the 3D pipeline. 





§ METHOD


In this section we describe our methodology for representing high resolution 3D objects. Our algorithm is a novel approach which uses the six axis-aligned orthographic depth maps (ODM), to efficiently scale 3D objects to high resolution without directly interacting with the voxels. To achieve this, a pair of networks is used for each view, decomposing the super-resolution task into predicting the silhouette and relative depth from the low resolution ODM. This approach is able to recover fine object details and scales better to higher resolutions than previous methods, due to the simplified learning problem faced by each network, and scalable computations that occur primarily in 2D image space.  



 §.§ Orthographic Depth Map Super-Resolution
 
 


Our method begins by obtaining the orthographic depth maps of the six primary views of the low-resolution 3D object. In an ODM, each pixel holds a value equal to the surface depth of the object along the viewing direction at the corresponding coordinate. This projection can be computed quickly and easily from an axis-aligned 3D array via z-clipping. Super-resolution is then performed directly on these ODMs, before being mapped onto the low resolution object to produce a high resolution object. 

Representing an object by a set of depth maps however, introduces a challenging learning problem, which requires both local and global consistency in depth. Furthermore, minimizing the mean squared error results in blurry images without sharp edges <cit.>. This is particularly problematic as a depth map is required to be bimodal, with large variations in depth to create structure and small variations in depth to create texture and fine detail. To address this concern, we propose decomposing the learning problem into predicting the silhouette and depth map separately. Separating the challenge of predicting gross shape from fine detail regularizes and reduces the complexity of the learning problem, leading to improved results when compared with directly estimating new surface depths. 

Our Multi-View Decomposition framework (MVD) uses a set of twin of deep convolutional models f_SIL and f_ΔD, to separately predict silhouette and variations in depth of the higher resolution ODM. We depict our system in figure <ref>. The deep convolutional network for predicting the high-resolution silhouette, f_SIL with parameters θ, is passed the low resolution ODM D_L, extracted from the input 3D object. The network outputs a probability that each pixel is occupied. It is trained by minimizing the mean squared error between the predicted and true silhouette of the high resolution ODM D_H:

    ℒ(θ) = ∑_i=1^N ‖ f_SIL(D_L^(i); θ) - 1_D_H^(i)≠ 0 (D_H^(i)) ‖_2,

where 1_D_H^(i)≠ 0 is an indicator function for each coordinate in the image.

The same low-resolution ODM D_L is passed through the second deep convolution neural network, denoted f_ΔD with parameters ϕ, whose final output is passed through a sigmoid, to produce an estimate for the variation of the ODM within a fixed range r. This output is added to the low-resolution depth map to produce our prediction for a constrained high-resolution depth map C_H: 

    C_H = r σ(f_ΔD(D_L; ϕ)) + g(D_L),

where g(·) denotes up-sampling using nearest neighbor interpolation.
 

We train our network f_ΔD by minimizing the mean squared error between our prediction and the ground truth high-resolution depth map D_H. During training only, we mask the output with the ground truth silhouette to allow effective focus on fine detail for f_ΔD. We further add a smoothing regularizer which penalizes the total variation V(x) = ∑_i,j√((x_i+1,j - x_i,j)^2 + (x_i,j+1 - x_i,j)^2) <cit.> within the predicted ODM. Our loss function is a simple combination of these terms:

    ℒ(ϕ) = ∑_i=1^N ‖ (C_H^(i)∘1_D_H^(i)(j,k) ≠ 0(D_H^(i))) - D_H^(i)‖_2 + λ V(C_H^(i)),

where ∘ is the Hadamard product. The total variation penalty is used as an edge-preserving denoising which smooths out irregularities in the output. 
 
The output of the constrained depth map and silhouette networks are then combined to produce a complete prediction for the high-resolution ODM. This accomplished by masking the constrained high-resolution depth map by the predicted silhouette: 

    D̂_H = C_H ∘ f_SIL(D_L; θ).

D̂_H denotes our predicted high resolution ODM which can then be mapped back onto the original low resolution object by model carving to produce a high resolution object. Each of the 6 high resolution ODMS are predicted using the same 2 network models, with the side information for each passed using a forth channel in the corresponding low resolution ODM passed to the networks. 



 §.§ 3D Model Carving
 

To complete our super-resolution procedure, the six ODMs are combined with the low-resolution object to form a high-resolution object. This begins by further smoothing the up-sampled ODM with an adaptive averaging filter, which only consider neighboring pixels within a small radius. To preserve edges, only neighboring pixels within a threshold of the value of the center pixel are included. This smoothing, along with the total variation regularization in the our loss function, are added to enforce smooth changes in local depth regions. 

Model carving begins by first up-sampling the low-resolution model to the desired resolution, using nearest neighbor interpolation. We then use the predicted ODMs D̂_H = C_H ∘ f_SIL(D_L; θ) to determine the surface of the new object. The carving procedure is separated into (1) structure carving, corresponding to the silhouette prediction f_SIL(D_L; θ), and (2) detail carving, corresponding to the constrained depth prediction C_H. 

For the structure carving, for each predicted ODM f_SIL(D_L; θ), if a coordinate is predicted unoccupied, then all voxels perpendicular to the coordinate are highlighted to be removed. The removal only occurs if there is agreement of at least two ODMs for the removal of a voxel. As there is a large amount of overlap in the surface area that the six ODMs observe, this silhouette agreement is enforced to maintain the structure of the object. 

This same process occurs for detail carving with C_H. However, we do not require agreement within the constrained depth map predictions. This is because, unlike the silhouettes, a depth map can cause or deepen concavities in the surface of the object which may not be visible from any other face. Requiring agreement among depth maps would eliminate their ability to influence these concavities. Thus, performing detail carving simply involves removing all voxels perpendicular to each coordinate of each ODM, up to the predicted depth. 



§ EXPERIMENTS

In this section we present our results for our method, Multi-View Decomposition Networks (MVD), for both 3D object super-resolution and 3D object reconstruction from single RGB images. 
Our results are evaluated across 13 classes of the ShapeNet <cit.> dataset. 3D super-resolution is the task of generating a high resolution 3D object conditioned on a low resolution input, while 3D object reconstruction is the task of re-creating high resolution 3D objects from a single RGB image of the object. 



 §.§ 3D Object Super-Resolution
 






Dataset   The dataset consists of 32^3 low resolution voxelized objects and their 256^3 high resolution counterparts. These objects were produced by converting CAD models found in the ShapeNetCore dataset <cit.> into voxel format, in a canonical view. We work with the three commonly used object classes from this dataset: Car, Chair and Plane, with around 8000, 7000, 4000 objects respectively. For training, we pre-process this dataset, to extract the six ODMs from each object at high and low-resolution. CAD models converted at this resolution do not remain watertight in many cases, making it difficult to fill the inner volume of the object. We describe an efficient method for obtaining high resolution voxelized objects in the supplementary material. Data is split into training, validation, and test set using a ratio of 70:10:20 respectively.


Evaluation   We evaluate our method quantitatively using the intersection over union metric (IoU) against a simple baseline and the prediction of the individual networks on the test set. The baseline method corresponds to the ground truth at 32^3 resolution, by up-scaling to the high resolution using nearest neighbor up-sampling. While our full method, MVD, uses a combination of networks, we present an ablation study to evaluate the contribution of each separate network. 


Implementation   The super-resolution task requires a pair of networks, f_ΔD and f_SIL, which share the same architecture. This architecture is derived from the generator of SRGAN <cit.>, a state of the art 2D super-resolution network. Exact network architectures and training regime are provided in the supplementary material. 

Results   The super-resolution IoU scores are presented in table <ref>. Our method greatly outperforms the naïve nearest neighbor up-sampling baseline in every class. While we find that the silhouette prediction contributes far more to the IoU score, the addition of the depth variation network further increases the IoU score. This is due to the silhouette capturing the gross structure of the object from multiple viewpoints, while the depth variation captures the fine-grained details, which contributes less to the total IoU score. To qualitatively demonstrate the results of our super-resolution system we render objects from the test set at both 256^3 resolution in figure <ref> and 512^3 resolution in figure <ref>. The predicted high-resolution objects are all of high quality and accurately mimic the shapes of the ground truth objects. Additional 512^3 renderings as well as multiple objects from each class at 256^3 resolution can be found in our supplementary material.



 §.§ 3D Object Reconstruction from RGB Images


 


Dataset   To match the datasets used by prior work, two datasets are used for 3D object reconstruction, both derived from the ShapeNet dataset. The first, which we refer to as Data_HSP, consists of only the Car, Chair and Plane classes from the Shapenet dataset, and we re-use the 32^3 and 256^3 voxel objects produced for these classes in the previous section. The CAD models for each of these object were rendered into 128^2 RGB images capturing random viewpoints of the objects at elevations between (-20^∘, 30^∘) and all possible azimuth rotations. The voxelized objects and corresponding images were split into a training, validation and test set, with a ratio of 70:10:20 respectively.

The second dataset, which we refer to as Data_3D-R2N2, is that provided by <cit.>. It consists of images and objects produced from  the 3 classes in the ShapeNet dataset used in the previous section, as well as 10 additional classes, for a total of around 50000 objects. From each object 137^2 RGB images are rendered at random viewpoints, and we again compute their 32^3 and 256^3 resolution voxelized models and ODMs. The data is split into a training, validation and test set with a ratio of 70:10:20. 

Evaluation   We evaluate our method quantitatively with two evaluation schemes. In the first, we use IoU scores when reconstructing objects at 256^3 resolution. We compare against HSP <cit.> using the first dataset Data_HSP, and against OGN <cit.> using the second dataset Data_3D-R2N2. To study the effectiveness of our super-resolution pipeline, we also compute the IoU scores using the low resolution objects predicted by our autoencoder (AE) with nearest neighbor up-sampling to produce predictions at 256^3 resolution. 

Our second evaluation is performed only on the second dataset, Data_3D-R2N2, by comparing the accuracy of the surfaces of predicted objects to those of the ground truth meshes. Following the evaluation procedure defined by <cit.>, we first convert the 256^3 voxel models into meshes by defining squared polygons on all exposed faces on the surface of the voxel models. We then uniformly sample points from the two mesh surfaces and compute F1 scores. Precision and recall are calculated using the percentage of points found with a nearest neighbor in the ground truth sampling set less than a squared distance threshold of 0.0001. We compare to state of the art mesh model methods, N3MR <cit.> and Pixel2Mesh <cit.>, a point cloud method, PSG <cit.>, and a voxel baseline, 3D-R2N2 <cit.>, using the values reported by <cit.>.

Implementation   For 3D object reconstruction, we first trained a standard autoencoder, similar to prior work <cit.>, to produce objects at 32^3 resolution. These low resolution objects are then used with our 3D super-resolution method, to generate 3D object reconstructions at a high 256^3 resolution. This process is described in figure <ref>. The exact network architecture and training regime are provided in the supplementary material. 







Results   The results of our IoU evaluation compared to the octree methods <cit.> can be seen in table <ref>. We achieve state-of-the-art performance on every object class in both datasets. Our surface accuracy results can be seen in table <ref> compared to <cit.>. Our method achieves state of the art results on all 13 classes. We show significant improvements for many object classes and demonstrate a large improvement on the mean over all classes when compared against the methods presented. To qualitatively evaluate our performance, we rendered our reconstructions for each class, which can be seen in figure <ref>. Additional renderings can be found in the supplementary material. 



§ CONCLUSION


In this paper we argue for the application of multi-view representations when predicting the structure of objects at high resolution. We outline our Multi-View Decomposition framework, a novel system for learning to represent 3D objects and demonstrate its affinity for capturing category-specific shape details at a high resolution by operating over the six orthographic projections of the object. 

In the task of super-resolution, our method outperforms baseline methods by a large margin, and we show its ability to produce objects as large as 512^3, with a 16 times increase in size from the input objects. The results produced are visually impressive, even when compared against the ground-truth. When applied to the reconstruction of high-resolution 3D objects from single RGB images, we outperform several state of the art methods with a variety of representation types, across two evaluation metrics. 

All of our visualizations demonstrate the effectiveness of our method at capturing fine-grained detail, which is not present in the low resolution input but must be captured in our network's weights during learning. Furthermore, given that the deep aspect of our method works entirely in 2D space, our method scales naturally to high resolutions. This paper demonstrates that multi-view representations along with 2D super-resolution through decomposed networks is indeed capable of modeling complex shapes. 

plainnat


[pages=1-,scale=1]images/supplementary-material.pdf


