SceneCAD: Predicting Object Alignments and Layouts in RGB-D Scans
Armen Avetisyan Tatiana Khanova Christopher Choy Denver Dash Angela Dai Matthias Nießner
cs.CV





headings





 




SceneCAD






Avetisyan et al.



Technical University of Munich Stanford University Occipital Inc.




 
SceneCAD: Predicting Object Alignments 
 and Layouts in RGB-D Scans 

    Armen Avetisyan^1 Tatiana Khanova^3 Christopher Choy^2 

Denver Dash^3 Angela Dai^1 Matthias Nießner^1 
    December 30, 2023
===========================================================================================================





       





We present a novel approach to reconstructing lightweight, CAD-based representations of scanned 3D environments from commodity RGB-D sensors.
Our key idea is to jointly optimize for both CAD model alignments as well as layout estimations of the scanned scene, explicitly modeling inter-relationships between objects-to-objects and objects-to-layout.
Since object arrangement and scene layout are intrinsically coupled, we show that treating the problem jointly significantly helps to produce globally-consistent representations of a scene.
Object CAD models are aligned to the scene by establishing dense correspondences between geometry, and we introduce a hierarchical layout prediction approach to estimate layout planes from corners and edges of the scene. 
To this end, we propose a message-passing graph neural network to model the inter-relationships between objects and layout, guiding generation of a globally object alignment in a scene.
By considering the global scene layout, we achieve significantly improved CAD alignments compared to state-of-the-art methods, improving from 41.83% to 58.41% alignment accuracy on SUNCG and from 50.05% to 61.24% on ScanNet, respectively.
The resulting CAD-based representations makes our method well-suited for applications in content creation such as augmented- or virtual reality. 




§ INTRODUCTION



The recent progress of 3D reconstruction of real-world environments from commodity range sensors has spurred interest in using such captured 3D data for applications across many fields, such as content creation, mixed reality, or robotics.
State-of-the-art 3D reconstruction approaches can now produce impressively-robust camera tracking and surface reconstruction <cit.>.

Unfortunately, the resulting 3D reconstructions are not well-suited for direct use with many applications, as the geometric reconstructions remain incomplete (e.g., due to occlusions and sensor limitations), are often noisy or oversmoothed, and often consume a large memory footprint due to high density of triangles or points used to represent a surface at high resolution.
There still remains a notable gap between these reconstructions and artist-modeled 3D content, which are clean, complete, and lightweight <cit.>.

Inspired by these attributes of artist-created 3D content, we aim to construct a CAD-based scene representation of an input RGB-D scan, with objects represented by individual CAD models and scene layout represented by lightweight meshes.
In contrast to previous approaches which have individually tackled the tasks of CAD model alignment <cit.> and of layout estimation <cit.>, we observe that object arrangement is typically tightly correlated with the scene layout.
We thus propose to jointly optimize for CAD model alignment and scene layout to produce a globally-consistent CAD-based representation of the scene.

From an input RGB-D scan along with a CAD model pool, we align CAD models to the scanned scene by establishing dense correspondences. 
To estimate the scene layout, we characterize the layout into planar elements, and propose a hierarchical layout prediction by first detecting corner locations, then predicting scene edges, and from sets of edges potentially presenting a layout plane, predicting the final layout.
We then propose a graph neural network architecture for optimizing the relationships between objects and layout by predicting object-object relative poses as well as object-layout support relationships.
This optimization guides both object and layout arrangement to be consistent with each other.
Our approach is fully-convolutional and trained end-to-end, generating a CAD-based scene representation of a scan in a single forward pass.




In summary, we present the following contributions:

    
  * We formulate a lightweight heuristic-free 3D layout prediction algorithm that hierarchically predicts corners, edges and then planes in an end-to-end fashion consisting of only ≈ 1M trainable parameters generating satisfactory layouts without the need for extensive heuristics.
    
  * We present a scene graph network that learns relationships between objects and scene layout, enabling globally consistent CAD model alignments and results in a significant increase in prediction performance in both synthetic as well as real-world datasets. 
    
  * We introduce a new richly-annotated real-world scene layout dataset consisting of  CAD shells and wireframes on top of the ScanNet RGB-D dataset, allowing large-scale data-driven training for layout estimation. 





§ RELATED WORK





  
CAD model alignment


Aligning an expert-generated 3D model or a 3D template to 3D scan data has been studied widely due to its wide range of applications, for instance motion capture <cit.>, 3D object detection and localization <cit.>, and scene registration <cit.>.
Our aim is to leverage large-scale datasets of CAD models to reconstruct a lightweight, semantically-informed, high-quality CAD representation of an RGB-D scan of a scene.
Several approaches have been developed to retrieve and align CAD models from a shape database and align them in real time to a scan during the 3D scanning process <cit.>, although their use of handcrafted features for geometric scan-to-CAD matching limit robustness.

Zeng et al. <cit.> developed a learned feature extractor using a siamese network design for geometric feature matching, which can be employed for scan-to-CAD feature matching, though this remains difficult due to the domain gap between synthetic CAD models and real-world scans.
Avetisyan et al. <cit.> proposed a scan-to-CAD retrieval and alignment approach leveraging learned features to detect objects in a 3D scan and establish correspondences across the domain gap of scan and CAD. 
They later built upon this work to develop a fully end-to-end trainable approach for this CAD alignment task <cit.>.
For such approaches, each object is considered independently, whereas our approach exploits contextual information from object-object and object-layout to produce globally consistent CAD model alignment and layout estimation.

Other approaches retrieve and align CAD models to RGB images <cit.>; our work instead focuses on geometric alignment of CAD models and layout.

 



  
Graph neural networks and relational inference in 3D.

Recent developments in graph inference and graph neural networks have shown significant promise for inference on 3D data. 
Recently, various approaches have viewed 3D meshes as graphs in order find correspondences between 3D shapes <cit.>, deform a template mesh to fit an image observation of a shape <cit.>, or generate a mesh model of an object <cit.>, among other applications.
Learning on graphs has also shown promise for estimating higher-level relational information in scenes, as a scene graph.
3D-RelNet <cit.> predicts 3D shapes and poses from single RGB images and establish pairwise pose constraints between objects to improve overall prediction quality. 
Our approach is similarly inspired to establish relationships between objects; we additionally employ relationships between objects and structural components (i.e., walls, floors, and ceilings), which considerably inform object arrangement.

Armeni et al. <cit.> propose a unified hierarchical structure that hosts building, room, and object relationships into one 3D scene graph. 
They leverage this graph structure to generate scene graphs from 2D images.

Our approach focuses on leveraging relational information to reconstruct imperfect scans with a CAD-based representation for each object and layout element.



  
Layout estimation.
Various layout estimation approaches have been developed to infer structural information from RGB and RGB-D data.
Scan2BIM <cit.> generates building information models (BIM) from 3D scans by detecting planes and finding plausible intersections to produce room-level segmentation of floors, ceilings and walls under Manhattan-style constraints.

PlaneRCNN <cit.> and PlaneNet <cit.> propose deep neural network architectures to detect planes from RGB images and estimate their 3D parameters.

FloorNet <cit.> estimates a 2D Manhattan-style floorplan representation for an input RGB-D scan using a point-based neural network architecture.
Floor-SP <cit.> relaxes the Manhattan constraints with an integer programming formulation, and produces more robust floorplan estimation. 
In contrast to these layout estimation approaches, our focus lies in leveraging global scene relations between objects as well as structural elements in order to produce a CAD-based representation of the scene.









§ SCENECAD: JOINT OBJECT ALIGNMENT AND LAYOUT ESTIMATION


From an RGB-D scan and a CAD model dataset, our method jointly predicts CAD model alignment and layout estimation to produce a lightweight, CAD-based representation of the scene (Figure <ref>).
The input scan is represented as a sparse 3D voxel grid of the occupied surface geometry carrying fused RGB data.
The scan is first encoded by a series of sparse 3D convolutional layers <cit.> to produce a feature volume F'. The sparse output F' is then densified into a dense 3D feature grid F ∈ℝ^N_f × N_x × N_y × N_z where N_F is the number of channels in the feature and N_x, N_y, and N_z are the resolution of the feature along x, y, and z axis respectively. Note that the encoder serves as backbone for proceeding modules. Hence, F is the input to the CAD alignment module as well as the layout estimation module.

Based on F, we detect objects along with their bounding box in the object detection module and layout planes in the layout detection module. 
We then establish our relational inference by formulating a message-passing graph neural network on the predicted objects and layout planes, where each node represents an object or layout plane, with losses on edge relationships representing relative poses and support. Finally, we predict a set of retrieved CAD models along with their 9-DoF poses (3 translation, 3 rotation, and 3 scale) for every detected object.

The message-passing graph neural network helps to inform objects of both relations between other objects as well as with the scene layout, e.g., certain types of furniture such as beds and chairs are typically directly supported by a floor, chairs near a table often face the table.
This joint optimization thus helps to enable globally consistent CAD model alignment in the final output. 



 §.§ Layout Prediction


The indoor scene of interest in our problem consists of planar or quadrilateral components such as walls, floors, and ceilings. However, some of these planar elements create complex geometry such as bars, beams, or other structures that effectively make template-matching approach to find the room layout challenging. Thus, we propose a bottom-up approach that predicts corners, edges, and planar elements sequentially to predict the room layout.
Our layout prediction pipeline is structured hierarchically: first predicting the corner locations, then predicting edges between the corners, and finally extracting quads from the predicted edges. We visualize the overview of the pipeline on Figure <ref>.

Corners are predicted by a convolutional network that decodes F to its original dimension by predicting a heatmap; i.e. a voxel-wise score that indicates a cornerness likeliness. The loss for this predicted heatmap is a voxel-wise binary cross-entropy classification loss in conjunction with a softmax and a negative log-likelihood over the entire voxel grid where the problem is formulated as a spatial multi-class problem. 
This is structured as an encoder-decoder, where the bottleneck lies at a spatial reduction of 4×.
Note that we make predictions for corners which have not been observed in the input scan (e.g., due to occlusions, c.f.). See supplemental material for a visual illustration of the layout prediction pipeline.  
From the output corner heatmap, we apply a non-maximum suppression to filter out weak responses, and define the final corner predictions as a set of xyz coordinates 𝒱 = {𝐯_i}_i, 𝐯_i = [x_i, y_i, z_i].

We the predict the layout edges from the predicted corners 𝒱.
We construct the candidate set of edges by taking all pair-wise combinations of corners 𝐞_ij = (𝐯_i, 𝐯_j) for all i ∈ [1, ..., |𝒱|] and j ∈ [1, ..., i - 1]. We denote all edges as ℰ = {𝐞_ij}_ij.
From the pool of candidate edges we predict a set of edges that belongs to the scene structure using a graph neural network. 
Specifically, for each potential edge 𝐞_ij = (𝐯_i, 𝐯_j), we extract corresponding features from the vertex prediction convolutional network, F[𝐯_i], F[𝐯_j] where F[·] denotes the feature vector at the specified x, y, z coordinate. We concatenate these features along with the normalized coordinates to form an input feature vector for each edge 𝐟_𝐞_ij = [F[𝐯_i], F[𝐯_j], 𝐍(𝐯_i), 𝐍(𝐯_j)].

For each edge we construct two feature descriptors with alternating order of corner features 𝐟_𝐞_ji to mitigate the effect of order dependency.
We feed these concatenated features into a graph network, which we train with edge-wise binary cross entropy loss against ground truth edges.
As the vertex predictions have uncertainty, we label edges with predicted vertices within a certain radius from the ground truth layout vertices to be positives.

This edge prediction limits the set of candidate layout quads which would otherwise be O(|𝒱|4).

From these predicted edges, we then compute the set of candidate layout quads as the set of planar, valid 4-cycles within these edges 𝐪_ijkl = {𝐞_ij, 𝐞_jk, 𝐞_kl, 𝐞_li}.
To detect valid cycles, we use the depth-first-search cycle detection algorithm
We predict the final set of layout quads as either positive or negative where the positive predictions constitute the scene layout, decomposed as quads.
The feature descriptor for a candidate quad is constructed by concatenating the features from F corresponding to the corner locations of its vertices and normalized corner locations, 𝐪_ijkl = [F[𝐯_i], F[𝐯_j], F[𝐯_k], F[𝐯_l], 𝐍(𝐯_i), 𝐍(𝐯_j), 𝐍(𝐯_k), 𝐍(𝐯_l)]. Similar to the edge features, every quad feature descriptor is 4-way permuted 𝐪_jkli, 𝐪_klij, and 𝐪_lijk in order to mitigate order-dependency.
This feature is input to an MLP followed by a binary cross entropy loss. From these predicted quads, we recover the scene layouts without heuristic post-processing.





 §.§ CAD Model Alignment



Along with the room layout, we aim to find and align light-weight CAD models to objects in the scanned scene. To this end, we propose a CAD model alignment pipeline that detects objects, retrieves CAD models, and finds transformations that aligns the CAD model to the scanned scene.
First, we use a single-shot anchor-based object detector to identify objects <cit.>, using the features from the backbone we extracted (𝐅) from the previous stage. 
We then filter the predicted anchors with non-maximum suppression following the standard single-shot object detection pipeline <cit.>. 
Given this set of object bounding boxes ℬ, we extract N_d × N_d × N_d feature volume F_o for all o ∈ [1,..., |ℬ|] from the feature map F around the object anchor a_o. 
We use this feature volume for CAD model retrieval and alignment. 
A corresponding CAD model is retrieved by calculating an object descriptor of length 512 and searching the nearest neighbor CAD model from an shared embedding space. 
This shared embedding space is established by minimizing the distance between descriptors of scanned objects and their CAD counterpart with an L1 loss during training.  

Finally, given the nearest CAD model for all object anchors, we find dense correspondences between the CAD model and the feature volume F_o. Dense correspondences are trained through an explicit voxel-wise L1 regression loss. We use Procrutes <cit.> to estimate a rotation matrix and an L1 distance loss with respect to the groundtruth rotation matrix to further enhance correspondence quality. 
Note that the Procrutes method yields a transformation matrix through the Singular Value Decomposition which is differentiable, allowing for end-to-end training. 




 §.§ Learning Object and Layout Relationships


From our layout prediction and CAD model alignment, we obtain a set of layout quads and aligned CAD models, both obtained independently from the same backbone features.
However, this can result in globally inconsistent arrangements; for instance, objects passing through the ground floor, or shelves misaligned with walls.
We thus propose to learn the object-layout as well as object-object relationships as a proxy loss used to guide the CAD model alignments and layout quads into a globally consistent arrangement.

We construct this relationship learning as a graph problem, where the set of objects and layout quads form the nodes of the graph. 
Edges are constructed between every object-object node-pair and every object-quad node-pair, forming a graph on which we formulate a message-passing graph neural network.

Each node of the graph is characterized by a feature vector of length 128.
For objects this feature vector is obtained by pooling the object feature volume to 8^3 resolution, followed by linearization.
For layout quads, this feature vector is constructed by concatenating the features from F or the associated corner locations, upon which an MLP is applied to obtain a 128-dimensional vector.

Figure <ref> shows an overview of our message-passing network.
Messages are passed from nodes to edges for a graph G = (V,E), with nodes v_i∈ V and edges e_j,k=(v_j,v_k)∈ E.
We define the message passing similar to <cit.>:

    v→ e: 𝐡_i,j^t+1 = f_e(concat(𝐡_i^t, 𝐡_j^t - 𝐡_i^t))




where 𝐡_i^t is the feature corresponding to vertex v_i at message passing step t, 𝐡_i,j^t is the feature corresponding edge e_i,j at step t, and f_e represents an MLP.
That is, edges features are computed as the concatenation of its constituent vertices.

We then take these output edge features from the message passing and perform a classification of various relationships using a cross entropy loss.
We describe the relationships as follows, which we chose as they do not require extra manual annotation effort given existing ground truth CAD alignments and scene layout; see Section <ref> for more detail regarding extraction of ground truth object and layout relationships.
For object-layout relationships, we formulate a 3-class classification task for support relations, predicting horizontal support, vertical support, or no support. Only one relationship per object-layout pair is allowed.
For object-object relationships, we predict the angular difference between the front-facing vectors of the respective objects, in order to recognize common relative arrangements of objects (e.g., chairs often face tables). This is trained with a 6-class cross entropy loss where the angular deviation up to 180∘ is discretized into 6 bins.

Here, the relationship prediction adds a proxy loss to the model in Figure <ref> which inter-correlates object and layout alignments, implicitly guiding the CAD model alignment and layout quad estimation to become more globally consistent.






§ OBJECT+LAYOUT DATASET


To train and evaluate our method, we introduce a new dataset of  CAD layout annotations to the real-world RGB-D scans of the ScanNet dataset <cit.>.
These layout annotations, in addition to the CAD annotations of Scan2CAD <cit.> to ScanNet scenes, inform our method and evaluation on real-world scan data.

In order to obtain these room layout annotations, we use a semi-automated  annotation process.
We then automatically extract the object-object and object-layout relations.




 §.§ Extraction of Scene Layouts


We performed a semi-automatic layout annotation for ScanNet scene data.
First, large planar surfaces are detected using RANSAC on the reconstructed scans.
We then employ a manual refinement step to modify potential errors in the automatic extraction. The surface extraction is preceded by a semantic instance segmentation to obtain wall, floor, ceiling, window, door, etc. instances. RANSAC is then applied to extract 3D planes from each instance. Planes that fall below a threshold will be merged or connected. All planes are projected onto the floor plane and through a set of various heuristics the most plausible intersection points are selected to ultimately become corner points for the final layout. The room height is either estimated by the maximum height of the detected wall instances or is spanned by the ceiling.

Following the proposals given by RANSAC, we then manually verified which proposals were plausible.
This step is relatively quick (≈ 2min per scene) and indicated that the RANSAC produced  plausible initial layouts.
These layouts were then refined through a manual annotation process.
We developed a Blender[https://www.blender.org]-based tool was introduced for the layout refinement, allowing annotators to edit/merge/delete corner junctions as well as add or modify edges and planes. 
All automatically generated layouts were verified and refined by two student annotators (≈ 15min per scene).
From the output planes, we then estimate their quad corners and edges as intersections between the planes. 
An illustration of layouts annotation samples on ScanNet can be found in the supplemental. 




 §.§ Extraction of Object and Layout Relationships




To support learning global scene relationships, we extract object and layout relations to supervised our message-passing approach to learning relationships.
We opt to learn relations which can be automatically extracted from given CAD model and layout annotations.

We extract object-object and object-layout relationships. 
For the object-object case, we compute the angular difference between the front-facing vectors of each object where symmetrical properties are ignored; in practice, we compute this on-the-fly during the training process.

Relationships between objects and layout elements are established by support:

  
  * A vertical support relationship between a layout element and an object is valid if the bottom side of the bounding box of the object within close proximity to and close to parallel to the layout element.
  
  * A horizontal touch relationship is valid if the left, right, front or back side of the bounding box of the object is within close proximity to and close to parallel to the layout element.



These relations are  extracted through an exhaustive search. 
That is, each pair of object-layout is checked for vertical support or horizontal touch.
To estimate proximity of objects, we expand the bounding box of the objects by τ_p, and expand the sides of the bounding boxes of the layout elements by τ_p.
We then consider the object and layout element to be in close proximity if their expanded bounding boxes overlap. 
We use τ_p=0.2 meters for all experiments.




 §.§ Synthetic Data

We additionally evaluate our approach on synthetic data, where CAD object and layout ground truth are provided in the construction of the synthetic 3D scenes.
We use synthetic scenes from the SUNCG dataset <cit.>. 
SUNCG contains models of indoor building environments including CAD models and room layouts. Layout components are given and hence extraction into planar quads can be performed automatically. 
To generate the input partial scans, we virtually scan the scenes to produce input scans similar to real-world scenarios, following previous approaches to generate synthetic partial scan data <cit.>.

Object and layout relational information was extracted following the same procedure for ScanNet data.






§ RESULTS








 §.§ CAD Alignment Performance


We evaluate our method on synthetic SUNCG <cit.> scans as well as real-world ScanNet <cit.> scans in Tables <ref> and <ref>, respectively.
We follow the CAD alignment evaluation metric proposed by <cit.>, which measures alignment accuracy where an alignment is considered successful if it falls within 20cm, 20^∘, and 20% scale of the ground truth.
On both SUNCG and ScanNet scans we compare to several state-of-the-art handcrafted geometric feature matching approaches <cit.> and learned approaches <cit.>.
We additionally show qualitative comparisons in Figures <ref> and <ref>.
On synthetic scan data we outperform the strongest baseline by 16.58%, and improve by 10.52% on real scan data. 
This demonstrates the benefit of leveraging global information regarding object and layout relations in improving object alignments.

We also perform an ablation study on the various design choices and impact of relation information.
We evaluate a dense convolutional backbone for our network architecture (dense) in contrast to our final sparse convolutional backbone leveraging the sparse convolutions proposed by <cit.>.
We additionally show that the object-to-object relational inference (obj-obj) as well as layout estimation (layout) improve upon no relational inference, and our full method incorporating both object and layout relational inference, the most contextual information, yields the best performance. 



 §.§ Layout Prediction








We evaluate the performance of our layout estimation based on its intermediate corner and edge prediction results as well as the final layout quad predictions.
In Table <ref>, we evaluate the precision and recall of each the corner, edge, and layout quad predictions.
Corners are considered as successfully detected if the predicted corner is within a radius of 40cm from the ground truth corner. 
Edges are considered as correctly predicted if they connect the same corners as the ground truth edges. 
Similarly, correctly predicted quads are spanned by the same 4 corners as the associated ground truth quad.
We aim to achieve a high recall for corners and edges due to our hierarchical prediction.
We achieve robust results on both datasets, although ScanNet is notably more difficult as many scenes can miss views of entire layout components (e.g., missing ceilings).


















§ LIMITATIONS


While the focus of this work was to show improved scene understanding through joint prediction of objects and layouts, we believe there is potential for further achievements. For instance, our layout prediction method is bound to predict quad planes only and hence more sophisticated methods could be used for more accurate layout estimation. Also, we used a very lightweight graph neural network for message passing. One could use a more sophisticated method for more accurate relationship prediction and a richer set of relationships that may contain functionality relationships, spatial relationships or room semantic relationships (for instance a bathtub and a shower curtain could be grouped into the same semantic class).




§ CONCLUSION


In this work we formulated a method to digitize 3D scans that goes beyond the focus of objects in the scene. We propose a novel method that estimates the layout of the scene by sequentially predicting corners, then edges and finally quads in a fully differentiable way. The estimated layout is used in conjunction with an object detector to predict contact relationships between objects and the layout and ultimately to predict a CAD arrangement of the scene. We can show that objects and the surrounding (scene layout) go hand in hand and are a crucial factor towards full scene digitization and scene understanding. Objects in the scene are often not arbitrarily arranged, for instance often cabinets are leaned at walls or a table is surrounded by chairs in a dining room, hence we leverage the inherent coupling between objects and layout structure in the learning process. Our approach improves global CAD alignment accuracy by learning those patterns on both real and synthetic scans. 
We hope that we can encourage further research towards this avenue, and see as next immediate steps for future work the necessity of texturing digitized shapes in order to enhance the immersive experience in VR environments. 



§ ACKNOWLEDGEMENTS

We would like to thank the expert annotators 
Soh Yee Lee, 
Suzana Spasova, and
Danil Bebnev
for their efforts in building our scene layout dataset. 
This work is supported by Occipital, the ERC Starting Grant Scan2CAD (804724), a Google Faculty Award, the ZD.B., a TUM-IAS Rudolf Mößbauer Fellowship, and the German Research Foundation (DFG) Grant Making Machine Learning on Static and Dynamic 3D Data Practical.






splncs04






§ DATASET


In this supplemental document, we provide additional details for our layout dataset that is used for training. 
Figure <ref> shows an illustration of the real-world layout annotations comprising of more than 1000 individual ScanNet <cit.> scenes. 
In addition to layouts from real-world scans, we also extract layouts from the synthetic SUNCG dataset <cit.>; see Figure <ref>.
From these dataset annotations, we create ground truth targets. as visualized in Figure <ref>. for our hierarchical layout estimation training.

In Figure <ref>, we illustrate visually the different kinds of object-to-layout relationship classes. Note that an object can have relationships with multiple layout elements.














