Title: 3D Sketching using Multi-View Deep Volumetric Prediction,
Abstract: Sketch-based modeling strives to bring the ease and immediacy of drawing to
the 3D world. However, while drawings are easy for humans to create, they are
very challenging for computers to interpret due to their sparsity and
ambiguity. We propose a data-driven approach that tackles this challenge by
learning to reconstruct 3D shapes from one or more drawings. At the core of our
approach is a deep convolutional neural network (CNN) that predicts occupancy
of a voxel grid from a line drawing. This CNN provides us with an initial 3D
reconstruction as soon as the user completes a single drawing of the desired
shape. We complement this single-view network with an updater CNN that refines
an existing prediction given a new drawing of the shape created from a novel
viewpoint. A key advantage of our approach is that we can apply the updater
iteratively to fuse information from an arbitrary number of viewpoints, without
requiring explicit stroke correspondences between the drawings. We train both
CNNs by rendering synthetic contour drawings from hand-modeled shape
collections as well as from procedurally-generated abstract shapes. Finally, we
integrate our CNNs in a minimal modeling interface that allows users to
seamlessly draw an object, rotate it to see its 3D reconstruction, and refine
it by re-drawing from another vantage point using the 3D reconstruction as
guidance. The main strengths of our approach are its robustness to freehand
bitmap drawings, its ability to adapt to different object categories, and the
continuum it offers between single-view and multi-view sketch-based modeling.