MarrNet: 3D Shape Reconstruction via 2.5D Sketches
MarrNet: 3D Shape Reconstruction via 2.5D Sketches

          
          
          Figure 1: MarrNet for 3D Reconstruction
        
          
          
          Figure 2: Shapes reconstructed by MarrNet
        
            Abstract
            3D object reconstruction from a single image is a highly under-determined problem, requiring strong prior knowledge of plausible 3D shapes. This introduces challenge for learning-based approaches, as 3D object annotations in real images are scarce. Previous work chose to train on synthetic data with ground truth 3D information, but suffered from the domain adaptation issue when tested on real data.
            In this work, we propose an end-to-end trainable framework, sequentially estimating 2.5D sketches and 3D object shapes. Our disentangled, two-step formulation has three advantages. First, compared to full 3D shape, the 2.5D sketch is much easier to be recovered from the 2D image, and to transfer from synthetic to real images. Second, for 3D reconstruction from the 2.5D sketches, we can easily transfer the learned model on synthetic data to real images, as rendered 2.5D sketches are invariant to object appearance variations in real images, including lighting, texture, etc. This further relieves the domain adaptation problem. Third, we derive differentiable projective functions from 3D shapes to 2.5D sketches, making the framework end-to-end trainable on real images, requiring no real-image annotations. Our framework achieves state-of-the-art performance on 3D shape reconstruction.
          
          
            Spotlight Video
          
          
            
            If you cannot access YouTube, please download our video here.
          
        
            Publication
            
              
              
                
                
              
            
          
            Related Publications
            
              
                
                
                
                
              
            
            
              
                
                
                
              
            
            
              
                
                
                
              
            
            
              
                
                
                
              
            
          