We propose a novel approach to recovering translucent objects from a single time-of-flight (ToF) depth camera using deep residual networks. When recording translucent objects using the ToF depth camera, their depth values are severely contaminated due to complex light interactions with surrounding environment. While existing methods suggested new capture systems or developed the depth distortion models, their solutions were less practical because of strict assumptions or heavy computational complexity. In this paper, we adopt deep residual networks for modeling the ToF depth distortion caused by translucency. To fully utilize both the local and semantic information of objects, multi-scale patches are used to predict the depth value. Based on the quantitative and qualitative evaluation on our benchmark database, we show the effectiveness and robustness of the proposed algorithm.
@cite_24: 
@cite_20: Time-of-flight (ToF) imaging has become a widespread technique for depth estimation, allowing affordable off-the-shelf cameras to provide depth maps in real time. However, multipath interference (MPI) resulting from indirect illumination significantly degrades the captured depth. Most previous works have tried to solve this problem by means of complex hardware modifications or costly computations. In this work, we avoid these approaches and propose a new technique to correct errors in depth caused by MPI, which requires no camera modifications and takes just 10 milliseconds per frame. Our observations about the nature of MPI suggest that most of its information is available in image space; this allows us to formulate the depth imaging process as a spatially-varying convolution and use a convolutional neural network to correct MPI errors. Since the input and output data present similar structure, we base our network on an autoencoder, which we train in two stages. First, we use the encoder (convolution filters) to learn a suitable basis to represent MPI-corrupted depth images; then, we train the decoder (deconvolution filters) to correct depth from synthetic scenes, generated by using a physically-based, time-resolved renderer. This approach allows us to tackle a key problem in ToF, the lack of ground-truth data, by using a large-scale captured training set with MPI-corrupted depth to train the encoder, and a smaller synthetic training set with ground truth depth to train the decoder stage of the network. We demonstrate and validate our method on both synthetic and real complex scenarios, using an off-the-shelf ToF camera, and with only the captured, incorrect depth as input.
@cite_4: Continuous-wave Time-of-flight (TOF) range imaging has become a commercially viable technology with many applications in computer vision and graphics. However, the depth images obtained from TOF cameras contain scene dependent errors due to multipath interference (MPI). Specifically, MPI occurs when multiple optical reflections return to a single spatial location on the imaging sensor. Many prior approaches to rectifying MPI rely on sparsity in optical reflections, which is an extreme simplification. In this paper, we correct MPI by combining the standard measurements from a TOF camera with information from direct and global light transport. We report results on both simulated experiments and physical experiments (using the Kinect sensor). Our results, evaluated against ground truth, demonstrate a quantitative improvement in depth accuracy.
@cite_12: Multipath interference is an inherent unsolved problem in amplitude modulated continuous wave time-of-flight imaging that results in erroneous depth and amplitude measurements. This kind of interference is the result of light which travels along different optical paths and reaches the same pixel. Important scenarios with strong interference include scenes with highly reflective surfaces, imaging through translucent material, and subsurface scattering and imaging a corner. In this paper, we demonstrate that multiple frequency measurements made with a noncustomized Kinect for the XBox One (the new Kinect) camera can be used to resolve the interfering paths. By formulating our problem as a spectral estimation problem, we provide a closed-form, noniterative technique.
In any case of depth acquisition, the recorded signal might be the result of combining multiple reflected signals from the source, each travels from different paths. This is namely the multi-path inference (MPI), which causes significant depth errors in a concave object or corner of the scene (indirect bounces), in mildly translucent material such as skin or wax (subsurface scattering), or along dense participating media. Alleviating those errors is an active research topic for various imaging systems @cite_12 @cite_4 @cite_24 . In case of the ToF depth sensors, MPI leads to the depth value farther than ground truth because the longer traveling path results in the larger depth. @cite_20 corrected MPI using a single ToF depth image and without any additional information, by learning from the simulated data, such as simulated indirect bounces and global illumination. @cite_4 used high-frequency illumination patterns to resolve ambiguities in multiple possible travel paths, reducing errors caused by sub-surface scattering.