Recent advances in photographic sensing technologies have made it possible to achieve light detection in terms of a single photon. Photon counting sensors are being increasingly used in many diverse applications. We address the problem of jointly recovering spatial and temporal scene radiance from very few photon counts. Our ConvNet-based scheme effectively combines spatial and temporal information present in measurements to reduce noise. We demonstrate that using our method one can acquire videos at a high frame rate and still achieve good quality signal-to-noise ratio. Experiments show that the proposed scheme performs quite well in different challenging scenarios while the existing approaches are unable to handle them.
@cite_30: Active optical imaging systems use their own light sources to recover scene information but typically operate with large number of photon detections. Here, the authors present a 3D imaging system that acquires depth and reflectivity information with a single photon camera operating in low-light conditions.
@cite_35: 
@cite_22: We use a femtosecond laser and a single-photon avalanche diode (SPAD) camera to develop a system that is able to locate objects hidden from view and track their motion.
@cite_3: Recently we have proposed a new image device called the gigavision camera. The main feature of this camera is that the pixels have a binary response. The response function of a gigavision sensor is non-linear and similar to a logarithmic function, which makes the camera suitable for high dynamic range imaging. Since the sensor can detect a single photon, the camera is very sensitive and can be used for night vision and astronomical imaging. One important aspect of the gigavision camera is how to estimate the light intensity through binary observations. We model the light intensity field as 2D piecewise constant and use Maximum Penalized Likelihood Estimation (MPLE) to recover it. Dynamic programming is used to solve the optimization problem. Due to the complex computation of dynamic programming, greedy algorithm and pruning quadtrees are proposed. They show acceptable reconstruction performance with low computational complexity. Experimental results with synthesized images and real images taken by a single-photon avalanche diode (SPAD) camera are given.
@cite_15: Imaging through fog has important applications in industries such as self-driving cars, augmented driving, airplanes, helicopters, drones and trains. Here we show that time profiles of light reflected from fog have a distribution (Gamma) that is different from light reflected from objects occluded by fog (Gaussian). This helps to distinguish between background photons reflected from the fog and signal photons reflected from the occluded object. Based on this observation, we recover reflectance and depth of a scene obstructed by dense, dynamic, and heterogeneous fog. For practical use cases, the imaging system is designed in optical reflection mode with minimal footprint and is based on LIDAR hardware. Specifically, we use a single photon avalanche diode (SPAD) camera that time-tags individual detected photons. A probabilistic computational framework is developed to estimate the fog properties from the measurement itself, without prior knowledge. Other solutions are based on radar that suffers from poor resolution (due to the long wavelength), or on time gating that suffers from low signal-to-noise ratio. The suggested technique is experimentally evaluated in a wide range of fog densities created in a fog chamber It demonstrates recovering objects 57cm away from the camera when the visibility is 37cm. In that case it recovers depth with a resolution of 5cm and scene reflectance with an improvement of 4dB in PSNR and 3.4x reconstruction quality in SSIM over time gating techniques.
@cite_38: 
@cite_21: 
@cite_39: Single-photon avalanche diode (SPAD) based transient imaging suffers from an aberration called pile-up. When multiple photons arrive within a single repetition period of the illuminating laser, the SPAD records only the arrival of the first photon; this leads to a bias in the recorded light transient wherein the transient response at later time-instants are under-estimated. An unfortunate consequence of this is the need to operate the illumination at low-power levels to reduce the probability of multiple photons returning in a single period. Operating the laser at low power results in either low signal-to-noise ratio (SNR) in the measured transients or reduced frame rate due to longer exposure durations to achieve a high SNR. In this paper, we propose a signal processing-based approach to compensate pile-up in post-processing, thereby enabling high power operation of the illuminating laser. While increasing illumination does cause a fundamental information loss in the data captured by SPAD, we quantify this information loss using Cramer-Rao bound and show that the errors in our framework are only limited to this information loss. We experimentally validate our hypotheses using real data from a lab prototype.
@cite_23: The pursuit of smaller pixel sizes at ever increasing resolution in digital image sensors is mainly driven by the stringent price and form-factor requirements of sensors and optics in the cellular phone market. Recently, Eric Fossum proposed a novel concept of an image sensor with dense sub-diffraction limit one-bit pixels (jots), which can be considered a digital emulation of silver halide photographic film. This idea has been recently embodied as the EPFL Gigavision camera. A major bottleneck in the design of such sensors is the image reconstruction process, producing a continuous high dynamic range image from oversampled binary measurements. The extreme quantization of the Poisson statistics is incompatible with the assumptions of most standard image processing and enhancement frameworks. The recently proposed maximum-likelihood (ML) approach addresses this difficulty, but suffers from image artefacts and has impractically high computational complexity. In this work, we study a variant of a sensor with binary threshold pixels and propose a reconstruction algorithm combining an ML data fitting term with a sparse synthesis prior. We also show an efficient hardware-friendly real-time approximation of this inverse operator. Promising results are shown on synthetic data as well as on HDR data emulated using multiple exposures of a regular CMOS sensor.
@cite_46: A quanta image sensor (QIS) is a class of single-photon imaging devices that measure light intensity using oversampled binary observations. Because of the stochastic nature of the photon arrivals, data acquired by QIS is a massive stream of random binary bits. The goal of image reconstruction is to recover the underlying image from these bits. In this paper, we present a non-iterative image reconstruction algorithm for QIS. Unlike existing reconstruction methods that formulate the problem from an optimization perspective, the new algorithm directly recovers the images through a pair of nonlinear transformations and an off-the-shelf image denoising algorithm. By skipping the usual optimization procedure, we achieve orders of magnitude improvement in speed and even better image reconstruction quality. We validate the new algorithm on synthetic datasets, as well as real videos collected by one-bit single-photon avalanche diode (SPAD) cameras.
@cite_37: 
@cite_7: We propose a data-driven method for photon-efficient 3D imaging using sensor fusion and computational reconstruction to rapidly and robustly estimate a dense depth map from low photon counts.
@cite_28: Imaging objects that are obscured by scattering and occlusion is an important challenge for many applications. For example, navigation and mapping capabilities of autonomous vehicles could be improved, vision in harsh weather conditions or under water could be facilitated, or search and rescue scenarios could become more effective. Unfortunately, conventional cameras cannot see around corners. Emerging, time-resolved computational imaging systems, however, have demonstrated first steps towards non-line-of-sight (NLOS) imaging. In this paper, we develop an algorithmic framework for NLOS imaging that is robust to partial occlusions within the hidden scenes. This is a common light transport effect, but not adequately handled by existing NLOS reconstruction algorithms, resulting in fundamental limitations in what types of scenes can be recovered. We demonstrate state-of-the-art NLOS reconstructions in simulation and with a prototype single photon avalanche diode (SPAD) based acquisition system.
@cite_40: By using time-of-flight information encoded in multiply scattered light, it is possible to reconstruct images of objects hidden from the camera’s direct line of sight. Here, we present a non-line-of-sight imaging system that uses a single-pixel, single-photon avalanche diode (SPAD) to collect time-of-flight information. Compared to earlier systems, this modification provides significant improvements in terms of power requirements, form factor, cost, and reconstruction time, while maintaining a comparable time resolution. The potential for further size and cost reduction of this technology make this system a good base for developing a practical system that can be used in real world applications.
@cite_27: A confocal scanning technique solves the reconstruction problem of non-line-of-sight imaging to give fast and high-quality reconstructions of hidden objects.
@cite_33: Computer vision algorithms build on 2D images or 3D videos that capture dynamic events at the millisecond time scale. However, capturing and analyzing transient images at the picosecond scale&#x2014;i.e., at one trillion frames per second&#x2014;reveals unprecedented information about a scene and light transport within. This is not only crucial for time-of-flight range imaging, but it also helps further our understanding of light transport phenomena at a more fundamental level and potentially allows to revisit many assumptions made in different computer vision algorithms. In this work, we design and evaluate an imaging system that builds on single photon avalanche diode (SPAD) sensors to capture multi-path responses with picosecond-scale active illumination. We develop inverse methods that use modern approaches to deconvolve and denoise measurements in the presence of Poisson noise, and compute transient images at a higher quality than previously reported. The small form factor, fast acquisition rates, and relatively low cost of our system potentially makes transient imaging more practical for a range of applications.
@cite_1: Quanta Image Sensor (QIS) is a single-photon image sensor that oversamples the light field to generate binary measurements. Its single-photon sensitivity makes it an ideal candidate for the next generation image sensor after CMOS. However, image reconstruction of the sensor remains a challenging issue. Existing image reconstruction algorithms are largely based on optimization. In this paper, we present the first deep neural network approach for QIS image reconstruction. Our deep neural network takes the binary bit stream of QIS as input, learns the nonlinear transformation and denoising simultaneously. Experimental results show that the proposed network produces significantly better reconstruction results compared to existing methods.
@cite_31: Time-of-flight depth imaging and transient imaging are two imaging modalities that have recently received a lot of interest. Despite much research, existing hardware systems are limited either in terms of temporal resolution or are prohibitively expensive. Arrays of Single Photon Avalanche Diodes (SPADs) promise to fill this gap by providing higher temporal resolution at an affordable cost. Unfortunately SPAD arrays are to date only available in relatively small resolutions. In this work we aim to overcome the spatial resolution limit of SPAD arrays by employing a compressive sensing camera design. Using a DMD and custom optics, we achieve an image resolution of up to 800A—400 on SPAD Arrays of resolution 64A—32. Using our new data fitting model for the time histograms, we suppress the noise while abstracting the phase and amplitude information, so as to realize a temporal resolution of a few tens of picoseconds.
@cite_13: The ability to record images with extreme temporal resolution enables a diverse range of applications, such as fluorescence lifetime imaging, time-of-flight depth imaging and characterization of ultrafast processes. Recently, ultrafast imaging schemes have emerged, which require either long acquisition times or raster scanning and have a requirement for sufficient signal that can only be achieved when light is reflected off an object or diffused by a strongly scattering medium. Here we present a demonstration of the potential of single-photon detector arrays for visualization and rapid characterization of events evolving on picosecond time scales. The single-photon sensitivity, temporal resolution and full-field imaging capability enables the observation of light-in-flight in air, as well as the measurement of laser-induced plasma formation and dynamics in its natural environment. The extreme sensitivity and short acquisition times pave the way for real-time imaging of ultrafast processes or visualization and tracking of objects hidden from view.
@cite_11: 
Recent works in range imaging through SPAD sensors include @cite_35 @cite_30 @cite_7 . SPAD cameras have been used to perform challenging tasks such as transient imaging @cite_33 @cite_13 @cite_38 @cite_31 @cite_39 , non-line-of sight imaging @cite_40 @cite_22 @cite_28 @cite_27 and imaging through fog @cite_15 . Closely related to the topic of single photon counting is that of Binary Quanta image sensors @cite_11 @cite_37 . These sensors were developed with the objective of shrinking the pixel pitch. In Quanta imaging, the densely defined sensors oversample the scene radiance to generate binary measurements @cite_11 @cite_3 . Image reconstruction schemes have been proposed for these sensors @cite_46 @cite_1 @cite_23 @cite_21 . The main difference between these reconstruction methods and our scheme is that these algorithms consider the availability of a higher number of samples to predict the image intensity at a pixel location. These methods have mainly focused on recovering a static image. In contrast, our focus is on recovering videos.