Illumination estimation is often used in mixed reality to re-render a scene from another point of view, to change the color texture of an object, or to insert a virtual object consistently lit into a real video or photograph. Specifically, the estimation of a point light source is required for the shadows cast by the inserted object to be consistent with the real scene. We tackle the problem of illumination retrieval given an RGBD image of the scene as an inverse problem: we aim to find the illumination that minimizes the photometric error between the rendered image and the observation. In particular we propose a novel differentiable renderer based on the Blinn-Phong model with cast shadows. We compare our differentiable renderer to state-of-the-art methods and demonstrate its robustness to an incorrect reflectance estimation.
@cite_26: Augmented Reality (AR) scenarios aim to provide realistic blending between real world and virtual objects. A key factor for realistic AR is thus a correct illumination simulation. This consists in estimating the characteristics of real light sources and use them to model virtual lighting. In this paper, we briefly introduce a novel method for recovering both 3D position and intensity of multiple light sources using detected cast shadows. Our algorithm has been successfully tested on a set of real scenes where virtual objects have visually coherent shadows.
@cite_2: This paper describes a new method for estimating the illumination distribution of a real scene from a radiance distribution inside shadows cast by an object in the scene. First, the illumination distribution of the scene is approximated by discrete sampling of an extended light source. Then the illumination distribution of the scene is estimated from a radiance distribution inside shadows cast by an object of known shape onto another object in the scene. Instead of assuming any particular reflectance properties of the surface inside the shadows, both the illumination distribution of the scene and the reflectance properties of the surface are estimated simultaneously, based on iterative optimization framework. In addition, this paper introduces an adaptive sampling of the illumination distribution of a scene. Rather than using a uniform discretization of the overall illumination distribution, we adaptively increase sampling directions of the illumination distribution based on the estimation at the previous iteration. Using the adaptive sampling framework, we are able to estimate overall illumination more efficiently by using fewer sampling directions. The proposed method is effective for estimating an illumination distribution even under a complex illumination environment.
A way to cope with cast shadows is to model illumination as a set of candidates, either point sources @cite_26 or directional sources @cite_2 . Sato al @cite_2 propose to estimate the illumination distribution by an adaptive sampling of the directional sources. They implicitly make use of shadows by incorporating a shadow term in their Bidirectional Reflectance Distribution Function (BRDF), computed from the geometry and the potential directional light candidates. In @cite_26 , Jiddi al approximate a set of point light sources equally distributed around the scene, and select the candidates whose cast shadows correlate with a binary mask of shadows preliminary detected in the image. Our method differs from @cite_26 and @cite_2 in that we perform a continuous optimization, and we do not approximate a set of candidates. Hence there is no need to compute a set of shadow maps prior to the optimization: a shadow map is rendered at each iteration given the current illumination estimate.