Flash Lightens Gray Pixels
Yanlin Qian Song Yan Joni-Kristian Kämäräinen Jiri Matas



El mètode de les línies per a
la resolució numèrica d'equacions en derivades parcials

The method of lines for numerical solutions of partial differential equations
Aquesta recerca té el suport del projecte 2017SGR1087 de l'Agència de Gestió d'Ajuts Universitaris i de Recerca (AGAUR) del Govern de Catalunya. 
    C. Dalfó

	Departament  de Matemàtica, Universitat de Lleida, Igualada (Barcelona), Catalunya

	

	<cristina.dalfo@matematica.udl.cat>

		M. A. Fiol

	Deptartament de Matemàtiques, Barcelona Graduate School of Mathematics,

		Universitat Politècnica de Catalunya, Barcelona, Catalunya

		<miguel.angel.fiol@upc.edu>

    
================================================================================================================================================================================================================================================================================================================================




In the real world, a scene is usually cast by multiple illuminants and herein we address the problem of spatial illumination estimation. Our solution is based on detecting gray pixels with the help of flash photography. We show that flash photography significantly improves the performance of gray pixel detection without
illuminant prior, training data or calibration of the flash. We also introduce a novel flash photography
dataset generated from the MIT intrinsic dataset.



spatial illumination estimation, gray pixel, flash photography, color constancy





§ INTRODUCTION





We address the illumination estimation problem which aims to measure the chroma of illumination in order to remove the color-bias from a captured image <cit.>.  Illumination estimation can help in high-level vision tasks, e.g. object recognition, tracking <cit.> and intrinsic image decomposition. There exists a large number of related works, from the
traditional non-learning approaches <cit.> to recent deep learning based
approaches <cit.>. However, the vast majority of these
works concentrate on the case of a single global illumination which is often an invalid assumption <cit.>. In this paper, we explore a more-complex less-optimistic setting – mixed illumination [We traverse related works and there exists multiple terms referring to the same thing (which may confuse readers), i.e. mixed illumination, spatially-varying illumination, multiple illumination, mixed lighting condition.]. 





Spatially-varying illumination refers to that on a captured scene, each pixel captures different number of light phantoms when the camera shutter is on. In other words, all pixels do not share the same configuration of lights  <cit.>, which is the default assumption for single-illumination estimation. Compared to the single global illumination setting, the mixed illumination setting better corresponds to the real world <cit.>, but is more challenging clearly, as it extends the ill-posed problem from one point to a spatial map  <cit.>, without extra knowledge or input. 







To circumvent the hardness that spatially-varying illumination brings, efforts are put as follows: user guidance or human interaction is given as a supervisory signal <cit.>; small patch is assumed to be cast by only one light <cit.>; light color and number need to be known before experiments <cit.>. Unlike these methods, we make use of flash photography.






Flash photography refers to image processing techniques which use
non-flash/flash image pairs.
This technique is well adopted to spatial illumination estimation <cit.>, which assume each patch illuminated by one light and obtain decent results. 
Hui et al. <cit.> proposed a closed-form solution of spatial illumination for the case of a calibrated flash.
In essence, flash calibration in <cit.> equals to knowing the “groundtruth” surface albedo in a flash-only image. What's more, flash may appear in other forms, i.e. varying sunlight, cast shadow, which may be hard to calibrate.

In this paper, we propose a novel spatial illumination estimation method, using flash photography, without need of flash calibration and any other illuminant prior. 
Our method relies on gray pixel detection <cit.>. The original work assumes Lambertian surfaces, and then revisited and improved by <cit.>. 
The original and extended gray pixel methods however
fail in the case of mixed illumination (the top row in Fig. <ref>), but we analytically show how flash photography circumvents the problem and “lightens” gray pixel photometrically and in performance (bottom row in Fig. <ref>). 


 The interplay of flash photography and gray pixel enables us to achieve a largely increased performance on our synthesized laboratory dataset and some real-world images, than running gray pixel methods alone, without knowing the flash color.


Our contributions are three-fold:


  * We revisit and revise the gray pixel methods for the case of mixed illumination.

  * Leveraging flash photography and the gray pixel method, we propose a novel learning-free and well-performing method for spatial illumination estimation.

  * We propose a novel flash-photography dataset for benchmarking multi-illuminant methods. The dataset is based on the MIT intrinsic dataset. 


The rest of this paper is organized as follows. Section 2 revisits gray pixel and its variants before we introduce the flash-photography gray pixel in Section 3. In Section 4 we describe the MIT intrinsic based dataset for our task. Section 5 covers the experiments and results. We conclude in Section 6.




§ GRAY PIXEL



Assuming one light source and narrow sensor response, Gray Pixel <cit.> is derived from the Lambertian model, given as:

    I^c(p) = R^c(p) max(n(p)^⊺ s,0) l^c,

which shows the color channel c at the location p in image I is a function of a surface albedo R, surface normal n, light direction s and illumination color l. 

Following the procedure in <cit.>, applying log and a Mexican hat filter δ on Eq. <ref> yields: 

    δlog I^c(p)   =δlog R^c(p)+δlogmax(n(p)^⊺ s,0)+δlog l^c.

A single light casting a small local neighborhood (the same color and direction), Eq. <ref> simplifies to:

    δlog I^c(p)   =δlog R^c(p),

which is the core of gray pixel. δlog I^c(p)=δlog I^c'(p), ∀ c,c' ∈{R,G,B} defines a "pure gray pixel". To rank pixels w.r.t. “grayness”, <cit.> defines the following grayness function, up to a scale:

    g(p)=∑_c ∈ R,G,B (δlog I^c(p) - δ̅log I(p) )^2 / δ̅log I(p),

where δ̅log I(p) is the mean value of δlog I^c(p). This method works robustly with single-illumination scenes where diffuse reflection (the
Lambertian assumption) is dominant.

Then <cit.> augments the above Gray Pixel by replacing Eq. <ref> with the following luminance-independent function:

    g'(p) =cos^-1(1/√(3)‖δlog I(p)‖_1/‖δlog I(p) ‖_2),

where ‖·‖_n refers to the ℓ n norm. g'(p)=0 refers to pure gray pixel. To remove spurious color pixels, <cit.> applies a mean shift clustering to choose the strongest mode – dominant illumination vector.

The mechanism to detect gray pixel is further improved in <cit.>. Different to <cit.>, which are based on the Lambertian model, Qian et al. <cit.> uses the dichromatic reflection model <cit.> to derive a set of more strict constraints for gray pixels: 

    g”(p) = ‖δ(log(I^R)-log(| I |)),
    δ(log(I^G)-log(| I |))‖_2,

where | I | refers to (I^R+I^G+I^B). We refer readers to the original paper for more details. In Section <ref> we report their performance in a mixed-lighting dataset and show how flash photography improves them. 













§ FLASH GRAY PIXEL



In the sequel, we first investigate what will happen to the existing gray pixel methods in the
case of mixed illumination. Then we propose a novel gray pixel method using flash photography, termed as Flash Gray Pixel.

Here we do analysis on the original Gray Pixel <cit.>, but similar conclusion for <cit.> can be inferred in an analogue
manner. A scene is illuminated by N light sources or arbitrary type and color.
To describe the image formation process in this case,
Eq. <ref> is modified to:

    I^c(p) = R^c(p) ∑_i λ_i(p) l_i^c  ,

where λ_i(p) represents the shading term max(n(p)^⊺ s_i,0). Eq. <ref> changes to: 

    δlog I^c(p)   =δlog R^c(p) + δlog(∑_i λ_i(p) l_i^c)  .

Since that is a mixed illumination image, the light configuration varies from pixel to pixel, making the right most term in Eq. <ref> non-zero and therefore Eq. <ref> fails. This finding explains that mixed illumination hinders the performance of the gray pixel method, which motivates us to leverage flash photography. 

When a flash light is present, the flash image I_f is expressed as: 

    I_f^c(p) = R^c(p) (∑_i λ_i(p) l_i^c + λ_f(p) l_f^c),

where λ_f(p) is the shading term of flash light at the position p and l_f the unit-norm chroma vector. Subtracting Eq. <ref> from Eq. <ref>, we get a flash-only image I_fo:

    I_fo^c(p) = R^c(p) λ_f(p) l_f^c,

which is a more solid ground for searching gray pixels. In other terms,
flash image helps to remove the negative effect of spatially-varying light configurations and gray pixels are now flash gray pixels.















Flash gray pixels can be found from the
grayness map of a residual flash/no-flash image. To select flash gray pixels
robustly, we follow <cit.> to compute flash-only illumination component for each pixel:
K-means is used to cluster the top N% gray pixels into preset M clusters; then the illumination at location p is computed using: 



    L_fo^c(p)=∑_m=1^Mω_m L^c_m,

where L_m refers to the average illumination for the cluster m and ω_m controls the connection between the pixel I(x,y) to the cluster m, unfolded as:



    ω_m=e^-D_m/2σ^2/∑_n=1^M e^-D_n/2σ^2 ,

where D_m is the Euclidean distance from the pixel to the centroid of the cluster m. Eq. <ref> encourages nearby pixels to share a similar illumination. 

Combining the flash-only illumination L_fo^c(p) with Eq. <ref> allows to color-corrected the image I_fo
by 

    I_gray^c(p) = I_fo^c(p) / L_fo^c(p),

and the mixed illumination is:

    L_^c(p) = I_^c(p) / I_gray^c(p)  .


Flash gray pixel can be filled by more advanced gray pixel methods for further improvement. 




§ DATASET





We adapted the MIT Intrinsic benchmark <cit.> for our task [There are recent datasets that provide flash/no-flash image pairs <cit.>, but these are unsuitable for our purposes due to unknown illumination number.]. This dataset was originally collected for the intrinsic image decomposition task, containing 20 single objects illuminated by uncalibrated whitish light sources from 10 different directions. This property allows us to render each image in the combinations of 1-9 directed lights with arbitrary chroma to compose a new no-flash image I. The 6-th direction is
always roughly frontal and was thus used as the flash source and together with I forms the flash image I_f. We generated I and I_f with varying number N of light sources, from 2 to 8. Note that even for the easiest case (N=2), a pixel may be simultaneously affected by two light sources, violating the global illumination assumption and breaking the original gray pixel as demonstrated in Fig. <ref>.

Considering the fact that gray pixel methods are designed for realistic consumer images "in the wild" <cit.> that contain at least a few gray pixels, we
left out the 5 chromatic objects[“apple, pear, frog2, potato, turtle”]. In total, the new dataset contains 105  flash/no-flash image pairs with spatial illumination map ground truth ((15 objects and 7 choices of N).





§ EXPERIMENTS
















Our setup is the following: we run the three variants of Gray Pixel methods with flash/no-flash image pairs. On each image the top 10% gray pixels are selected to cover enough area. The cluster number is set to M=N, which is the number of illuminants. 


Evaluation metric is the standard
average angular error <cit.>. 


Results are summarized in
Table <ref> which shows the performance of flash gray pixel on the dataset in Section <ref>. “GP+f” refers to flash gray pixel on the basis of the original gray pixel method “GP”. Results show that flash photography extensions
of all gray pixel variants <cit.>
systematically improve the results. For all three methods, the improvement over the whole dataset is over 40% in median and 30% in mean. The flash photography
variants achieve the sufficient color constancy accuracy (≤ 3.0^∘) in
almost all cases. Fig. <ref> illustrates predicted illumination between gray pixel methods and their flash variants. It is clear that the
original versions cannot find the fine-grained details of mixed illumination.
For example, the frog back and stomach are cast with different colors that
confuses the original GP methods.

Among all gray pixel methods, the original GP <cit.> suffers from the mixed illumination the most. MSGP <cit.> and DGP <cit.> perform slightly better, but not to a satisfying degree without
the flash. With flash gray pixel, all three methods perform similarly thus
verifying the efficiency of flash photography. DGP <cit.> does
not perform better than the original GP which is due to the fact that our
dataset does not contain specular reflectance components.

As the number of diverse illumiants increases, all GP methods performs better.
This can be explained by the fact
that a large number of lights with various colors additively mix toward a
whitish color. Flash gray pixel variants are not affected by this, obtaining consistent results. 

Images from real-world scenes (Fig. <ref>, images retrieved from <cit.>), show that the proposed methods also produce high quality results (e.g. the white wall is white). 







§ CONCLUSION



In this paper we reconsider gray pixel detection through the medium of flash photography. We find that computing a residual map from the flash/no-flash pair, gray pixel methods can effectively measure the grayness of each pixel and provide spatial color constancy.



 We find that computing a residual map from the flash/no-flash pair, gray pixel methods can effectively measure the grayness of each pixel, allowing a large margin improvement in spatially-varying illumination estimation.  The method is pragmatic – it is lightweight,  does not
need any illuminant prior, training, flash light calibration or
user input.
































IEEEbib


