Separation of Transparent Layers using Focus.
Consider situations where the depth at each point in the scene is multi-valued, due to the presence of a virtual image semi-reflected by a transparent surface. The semi-reflected image is linearly superimposed on the image of an object that is behind the transparent surface. A novel approach is proposed for the separation of the superimposed layers. Focusing on either of the layers yields initial separation, but crosstalk remains. The separation is enhanced by mutual blurring of the perturbing components in the images. However, this blurring requires the estimation of the defocus blur kernels. We thus propose a method for self calibration of the blur kernels, given the raw images. The kernels are sought to minimize the mutual information of the recovered layers. Autofocusing and depth estimation in the presence of semi-reflections are also considered. Experimental results are presented.
Introduction
The situation in which several (typically two) linearly superimposed contributions exist is often
encountered in real-world scenes. For example [12, 20], looking out of a car (or room) window,
we see both the outside world (termed real object [35, 36, 41, 42, 43, 45]), and a semi-reflection
of the objects inside, termed virtual objects. The treatment of such cases is important, since
the combination of several unrelated images is likely to degrade the ability to analyze and
understand them. The detection of the phenomenon is of importance itself, since it indicates
the presence of a clear, transparent surface in front of the camera, at a distance closer than the
imaged objects [35, 42, 45].
The term transparent layers has been used to describe situations in which a scene is semi-
reflected from a transparent surface [6, 12, 58]. It means that the image is decomposed into
depth ordered layers, each with an associated map describing its intensity (and, if applicable, its
motion [58]). We adopt this terminology, but stress the fact that this work does not deal with
imaging through an object with variable opacity. Approaches to recovering each of the layers
by nulling the others relied mainly on triangulation methods like motion [6, 12, 13, 22, 36, 49],
and stereo [7, 48]. Algorithms were developed to cope with multiple superimposed motion
fields [6, 49] and ambiguities in the solutions were discovered [47, 60]. Another approach to
the problem has been based on polarization cues [18, 20, 35, 41, 42, 43, 45]). However, that
approach needs a polarizing filter to be operated with the camera, may be unstable when the
angle of incidence is very low, and is di#cult to generalize to cases in which more than two
layers exist.
In recent years, range imaging relying on the limited depth of field (DOF) of lenses has
been gaining popularity. An approach for depth estimation using a monocular system based on
focus sensing [14, 16, 25, 31, 32, 33, 34, 52, 53, 61] is termed Depth from Focus (DFF) in the
computer-vision literature. In that approach, the scene is imaged with di#erent focus settings
(e.g., by axially moving the sensor, the object or the lens), thus obtaining image slices of the
scene. In each slice, a limited range of depth is in focus. Depth is extracted by a search for
the slice that maximizes some focus criterion [21, 25, 31, 32, 34, 52, 55, 62] (usually related to
the two dimensional intensity variations in the region), and corresponds to the plane of best
focus. DFF and image-based rendering based on focused slices has usually been performed on
opaque (and occluding) layers. In particular, just recently a method has been presented for
generating arbitrarily focused images and other special e#ects performed separately on each
occluding layer [5].
Physical modeling of DOF as applied to processing images of transparent objects has long
been considered in the field of microscopy [2, 3, 8, 10, 15, 17, 19, 23, 29, 30, 37, 51], where the
defocus e#ect is most pronounced. An algorithm for DFF was demonstrated [23] on a layered
microscopic object, but due to the very small depth of field used, the interfering layer was very
blurred so no reconstruction process was necessary. Note that microscopic specimens usually
contain detail in a continuum of depth, and there is correlation between adjacent layers, so
their crosstalk is not as disturbing as in semi-reflections. Fundamental consequences of the
imaging operation (e.g. the loss of biconic regions in the three dimensional frequency domain)
that pose limits on the reconstruction ability, and the relation to tomography, were discovered
[9, 29, 51, 50, 54]. Some of the three dimensional reconstruction methods used in microscopy
[2, 3, 10] may be applicable to the case of discrete layers as well.
We study the possibility of exploiting the limited depth of field to detect, separate and
recover the intensity distribution of transparent, multi-valued layers. Focusing yields an initial
separation, but crosstalk remains. The layers are separated based on the focused images, or by
changing the lens aperture. The crosstalk is attenuated by mutual blurring of the disturbing
components in the images (Section 2). Proper blurring requires the point spread functions (PSF)
in the images to be well estimated. A wrong PSF will leave each recovered layer contaminated
by its complementary. We therefore study the e#ect of error in the PSFs. Then, we propose
a method for estimating the PSFs from the raw images (Section 3). It is based on seeking
the minimum of the mutual information between the recovered layers. Recovery experiments
are described in Section 4. We also discuss the implication of semi-reflections on the focusing
process and the depth extracted from it (Section 5). Preliminary and partial results were
presented in [40, 44].
2 Recovery from focused slices
2.1 Using two focused slices
Consider a two-layered scene. Suppose that either manually or by some automatic procedure
(see Section 5), we acquire two images, such that in each image one of the layers is in focus.
Assume for the moment that we also have an estimate of the blur kernel operating on each layer,
when the camera is focused on the other one. This assumption may be satisfied if the imaging
system is of our design, or by calibration. Due to the change of focus settings, the images may
undergo a scale change. If a telecentric imaging system (Fig. 1) is used, this problem is avoided
[33, 59]. Otherwise, we assume that the scale change 1 is corrected during preprocessing [27].
Let layer f 1 be superimposed 2 on layer f 2 . We consider only the slices g a and g b , in which
either layer f 1 or layer f 2 , respectively, is in focus. The other layer is blurred. Modeling the
blur as convolution with blur kernels,
(The assumption of a space-invariant response to constant depth objects is very common in
analysis of defocused images, and is approximately true for paraxial systems or in systems
corrected for aberrations). If a telecentric system is used, h
1 The depth dependence of the scale change can typically be neglected.
2 The superposition is linear, since the real/virtual layers are the images of the objects multiplied by the
transmission/reflection coe#cients of the semi-reflecting surface, and these coe#cients do not depend on the light
intensities. The physical processes in transparent/semi-reflected scenes are described in Refs. [41, 42, 43, 45].
Nonlinear transmission and reflection e#ects (as appear in photorefractive crystals) are negligible at intensities
and materials typical to imaging applications.
d
F

Figure

1: A telecentric imaging system [59]. An aperture D is situated at distance F (the focal
length) in front of the lens. An object point at distance u is at best focus if the sensor is at v. If the
sensor is at - v, the image of the point is a blurred spot parameterized by its e#ective diameter d.
In the frequency domain Eqs. (1) take the form
Assuming that the kernels are symmetric, ImH so the real components
of G a and G b are respectively
ReG
with similar expressions for the imaginary components of the images. These equations can be
visualized as two pairs of straight lines (see Fig. 2). The solution, which corresponds to the
line intersection, uniquely exists for H 2a H 1b #= 1. Since the imaging system cannot amplify any
component (H 1b , H 2a # 1), a unique intersection exists unless H
To gain insight, consider a telecentric system (the generalization is straightforward). In
this case, H H, and the slopes of the lines in Fig. 2 (representing the constraints)
are reciprocal to each other. As H # 1 the slopes of the two lines become similar, hence the
solution is more sensitive to noise in G a and G b . As the frequency decreases, H # 1, hence at
frequency
Transversal domain+H
F
G
G a
F FG a
G a F

Figure

2: Visualization of the constraints on reconstruction from focused slices and the convergence of
a suggested algorithm. For each frequency, the relations (3) between the real components of G a , G b , F 1
and F 2 take the form of two straight lines. The visualization of the imaginary parts is similar.
low frequencies the recovery is ill conditioned. Due to energy conservation, the average gray level
(DC) is not a#ected by defocusing. Thus, at DC, H = 1. In the noiseless case the constraints
on the DC component coincide into a single line, implying an infinite number of solutions.
In the presence of noise the lines become parallel and there is no solution. The recovery of
the DC component is thus ill posed. This phenomenon is also seen in the three dimensional
frequency domain. The image space is band limited by a missing cone of frequencies [9, 29],
whose axis is in the axial frequency direction # v and its apex is at the origin. Recovery of the
average intensity in each individual layer is impossible since the information about inter-layer
variations of the average transversal intensity is in the missing cone [46]. A similar conclusion
may be derived from observing the three dimensional frequency domain support that relies on
di#raction limited optics [54].
In order to obtain another point of view on these di#culties, consider the naive inverse
filtering approach to the problem given by Eq. (2). In the transversal spatial frequency domain,
the reconstruction is
where
As hence the solution is instable. Note, however, that the problem is well
posed and stable at the high frequencies. Since H is a LPF, then B # 1 at high frequencies.
As seen in Eqs. (4), the high frequency contents of the slice in which a layer is in focus are
retained, while those of the other slice are diminished. Even if high frequency noise is added
during image acquisition, it is amplified only slightly in the reconstruction. This behavior is
quite opposite to typical reconstruction problems, in which instability and noise amplification
appear in the high frequencies.
Iterative solutions have been suggested to similar inversion problems in microscopy [2, 3, 15]
and in other fields. A similar approach was used in [5] to generate special e#ects on occluding
layers, when the inverse filtering needed special care in the low frequency components. The
method that we consider can be visualized as progression along vectors in alternating directions
parallel to the axes in Fig. 2. It converges to the solution from any initial hypothesis for |H| < 1.
As |H| decreases (roughly speaking, as the frequency increases), the constraint lines approach
orthogonality, thus convergence is faster. A single iteration is described in Fig. 3. This is
a version of the Van-Cittert restoration algorithm [24]. With slices g a and g b as the initial
hypotheses for
respectively, at the l'th iteration
SchechnerIJCVfig3.ps

Figure

3: A step in the iterative process. Initial hypotheses for the layers serve as input images for a
processing step, based on Eq. (1). The new estimates are fed back as input for the next iteration.
for odd l, where
. (7)
B(m) has a major e#ect on the amplification of noise added to the raw images g a and g b (with
the noise of the unfocused slice attenuated by H). Again, we see that at high frequencies the
amplification of additive noise approaches 1. As the frequency decreases, noise amplification
increases. The additive DC error increases linearly with m.
Let us define the basic solution as that result of using indicates that we can
do the recovery directly, without iterations, by calculating the kernel (filter) beforehand. m is
a parameter that controls how close the filter
B(m) is to the inverse filter, and is analogous to
regularization parameters in typical inversion methods.
In the spatial domain, Eq. (7) turns into a convolution kernel
| {z }
once
| {z } # h 1b # h 2a
| {z }
| {z }
twice
| {z } #h 1b # h 2a # h 1b # h 2a
| {z }
| {z }
times
The spatial support of - b m is approximately 2dm pixels wide, where d is the blur diameter
(assuming for a moment that both kernels have a similar support). Here, the finite support
of the image has to be taken into account. The larger m is, the larger the disturbing e#ect
of the image boundaries. The unknown surroundings a#ect larger portions of the image. It is
therefore preferable to limit m even in the absence of noise.
This di#culty seems to indicate at a basic limit to the ability to recover the layers. If the
blur diameter d is very large, only a small m can be used, and the initial layer estimation
achieved only by focusing cannot be improved much. In this case the initial slices already show
a good separation of the individual layers, since in each of the two slices, one layer is very
blurred and thus hardly disturbs the other one. On the other hand, if d is small, then in each
slice one layer is focused, while the other is nearly focused - creating confusing images. But
then, we are able to enhance the recovery using a larger m with only a small e#ect of the image
boundaries. Using a larger m leads, however, to noise amplification and to greater sensitivity
to errors in the assumed PSF (see subsection 2.4).
Example
A simulated scene consists of the image of Lena, as the close object, seen reflected through a
window out of which Mt. Shuksan 3 is seen. The original layers appear in the top of Fig. 4.
While any of the layers is focused, the other is blurred by a Gaussian kernel with standard
deviation (STD) of 2.5 pixels. The slices in which each of the layers is focused are shown in the
second row of Fig. 4 (all the images in this work are presented contrast-stretched).
During reconstruction, "mirror" [4] extrapolation was used for the surroundings of the image
in order to reduce the e#ect of the boundaries. The basic solution removes the crosstalk
between the images, but it lacks contrast due to the attenuation of the low frequencies. Using
which is equivalent to 13 iterations, improves the balance between the low frequency
components to the high ones. With larger m's the results are similar.
3 Courtesy of Bonnie Lorimer
SchechnerIJCVfig4.ps
solution
Basic
Originals
Far layer
Close layer
solution
Enhanced
slices
Focused

Figure

4: Simulation results. In the focused slices one of the original layers is focused while the
other is defocus blurred. The basic solution with the correct kernel removes the crosstalk, but the low
frequency content of the images is too low. Approximating the inverse filter with 6 terms
amplifies the low frequency components.
2.2 Similarity to motion-based separation
In separating transparent layers, the fact that the high frequencies can be easily recovered, while
the low ones are noisy or lost, is not unique to this approach. It also appears in results obtained
using motion. Note that, like focus changes, motion leaves the DC component unvaried. In [6],
the results of motion-based recovery of semi-reflected scenes are clearly highpass filtered versions
of the superimposing components. An algorithm presented in [22] was demonstrated in a setup
similar to [6]. In [22], one of the objects is "dominant". It can easily be seen there that even
as the dominant object is faded out in the recovery, considerable low-frequency contamination
remains.
Shizawa and Mase [49] have shown that, in regions of translational motion, the spatiotemporal
energy of each layer resides in a plane, which passes through the origin in the spatiotemporal
frequency domain. This idea was used [12, 13] to generate "nulling" filters to eliminate the contribution
of layers, thus isolating a single one. However, any two of these frequency planes have
a common frequency "line" passing through the origin (the DC), whose components are thus
generally inseparable.
These similarities are examples of the unification of triangulation and DOF approaches
discussed in [38]. In general, Ref. [38] shows that the depth from focus or defocus approaches
are manifestations of the geometric triangulation principle. For example, it was shown that
for the same system dimensions, the depth sensitivity of stereo, motion blur and defocus blur
systems are basically the same. Along these lines, the similarity of the inherent instabilities of
separation based on motion and focus is not surprising.
2.3 Using a focused slice and a pinhole image
Another approach to layer separation is based on using as input a pinhole image and a focused
slice, rather than two focused slices. Acquiring one image via a very small aperture ("pinhole
camera") leads to a simpler algorithm, since just a single slice with one of the layers in focus is
needed. The advantage is that the two images are taken without changing the axial positions
of the system components, hence no geometric distortions arise. Acquisition of such images is
practically impossible in microscopy (due to the significant di#raction e#ects associated with
small objects) but is possible in systems inspecting macroscopic objects.
The "pinhole" image is described by
where 1/a is the attenuation of the intensity due to contraction of the aperture. This image
is used in conjunction with one of the focused slices of Eq. (1), for example g a . The inverse
filtering solution is
where
As in subsection 2.1, S can be approximated by
2a . (12)
2.4 E#ect of error in the PSF
The algorithm suggested in subsection 2.1 computes
B(m)[G a -G b H 2a ]. We normally
assume (Eq. (2)) that G a . If the assumption holds,
B(m) . (13)
Note that, regardless of the precise form of the PSFs, had the imaging PSFs and the PSFs used
in the recovery been equal, the reconstruction would have converged to F 1 as m # when
|H 1b |, |H 2a | < 1. In practice, the imaging PSFs are slightly di#erent, i.e., G a
f
f
f
are some functions of the spatial frequency. This di#erence may be due to inaccurate
prior modeling of the imaging PSFs or due to errors in depth estimation. The reconstruction
process leads to
e
similar relation is obtained for the other layer.
An error in the PSF leads to contamination of the recovered layer by its complementary. The
larger
is, the stronger is the amplification of this disturbance. Note that -
monotonically
increases with m, within the support of the blur transfer function if H 1b H 2a > 0, as is the case
when the recovery PSF's are Gaussians. Note that usually in the low frequencies (which is the
regime of the crosstalk) H 1b , H 2a > 0. Thus, we may expect that the best sense of separation
will be achieved using a small m, actually, one iteration should provide the least contamination.
This is so although the uncontaminated solution obeys -
increases. In other words,
decreasing the reconstruction error does not necessarily lead to less crosstalk.
Both H and
f
H (of any layer) are low-pass filters that conserve the average value of the
images. Hence, E # 0 at the very low and at the very high frequencies, i.e., E is a bandpass
filter. However,
B(m) amplifies the low frequencies. At the low frequencies, their combined
e#ect may have a finite or infinite limit as m #, depending on the PSF models used.
Continuing with the example shown in Fig. 4, where the imaging PSF had an STD of
2.5 pixels, the e#ects of using a wrong PSF in the reconstruction are demonstrated in
Fig. 5. When the PSF used in the reconstruction has STD of 1.25 pixels, negative traces
remain (i.e., brighter areas in one image appear as darker areas in the other). When the PSF
used in the reconstruction has STD of 5 pixels, positive traces remain (i.e., brighter areas in
one image appear brighter in the other). The contamination is slight in the basic solution
but is more noticeable with larger m's, that is, when -
B. So, the separation
seems worse, even though each of the images has a better balance (due to the enhancement of
the low frequencies).
SchechnerIJCVfig5.ps
r =1.25
r =5
r =5
Far layer
Close layer
r =1.25

Figure

5: Simulated images when using the wrong PSF in the reconstruction. The original blur kernel
had a STD of r = 2.5 pixels. Crosstalk between the recovered layers is seen clearly if the STD of the
kernels used is 1.5 or 5 pixels. The contamination increases with m.
We can perform the same analysis for the method described in subsection 2.3. Now there
is only one filter involved, H 2a , since the layer f 1 is focused. Suppose that, in addition to using
H 2a in the reconstruction rather than the true imaging transfer function
f
H 2a , we inaccurately
use the scalar a rather than the true value - a used in the imaging process. Let e denote the
relative error in this parameter, e = (a - a)/-a. We obtain that
e
e
where here
are the results had the imaging defocus kernel been the same as
the one used in the reconstruction and had a. Note the importance of the estimation of
e
defocused layer) is recovered uncontaminated by F 1 . However, even in
this case
e
focused layer) will have a contamination of F 2 , amplified by
3 Seeking the blur kernels
The recovery methods outlined in Section 2 are based on the use of known, or estimated blur
kernels. If the imaging system is of our design, or if it is calibrated, and in addition we have
depth estimates of the layers obtained during the focusing process (e.g., as will be described
in Section 5), we may know the kernels a-priori. Generally, however, the kernels are unknown.
Even a-priori knowledge is sometimes inaccurate. We thus wish to achieve self-calibration, i.e.,
to estimate the kernels out of the images themselves. This will enable blind separation and
restoration of the layers.
To do that, we need a criterion for layer separation. Note that the method for estimating
the blur kernels based on minimizing the fitting error in di#erent layers as in [5] may fail in this
case as the layers are transparent and there is no unique blur kernel at each point. Moreover,
the fitting error is not a criterion for separation. Assume that the statistical dependence of
the real and virtual layers is small (even zero). This is reasonable since they usually originate
from unrelated scenes. The Kullback-Leibler distance measures how far the images are from
statistical independence, indicating their mutual information [11]. Let the probabilities for
certain values -
In practice these probabilities are
estimated by the histograms of the recovered images. The joint probability is
is in practice estimated by the joint histogram of the images, that is, the relative number of
pixels in which -
f 1 has a certain value -
f 2 has a certain value -
f 2 at corresponding pixels.
The mutual information is then
. (18)
In this approach we assume that if the layers are correctly separated, each of their estimates
contains minimum information about the other. Mutual information was suggested and used as
a criterion for alignment in [56, 57], where its maximum was sought. We use this measure to look
for the highest discrepancy between images, thus minimizing it. The distance (Eq. 18) depends
on the quantization of -
, and on their dynamic range, which in turn depends on the
brightness of the individual layers f 1 and f 2 . To decrease the dependence on these parameters,
we performed two normalizations. First, each estimated layer was contrast-stretched to a
standard dynamic range. Then, I was normalized by the mean entropy of the estimated layers,
when treated as individual images. The self information [11] (entropy) of -
f 1 is
and the expression for -
f 2 is similar. The measure we used is
I
indicating the ratio of mutual information to the self information of a layer.
The recovered layers depend on the kernels used. Therefore, the problem of seeking the
kernels can be stated as a minimization problem:
According to subsection 2.4, errors in the kernels lead to crosstalk (contamination) of the
estimated layers, which is expected to increase their mutual information.
There are generally many degrees of freedom in the form of the kernels. On the other hand,
the kernels are constrained: they are non-negative, they conserve energy etc. To simplify the
problem, the kernels can be assumed to be Gaussians. Then, the kernels are parameterized
only by their standard deviations (proportional to the blur radii). This limitation may lead to
a solution that is suboptimal but easier to obtain.
Another possible criterion for separation is decorrelation. Decorrelation was a necessary
condition for the recovery of semi-reflected layers by independent components analysis in [18],
and by polarization analysis in [42, 43]. Note that requiring decorrelation between the estimated
layers is based on the assumption that the original layers are decorrelated: that assumption is
usually only an approximation.
To illustrate the use of these criteria, we search for the optimal blur kernels to separate the
images shown in the second row of Fig. 4. Here we simplified the calculations by restricting
both kernels to be isotropic Gaussians of the same STD, as these were indeed the kernels used in
the synthesis. Hence, the correlation and mutual information are functions of a single variable 4 .
As seen in Fig. 6, using the correct kernel (with STD of 2.5 pixels) yields decorrelated basic
solutions 1), with minimal mutual information (I n is plotted). The positive correlation
for larger values of assumed STD, and the negative correlation for smaller values, is consistent
with the visual appearance of positive and negative traces in Fig. 5. Observe that, as expected
from the theory, in Fig. 5 the crosstalk was stronger for larger m. Indeed, in Fig. 6 the absolute
correlation and mutual information are greater for when the wrong
kernel is used.
In a di#erent simulation, the focused slices corresponding to the original layers shown in the
top of Fig. 4 were created using an exponential imaging kernel rather than a Gaussian, but the
4 The STD was sampled on a grid in our demonstrations. A practical implementation will preferably use
e#cient search algorithms [28] to optimize the mutual information [56, 57].
SchechnerIJCVfig6.ps
normalized information
-0.4
-0.20.20.6correlation
2.5 4
-0.2
-0.4
m0.60.2
Mutual information
Figure

At the assumed kernel STD of 2.5 pixels the basic solutions are decorrelated and
have minimal mutual information (shown normalized), in consistency with the true STD. [Dashed]
The absolute correlation and the mutual information are larger for a large value of m.
STD was still 2.5. The recovery was done with Gaussian kernels. The correlation and mutual
information curves (as a function of the assumed STD) were similar to those seen in Fig. 6.
The minimal mutual information was however at STD of r = 2.2 pixels. There was no visible
crosstalk in the resulting images.
The blurring along the sensor raster rows may be di#erent than the blurring along the
columns. This is because blurring is caused not only by the optical processes, but also from
interpixel crosstalk in the sensors, and the raster reading process in the CCD. Moreover, the
inter-pixel spacing along the sensor rows is generally di#erent than along the columns, thus
even the optical blur may a#ect them di#erently. We assigned a di#erent blur "radius" to each
axis: r row and r column . When two slices are used, as in subsection. 2.1, there are two kernels,
with a total of four parameters. Defining the parameter vector p # (r row
the estimated vector -
p is
When a single focused slice is used in conjunction with a "pinhole" image, as described
in subsection 2.3, the problem is much simpler. There are three parameters to determine:
r row
2a and a. The parameter a is easier to obtain as it indicates the ratio of the light
energy in the wide-aperture image relative to the pinhole image. Ideally, it is the square of the
reciprocal of the ratio of the f-numbers of the camera, in the two states. If, however, the optical
system is not calibrated, or if there is automatic gain control in the sensor, this ratio is not
an adequate estimator of a. a can then be estimated by the ratio of the average values of the
images, for example. Such an approximation may serve as a starting point for better estimates.
When using the decorrelation criterion in the multi-parameter case, there may be numerous
parameter combinations that lead to decorrelation, but will not all lead to the minimum mutual
information, or to good separation. If p is N-dimensional, the zero-correlation constraint defines
a dimensional hypersurface in the parameter space. It is possible to use this criterion
to obtain initial estimates of p, and search for minimal mutual information within a lower
dimensional manifold. For example, for each combination of r row and r column , a that leads to
decorrelation can be found (near the rough estimate based on intensity ratios). Then the search
for minimum mutual information can be limited to a subspace of only two parameters.
4 Recovery experiments
4.1 Recovery from two focused slices
A print of the "Portrait of Doctor Gachet" (by van-Gogh) was positioned closely behind a glass
window. The window partly reflected a more distant picture, a part of a print of the "Parasol"
(by Goya). The f# was 5.6. The two focused slices 5 are shown at the top of Fig. 7. The cross
correlation between the raw (focused) images is 0.98. The normalized mutual information is
I n # 0.5 indicating that significant separation is achieved by the focusing process, but that
substantial crosstalk remains.
The optimal parameter vector -
p in the sense of minimum mutual information is [1.9, 1.5,
1.5, 1.9] pixels, where r 1b corresponds to the blur of the close layer, and r 2a corresponds to the
blur of the far layer. With these parameters, the basic solution shown at the middle
row of Fig. 7 has I n # 0.006 (two orders of magnitude better than the raw images). Using
better balance between the low and high frequency components, but I n increased
to about 0.02. We believe that this is due to the error in the PSF model, as discussed above.
In another example, a print of the "Portrait of Armand Roulin" (by van-Gogh) was positioned
closely behind a glass window. The window partly reflected a more distant picture, a
print of a part of the "Miracle of San Antonio" (by Goya). As seen in Fig. 8, the "Portrait"
is hardly visible in the raw images. The cross correlation between the raw (focused) images
is 0.99, and the normalized mutual information is I n # 0.6. The optimal parameter vector -
5 The system was not telecentric, so there was slight magnification with change of focus settings. This was
compensated for manually by resizing one of the images.
solution
Basic
Close layer Far layer
slices
Focused

Figure

7: [Top] The slices in which either of the transparent layers is focused. [Middle row] The basic
solution 1). [Bottom row]: Recovery with
Close layer Far layer
Focused
slices
Basic
solution

Figure

8: [Top] The slices in which either of the transparent layers is focused. [Middle row]
The basic solution. [Bottom row]: Recovery with
here is [1.7, 2.4, 1.9, 2.1] pixels. With these parameters I n # 0.004 at the basic solution, rising
to about
In a third example, the scene consisted of a distant "vase" picture that was partly-reflected
from the glass-cover of a closer "crab" picture. The imaging system was telecentric [33, 59],
so no magnification corrections were needed. The focused slices and the recovered layers are
shown in Fig. 9. For the focused slices I n # 0.4, and the cross correlation is 0.95. The
Far layer
Close layer

Figure

9: [Top] The slices in which either of the transparent layers is focused. [Bottom] The basic
solution for the recovery of the "crab" (left) and "vase" (right) layers.
optimal parameter vector -
p in the sense of minimum mutual information is [4,4,11,1] pixels.
The basic recovery, using -
B(1), are shown in the bottom of Fig. 9. The crosstalk is significantly
reduced. The mutual information I n and correlation decreased dramatically to 0.009 and 0.01,
respectively.
4.2 Recovery from a focused slice and a pinhole image
The scene consisted of a print of the "Portrait of Armand Roulin" as the close layer and a
print of a part of the "Miracle of San Antonio" as the far layer. The imaging system was not
telecentric, leading to magnification changes during focusing. Thus, in such a system it may be
preferable to use a fixed focus setting, and change the aperture between image acquisitions. The
"pinhole" image was acquired using the state corresponding to the mark on the lens,
layer
focused
layer
defocused
layer
defocused
layer
focused

Figure

10: [Top left] The slice in far layer is focused, when viewed with the wide aperture. [Top right]
The "pinhole" image. [Middle row]: The basic recovery. [Bottom row]: Recovery with
while the wide aperture image was acquired using the state corresponding to the
We stress that we have not calibrated the lens, so these marks do not necessarily correspond to
the true values. The slice in which the far layer is focused (using the wide aperture) is shown
in the top left of Fig. 10. In the "pinhole" image (top right), the presence of the "Portrait"
layer is more noticeable.
According to the ratio of the f#'s, the wide aperture image should have been brighter than
the "pinhole" image by (11/4) 2
# 7.6. However, the ratio between the mean intensity of the
wide aperture image to that of the pinhole image was 4.17, not 7.6. This could be due to
poor calibration of the lens by its manufacturer, or because of some automatic gain control in
the sensor. We added a to the set of parameters to be searched in the optimization process.
In order to get additional cues for a, we calculated ratios of other statistical measures: the
ratios of the STD, median, and mean absolute deviation were 4.07, 4.35 and 4.22, respectively.
We thus let a assume values between 4.07 and 4.95. In this example we demonstrate the
possibility of using decorrelation to limit the minimum mutual information search. First, for
each hypothesized pair of blur diameters, the parameter a that led to decorrelation of the basic
solution was sought. Then, the mutual information was calculated over the parameters that
cause decorrelation. The blur diameters that led to minimal mutual information at
pixels, with the best parameter a being 4.28. The reconstruction results
are shown in the middle row of Fig. 10. Their mutual information (normalized) is 0.004.
Using a larger m with these parameters increased the mutual information, so we looked
for a better estimate, minimizing the mutual information after the application of -
B(m). For
the resulting parameters were di#erent: r row = r pixels, with a = 4.24. The
recovered layers are shown in the bottom row of Fig. 10. Their mutual information (normalized)
is 0.04. As discussed before, the increase is probably due to inaccurate modeling of the blur
kernel.
5 Obtaining the focused slices
5.1 Using a standard focusing technique
We have so far assumed that the focused slices are known. We now consider their acquisition
using focusing as in Depth from Focus (DFF) algorithms. Depth is sampled by changing the
focus settings, particularly the sensor plane. According to Refs. [1, 26, 38, 39], the sampling
should be at depth of field intervals, for which d #x, where #x is the inter-pixel period
(similar to stereo [38]). An imaging system telecentric on the image side [33, 59] is a preferred
configuration, since it ensures constant magnification as the sensor is put out of focus. For such
a system it is easy to show that the geometrical-optics blur-kernel diameter is
where D is the aperture width, F is the focal length (see Fig. 1), and #v is the distance of the
sensor plane from the plane of best focus. The axial sampling period is therefore #v # F#x/D.
The sampling period requirement can also be analyzed in the frequency domain, as in [54].
Focus calculations are applied to the image slices acquired. The basic requirement from the
focus criterion is that it will reach a maximum when the slice is in focus. Most criteria suggested
in the literature [23, 25, 32, 34, 52, 55, 62] are sensitive to two dimensional variations in the
slice 6 . Local focus operators yield "slices of local focus-measure", FOCUS (x, y, -
v), where - v is
the axial position of the sensor (see Fig. 1). If we want to find the depth at a certain region
(patch) [31], and the scene is composed of a single layer, we can average FOCUS (x, y, -
v) over
the region, to obtain FOCUS (-v) from which a single valued depth can be estimated. This
approach is inadequate in the presence of multiple layers. Ideally, each of them alone would
lead to a main peak 7 in FOCUS (-v). But, due to mutual interference, the peaks can move
from their original positions, or even merge into a single peak in some "average" position, thus
spoiling focus detection.
This phenomenon can be observed in experimental results. The scene, the focused slices of
which are shown in Fig. 9, had the "crab"and the "vase" objects at distances of 2.8m and 5.3m
from the lens, respectively. The details of the experimental imaging system are described in
[40]. Depth variations within these objects were negligible with respect to the depth of field.
6 It is interesting to note that a mathematical proof exists [21] for the validity of a focus criterion that is
completely based on local calculations which do not depend on transversal neighbors: As a function of axial
position, the intensity at each transversal point has an extremum at the plane of best focus.
7 There are secondary maxima, though, due to the unmonotonicity of the frequency response of the blur
operator, and due to edge bleeding. However, the misleading maxima are usually much smaller than the
maximum associated with the focusing on feature-dense regions, as edges.
Extension of the STD of the PSF by about 0.5 pixels was accomplished by moving the sensor
array 0.338mm from the plane of best focus 8 . This extended the e#ective width of the kernel by
about 1 pixel (#d # 1pixel), and was also consistent with our subjective sensation of DOF. The
results of the focus search, shown by the dashed-dotted line in Fig. 11, indicate that the focus
measure failed to detect the layers, as it yielded a single (merged) peak, somewhere between
the focused states of the individual layers. This demonstrates the confusion of conventional
autofocusing devices when applied to transparent scenes.
5.2 A voting scheme
Towards solving the merging problem, observe that the layers are generally unrelated and that
edges are usually sparse. Thus, the positions of brightness edges in the two layers will only
sporadically coincide. Since edges (and other feature-dense regions) are dominant contributors
to the focus criterion, it would be wise not to mix them by brute averaging of the local focus
measurements over the entire region. If point (x, y) is on an edge in one layer, but on a smooth
region in the other layer, then the peak in FOCUS (x, y, - v) corresponding to the edge will not
be greatly a#ected by the contribution of the other layer.
The following approach is proposed. For each pixel (x, y) in the slices, the focus measure
FOCUS (x, y, - v) is analyzed as a function of - v, to find its local maxima. The result is expressed
as a binary vector of local maxima positions. Then, a vote table analogous to a histogram of
maxima locations over all pixels is formed by summing all the "hits" in each slice-index. Each
vote is given a weight that depends on the corresponding value of FOCUS (x, y, -
v), to enhance
the contribution of high focus-measure values, such as those arising from edges, while reducing
the random contribution of featureless areas. The results of the voting method are shown
as a solid line in Fig. 11, and demonstrate its success in creating significant, separate peaks
8 Near the plane of best focus, the measured rate of increase of the STD as a function of defocus was much
lower than expected from geometric considerations. We believe that this is due to noticeable di#raction and
spherical aberration e#ects in that regime.
SchechnerIJCVfig11.ps
Slice index
votes
Weighted
Traditional
measure
focus
Figure

11: Experimental results. [Dashed-dotted line]: The conventional focus measure as a function
of the slice index. It mistakenly detects a single focused state at the 6th slice. [Solid line]: The
locations histogram of detected local maxima of the focus measure (the same scene). The highest
numbers of votes (positions of local maxima) are correctly accumulated at the 4th and 7th slices - the
true focused slices.
corresponding to the focused layers. Additional details can be found in [40]. The estimated
depths were correct, within the uncertainty imposed by the depth of field of the system. Optimal
design and rigorous performance evaluation of DFF methods in the presence of transparencies
remains an open research problem.
6 Conclusions
This paper presents an approach based on focusing to separate transparent layers, as appear
in semi-reflected scenes. This approach is more stable with respect to perturbations [38] and
occlusions than separation methods that rely on stereo or motion. We also presented a method
for self calibration of the defocus blur kernels given the raw images. It is based on minimizing
the mutual information of the recovered layers. Note that defocus blur, motion blur, and
stereo disparity have similar origins [38] and di#er mainly in the scale and shape of the kernels.
Therefore, the method described here could possibly be adapted to finding the motion PSFs or
stereo disparities in transparent scenes.
In some cases the methods presented here are also applicable to multiplicative layers [49]: If
the opacity variations within the close layer are small (a "weak" object), the transparency e#ect
may be approximated as a linear superposition of the layers, as done in microscopy [2, 10, 29, 37].
In microscopy and in tomography, the suggested method for self calibration of the PSF can
improve the removal of crosstalk between adjacent slices.
In the analysis and experiments, depth variations within each layer have been neglected.
This approximation holds as long as these depth variations are small with respect to the depth
of field. Extending our analysis and recovery methods to deal with space-varying depth and
blur is an interesting topic for future research. A simplified interim approach could be based
on application of the filtering to small domains in which the depth variations are su#ciently
small. Note that the mutual information recovery criterion can still be applied globally, leading
to a higher-dimensional optimization problem. We believe that fundamental properties, such
as the inability to recover the DC of each layer, will hold in the general case. Other obvious
improvements in the performance of the approach can be achieved by incorporating e#cient
search algorithms to solve the optimization problem [28], with e#cient ways to estimate the
mutual information [56, 57].
Semi-reflections can also be separated using polarization cues [18, 41, 42, 43, 45]. It is
interesting to note that polarization based recovery is typically sensitive to high frequency
noise at low angles of incidence [45]. On the other hand, DC recovery is generally possible
and there are no particular di#culties at the low frequencies. This nicely complements the
characteristics of focus-based layer separation, where the recovery of the high frequencies is
stable but problems arise in the low frequencies. Fusion of focus and polarization cues for
separating semi-reflections is thus a promising research direction.
The ability to separate transparent layers can be utilized to generate special e#ects. For
example, in Ref. [5] images were rendered with each of the occluding (opaque) layers defocused,
moved and enhanced arbitrarily. The same e#ects, and possibly other interesting ones can now
be generated in scenes containing semireflections.

Acknowledgments

This work was conducted while Yoav Y. Schechner was at the Department of Electrical Engi-
neering, Technion - Israel Institute of Technology, Haifa. We thank Joseph Shamir and Alex
Bekker for their advice, support, and significant help, and for making the facilities of the Elec-
trooptics Laboratory of the Electrical Engineering Department, Technion, available to us. We
thank Bonnie Lorimer for the permission to use her photograph of Mt. Shuksan. This research
was supported in part by the Eshkol Fellowship of the Israeli Ministry of Science, by the Ol-
lendor# Center of the Department of Electrical Engineering, Technion, and by the Tel-Aviv
University Research Fund. The vision group at the Weizmann Institute is supported in part
by the Israeli Ministry of Science, Grant No. 8504. Ronen Basri is an incumbent of Arye
Dissentshik Career Development Chair at the Weizmann Institute.



--R

Active stereo: integrating disparity

Optical sectioning microscopy: Cellular architecture in three dimensions.
Reduction of boundary artifacts in image restoration.
Producing object-based special e#ects by fusing multiple di#erently focused images
A three-frame algorithm for estimating two-component image motion
Estimating multiple depths in semi-transparent stereo images
Digital image processing.
Three dimensional radiographic imaging with a restricted view angle.
Enhanced 3-D reconstruction from confocal scanning microscope images
Elements of information theory.
Separation of transparent motion into layers using velocity-tuned mechanisms
'Nulling' filters and the separation of transparent mo- tions
Pyramid based depth from focus.
3D representation of biostructures imaged with an optical microscope.
Acquisition of 3-D data by focus sensing
Reconstructing 3-D light-microscopic images by digital image processing
Separating reflections from images by use of independent components analysis.
Distribution of actinin in single isolated smooth muscle cells

Simple focusing criterion.
Computing occluding and transparent motions.
Digitized optical microscopy with extended depth of field.
Resolution enhancement of spectra.
A perspective on range-finding techniques for computer vision
Panoramic image acquisition.
Registration and blur estimation methods for multiple di
Linear and nonlinear programming
The missing cone problem and low-pass distortion in optical serial sectioning microscopy
Artifacts in computational optical-sectioning microscopy
Robust focus ranging.
Shape from focus system.
Real time focus range sensor.
Microscopic shape from focus using active illumination.

A theory of specular surface geometry.
Regularized linear method for reconstruction of three-dimensional microscopic objects from optical sections
Depth from defocus vs. Stereo: How di
The optimal axial interval in estimating depth from defocus.
Separation of transparent layers using focus.
Separation of transparent layers by polarization analysis.
Vision through semireflecting media: Polarization analysis.

Blind recovery of transparent and semireflected scenes.
Polarization and statistical analysis of scenes containing a semireflector.
Three dimensional optical transfer function for an annular lens.
On visual ambiguities due to transparency in motion and stereo.
Direct estimation of multiple disparities for transparent multiple surfaces in binocular stereo.
Simultaneous multiple optical flow estimation.
Fundamental restrictions for 3-D light distributions

The optimal focus measure for passive autofocusing and depth from focus.
Digital composition of images with increased depth of focus considering depth information.
Are textureless scenes recoverable?
Defocus detection using a visibility criterion.

III, <Year>1997</Year>. Alignment by maximization of mutual information.
Layered representation for motion analysis.
"Telecentric optics for computational vision"
Perception of multiple transparent planes in stereo vision.
Depth from focusing and defocusing.
Jayasooriah and Sinniah
--TR

--CTR
Thanda Oo , Hiroshi Kawasaki , Yutaka Ohsawa , Katsushi Ikeuchi, The separation of reflected and transparent layers from real-world image sequence, Machine Vision and Applications, v.18 n.1, p.17-24, January 2007
Javier Toro , Frank Owens , Rubn Medina, Using known motion fields for image separation in transparency, Pattern Recognition Letters, v.24 n.1-3, p.597-605, January
D.-M. Tsai , C.-C. Chou, A fast focus measure for video display inspection, Machine Vision and Applications, v.14 n.3, p.192-196, July
Amit Agrawal , Ramesh Raskar , Shree K. Nayar , Yuanzhen Li, Removing photography artifacts using gradient projection and flash-exposure sampling, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
Morgan McGuire , Wojciech Matusik , Hanspeter Pfister , John F. Hughes , Frdo Durand, Defocus video matting, ACM Transactions on Graphics (TOG), v.24 n.3, July 2005
Sarit Shwartz , Michael Zibulevsky , Yoav Y. Schechner, Fast kernel entropy estimation and optimization, Signal Processing, v.85 n.5, p.1045-1058, May 2005
Zhang , Shree Nayar, Projection defocus analysis for scene capture and image display, ACM Transactions on Graphics (TOG), v.25 n.3, July 2006
Marc Levoy , Ren Ng , Andrew Adams , Matthew Footer , Mark Horowitz, Light field microscopy, ACM Transactions on Graphics (TOG), v.25 n.3, July 2006
blur estimation;semireflections;enhancement;optical sectioning;blind deconvolution;depth from focus;inverse problems;signal separation;image reconstruction and recovery;decorrelation