Synthesis of Novel Views from a Single Face Image.
Images formed by a human face change with viewpoint. A new technique is described for synthesizing images of faces from new viewpoints, when only a single 2D image is available. A novel 2D image of a face can be computed without explicitly computing the 3D structure of the head. The technique draws on a single generic 3D model of a human head and on prior knowledge of faces based on example images of other faces seen in different poses. The example images are used to learn a pose-invariant shape and texture description of a new face. The 3D model is used to solve the correspondence problem between images showing faces in different poses. The proposed method is interesting for view independent face recognition tasks as well as for image synthesis problems in areas like teleconferencing and virtualized reality.
Introduction
Given only a driver's license photograph of a
person's face, can one infer how the face might
look like from a different viewpoint? The three-dimensional
structure of an object determines how
the image of the object changes with a change in
viewpoint. With viewpoint changes, some previously
visible regions of the object become oc-
cluded, while other previously invisible regions become
visible. Additionally, the arrangement or
configuration of object regions that are visible in
both views may change. Accordingly, to synthesize
a novel view of an object, two problems must
be addressed and resolved. First, the visible regions
that the new view shares with the previous
view must be redrawn at their new positions. Sec-
ond, regions not previously visible from the view
of the example image must be generated or syn-
thesized. It is obvious that this latter problem is
unsolvable without prior assumptions. For human
which share a common structure, such prior
knowledge can be obtained through extensive experience
with other faces.
The most direct and general solution for the synthesis
of novel views of a face from a single example
image is the recovery the three-dimensional structure
of the face. This three-dimensional model
can be rotated artificially and would give the correct
image for the all points visible in the example
image (i.e. the one from which the model was
obtained). However, without additional assump-
tions, the minimal number of images necessary to
reconstruct a face using localized features is three
(Huang and Lee, 1989), and even the assumption
that a face is bilaterally symmetric reduces this
number only to two (Rothwell et al., 1993; Vetter
and Poggio, 1994). While shape from shading
algorithms have been applied in previous work
to recover the surface structure of a face (Horn,
1987), the inhomogeneous reflectance properties
of faces make surface integration over the whole
face imprecise and questionable. Additionally, the
fact that the face regions visible from a single image
are insufficient to obtain the three-dimensional
structure makes clear, that the task of synthesizing
new views to a given single image of a face,
cannot be solved without prior assumptions about
the structure and appearance of faces in general.
Models that have been proposed previously to
generalize faces from images can be subdivided
into two groups: those drawing on the three-dimensional
head structure and those considering
only view- or image-dependent face models.
In general, the knowledge about faces, which has
been incorporated into flexible three-dimensional
head models, consists of hand-constructed representations
of the physical properties of the muscles
and the skin of a face (Terzopoulos and Waters,
1993; Thalmann and Thalmann, 1995). To adjust
such a model to a particular face, two or more
images were used (Akimoto et al., 1993; Aizawa
et al., 1989). For present purposes, it is difficult
to assess the usefulness of this approach, since generalization
performance to new views from a single
image only has never been reported.
In recent years, two-dimensional image-based face
models have been applied for the synthesis of rigid
and nonrigid face transitions (Craw and Cameron,
1991; Poggio and Brunelli, 1992; Beymer et al.,
1993; Cootes et al., 1995). These models exploit
prior knowledge from example images of
prototypical faces and work by building flexible
image-based representations (active shape models)
of known objects by a linear combination of labeled
examples. These representations are applied
for the task of image search and recognition
(Cootes et al., 1995) or synthesis (Craw and
Cameron, 1991). The underlying coding of an
image of a new object or face is based on linear
combinations of the two-dimensional shape of examples
of prototypical images. A similar method
has been used to synthesize new images of a face
with a different expression or a changed viewpoint
(Beymer et al., 1993) making use of only a single
given image. The power of this technique is
that it uses an automated labeling algorithm that
computes the correspondence between every pixel
in the two images, rather than for only a hand-selected
subset of feature points. The same technique
has been applied recently to the problem of
face recognition across viewpoint change with the
aim of generating additional new views given an
example face image (Beymer and Poggio, 1995).
In spite of the power of this technique, its most
serious limitation is its reliance on the solution of
the correspondence problem across view changes.
Over large changes in viewpoint, this is still highly
problematic due to the frequency with which occlusions
and occluding contours occur. To overcome
this difficulty in the present work, we draw
on the concept of linear object classes, which we
have introduced recently in the context of object
representations (Vetter and Poggio, 1996). The
application of the linear object class approach to
this problem mediates the requirement of image
correspondence across large view changes for success
in novel view synthesis.
MAPPING PROCESS

Figure

1: Two examples of face images (top row) mapped onto a reference face (center) using pixelwise correspondence
established through an optical flow algorithm are shown (lower row). This separates the 2D-shape
information captured in the correspondence field from the texture information captured in the texture mapped onto
the reference face (lower row).
Overview of the Approach
In the present paper, the linear object class approach
is improved and combined with a single
three-dimensional model of a human head for generating
new views of a face. By using these techniques
in tandem, the limitations inherent in each
approach (used alone) can be overcome. Specif-
ically, the present technique is based on the linear
object class method described in (Vetter and
Poggio, 1996), but is more powerful because the
addition of the 3D model allows a much better
utilization of the example images. The 3D-model
also allows the transfer of features particular to an
individual face from the given example view into
new synthetic views. This latter point is an important
addition to the linear class approach, because
it now allows for individual identifying features
like moles and blemishes that are present in "non-
standard" locations on a given individual face, to
be transferred onto synthesized novel views of the
face. This is true even when these blemishes, etc.,
are unrepresented in the "general experience" that
the linear class model has acquired from example
faces. On the other hand, the primary limitation
of a single 3D head model is the well-known difficulty
of representing the variability of head shapes
in general, a problem that the linear class model,
with its exemplar-based knowledge of faces will allow
us to solve.
Another way of looking at the combination of
these approaches returns us to the two-fold problem
we described at the beginning of this paper.
The synthesis of novel views from a single exemplar
image requires the ability to redraw the regions
shared by the two views, and also the ability
to generate the regions of the novel face that
are invisible in the exemplar view. The 3D head
model allows us to solve the former, and linear
object class approach the allows us to solve the
latter.
Linear Object Classes
A linear object class is defined as a 3D object class
for which the 3D shape can be represented as a linear
combination of a sufficiently small number of
prototypical objects. Objects that meet this criterion
have the following important property. New
orthographic views according to uniform affine 3D
transformation can be generated for any object
of the class. Specifically, rigid transformations in
3D, can be generated exactly if the corresponding
transformed views are known for the set of proto-
types. Thus, if the training set consists of frontal
and rotated views of a set of prototype faces, any
rotated view of a new face can be generated from a
single frontal view - provided that the linear class
assumption holds.
The key to this approach is a representation of
an object or face view in terms of a shape vector
and a texture vector (see also (Cootes et al.,
1995; Jones and Poggio, 1995; Beymer and Pog-
gio, 1995)). The separation of 2D-shape and texture
information in images of human faces requires
correspondence to be established for all feature
points. At its extreme, correspondence must be
established for every pixel, between the given face
image and a reference image. As noted previ-
ously, while this is an extremely difficult problem
when large view changes are involved, the linear
object class assumption requires correspondence
only within a given viewpoint - specifically, the
correspondence between a single view of an individual
face and a single reference face imaged
from the same view. Separately for each orien-
tation, all example face images have to be set in
correspondence to the reference face in the same
pose, correspondence between different poses is
not needed. This can be done off-line manually
(Craw and Cameron, 1991; Cootes et al., 1995)
or automatically (Beymer et al., 1993; Jones and
Poggio, 1995; Beymer and Poggio, 1995; Vetter
and Poggio, 1996). Once the correspondence problem
within views is solved, the resultant data can
be separated into a shape and texture vector. The
shape vector codes the 2D-shape of a face image
as deformation or correspondence field to a reference
face (Beymer et al., 1993; Jones and Poggio,
1995; Beymer and Poggio, 1995; Vetter and Pog-
gio, 1996), which later also serves as the origin of
a linear vector space. Likewise the texture of the
exemplar face is coded in a vector of image intensities
being mapped onto corresponding positions
in the reference face image (see also figure 1 lower
row).
The Three-dimensional Head Model
The linear class approach works well for features
shared by all faces (e.g. eyebrows, nose, mouth
or the ears). But, it has limited representational
possibilities for features particular to a individual
face (e.g. a mole on the cheek). For this reason, a
single 3D model of a human head is added to the
linear class approach. Face textures mapped onto
the 3D model can be transformed into any image
showing the model in a new pose. The final "ro-
tated" version of a given face image (i.e. including
moles, etc.) can be generated by applying to this
new image of the 3D model the shape transformation
given through the linear object class ap-
proach. This is described in more detail shortly.
The paper is organized as follows. First, the algorithm
for generating new images of a face from
a single example image is described. The technical
details of the implementation used to realize
the algorithm on grey level images of human
faces are described in the Appendix. Under Results
a comparison of different implementations
of the generalization algorithm are shown. Two
variations of the combined approach are compared
with a method based purely on the linear object
class as described previously (Vetter and Poggio,
1996). First, the linear class approach is applied
to the parts of a face separately. The individual
parts in the two reference face images were separated
using the 3D-model. Second, the 3D-model
was used additionally to establish pixelwise correspondence
between the two reference faces images
in the two different orientations. This correspondence
field allows texture mapping across the view
point change. Finally, the main features and possible
future extensions of the technique are discussed

Approach and Algorithm
In this section an algorithm is developed that allows
for the synthesis of novel views of a face from
from a single example view of the face. For brevity,
in the present paper we describe the application of
the algorithm to the synthesis of a "frontal" view
(i.e., defined in this paper as the novel view) from
an example "rotated" view (i.e., defined in this
paper as the view 24 ffi from frontal). It should be
noted, however, that the algorithm is not at all
restricted to a particular orientation of faces.
The algorithm can be subdivided into three
parts (for an overview see figure 3).
ffl First, the texture and shape information in
an image of a face are separated.
ffl Second, two separate modules, one for texture
and one for shape, compute the texture
and shape representations of a given "ro-
tated" view of a face (in terms of the appropriate
view of the reference face). These
modules are then used to compute the shape
and texture estimates for the new "frontal"
view of that face.
ffl Finally the new texture and shape for a
"frontal" view are combined and warped to
the "frontal" image of the face.
Separation of texture and shape in images of faces:
The central part of the approach is a representation
of face images that consists of a separate texture
vector and 2D-shape vector, each one with
components referring to the same feature points
- in this case pixels. Assuming pixelwise correspondence
to a reference face in the same pose,
a given example image can be represented as fol-
lows: its 2D-shape will be coded as the deformation
field of n selected feature points - in the limit
of each pixel - to the reference image. So the
shape of a face image is represented by a vector
that is by the
y distance or displacement of each feature with
respect to the corresponding feature in the reference
face. The texture is coded as a difference map
between the image intensities of the exemplar face
and its corresponding intensities in the reference
face. Thus, the mapping is defined by the correspondence
field. Such a normalized texture can
be written as a vector
contains the image intensity differences i of the
n pixels of the image. All images of the training
set are mapped onto the reference face of the corresponding
orientation. This is done separately
for each rotated orientation. For real images of
faces the pixelwise correspondences necessary for
this mappings where computed automatically using
a gradient based optical technique which was
already used successfully previously on face images
(Beymer et al., 1993; Vetter and Poggio, 1996).
The technical details for this technique can be
found in appendix B.
Linear shape model of faces: The shape model of
human faces used in the algorithm is based on the
linear object class idea (the necessary and sufficient
conditions are given in (Vetter and Poggio,
is built on a training set of pairs of
images of human faces. From each pair of im-
ages, each consisting of a "rotated" and a "frontal"
view of a face, the 2D-shape vectors s r for the
"rotated" shape and s f for the "frontal"shape are
computed. Consider the three-dimensional shape
of a human head defined in terms of pointwise
features. The 3D-shape of the head can be represented
by a vector
that contains the x; y; z-coordinates of its n feature
points. Assume that S 2 ! 3n is the linear combination
of q 3D shapes S i of other heads, such
It is quite obvious that for
any linear transformation R (e.g. rotation in 3D)
Thus, if a 3D head shape can be represented as
the weighted sum of the shapes of other heads,
its rotated shape is a linear combination of the
rotated shapes of the other heads with the same
weights fi i .
To apply this to the 2D face shapes computed
from images, we have to consider the following.
projection P from 3D to 2D with s
under which the minimal number q of shape vectors
necessary to represent
i does not change, it allows the
correct evaluation of the coefficients fi i from the
images. Or in other words, the dimension of a
three-dimensional linear shape class is not allowed
to change under a projection P . Assuming such a
projection, and that s r , a 2D shape of a given "ro-
tated" view, can be represented by the "rotated"
shapes of the example set s r
i as
then the "frontal" 2D-shape s f to a given s r can be
computed without knowing S using fi i of equation
(1) and the other s f
given through the images in
the training set with the following equation:
In other words, a new 2D face shape can be
computed without knowing its three-dimensional
structure. It should be noted that no knowledge of
correspondence between equation (1) and equation
(2) is necessary (rows in a linear equation system
can be exchanged freely).
Texture model of faces: In contrast to the shape
model, two different possibilities for generating a
"frontal" texture given a "rotated" texture are de-
scribed. The first method is again based on the linear
object class approach and the second method
uses a single three-dimensional head model to map
the texture from the "rotated" texture onto the
"frontal" texture. The linear object class approach
for the texture vectors is equivalent to the method
described earlier for the 2D-shape vectors. It is
assumed that a "rotated" texture T r can be represented
by the q "rotated" textures T r
computed
from the given example set as follows:
It is assumed further that the new texture T f
can be computed using ff i of equation (3) and the
other
given through the "frontal" images in
the training set by the following equation:
The three-dimensional head model: Whereas the
linear texture approach is satisfactory for generating
new "frontal" textures for regions not visible
Correspondence
of parts
Correspondence
of pixels
Reference Faces

Figure

2: A three-dimensional model of a human head was used to render the reference images (column
the linear shape and texture model. The model defines corresponding parts in the two images (column B) and
also establishes pixelwise correspondence between the two views (column C). Such a correspondence allows texture
mapping from one view (C1) to the other (C2).
in the "rotated" texture, it is not satisfactory for
the regions visible in both views. The linear texture
approach is hardly able to capture or represent
features which are particular to an individual
face (e.g. freckles, moles or any similar distinct
aspect of facial texture). Such features ask for a
direct mapping from the given "rotated" texture
onto the new "frontal" texture. However, this requires
pixelwise correspondence between the two
views (see (Beymer et al., 1993)) .
Since all textures are mapped onto the reference
face, it is sufficient to solve the correspondence
problem across the the viewpoint change for the
reference face only. A three-dimensional model
of an object intrinsically allows the exact computation
of a correspondence field between images
of the object from different viewpoints, because
the three-dimensional coordinates of the whole object
are given, occlusions are not problematic and
hence the pixels visible in both images can be separated
from the pixels which are only visible from
one viewpoint.
A single three-dimensional model of a human
head is incorporated into the algorithm for three
different processing steps.
1. The reference face images used for the formation
of the linear texture and 2D-shape
representations were rendered from the 3D-
model under ambient illumination conditions
(see figure 2A).
2. The 3D-model was manually divided into separate
parts, the nose, the eye and mouth region
and the rest of the model. Using the projections
of these parts, the reference images
for different orientations could be segmented
into corresponding parts for which the linear
texture and 2D-shape representation could
be applied separately (see next paragraph on
"The shape and texture models applied to
parts" and also figure 2B).
3. The correspondence field across the two different
orientations was computed for the two
reference face images based on the given 3D-
model. So the visible part of any texture,
mapped onto the reference face in one orien-
tation, can now be mapped onto the reference
face in the second orientation (see figure 2C
and 3).
To synthesize a complete texture map on the
"frontal" reference face for a new view, (i.e., the
regions invisible in the exemplar view are lacking),
the texture of the region visible in both views,
which has been obtained through direct texture
mapping across the viewpoint change, is merged
with the texture obtained through the linear class
approach (see figure 3). The blending technique
used to merge the regions is described in detail in
the appendix D.
The shape and texture models applied to parts.
The linear object class approach for 2D-shape and
texture, as proposed in (Vetter and Poggio, 1996),
can be improved through the 3D-model of the reference
face. Since the linear object class approach
CONSTRUCTED TEXTURE
COMBINED TEXTURE
LINEAR COMBINATION
OF FRONTAL TEXTURES
OUTPUT IMAGES
REAL FACE
l i
INPUT IMAGE
NORMALIZED TEXTURE
MAPPED TEXTURE
Input - S l T I i)
INPUT SHAPE S Input
CONSTUCTED SHAPE SF
of Frontal View
LINEAR COMBINATION
OF FRONTAL SHAPES
APPROXIMATION
APPROXIMATION
MAPPING INPUT IMAGE ONTO REFERENCE IMAGE

Figure

3: Overview of the algorithm for synthesizing a new view from a single input image. After mapping the
input image onto a reference face in the same orientation, texture and 2D-shape can be processed separately. The
example based linear face model allows the computation of 2D-shape and texture of a new "frontal" view. Warping
the new texture along the new deformation field (coding the shape) results in the new "frontal" views as output.
In the lower row on the right the result purely based on the linear class approach applied to parts is shown, in the
center the result with texture mapping from the "rotated" to the "frontal" view using a single generic 3D model of
a human head. On the bottom left the real frontal view of the face is shown.
did not assume correspondence between equations
(1) and (2) or (3) and (4), shape and texture vectors
had to be constructed for the complete face
as a whole. On the other hand, modeling parts
of a face (e.g. nose, mouth or eye region ) in independent
separate linear classes is highly prefer-
able, because it allows a much better utilization of
the example image set and therefore gives a much
more detailed representation of a face. A full set of
coefficients for shape and texture representation is
evaluated separately for each part instead of just
one set for the entire face.
To apply equations (1 - 4) to individual parts of
a face, it is necessary to isolate the corresponding
it areas in the "rotated' and ``frontal'' reference
images. Such a separation requires the correspondence
between the "rotated' and ``frontal'' reference
image or equivalent between equations (1)
and (2) of the shape representation and also between
equations (3) and (4) for the texture. The
3D-model, however, used for generating the reference
face images determines such a correspondence
immediately (for example see figure 2B) and
allows the separate application of the linear class
approach to parts. To generate the final shape and
texture vector for the whole face, this separation
adds only a few complexities to the computational
process . Shape and texture vectors obtained for
the different parts must be merged, which requires
the use of blending techniques to suppress visible
border effects. The blending technique used
to merge the regions is described in detail in appendix
D.
The algorithm was tested on 100 human faces. For
each face, images were given in two orientations
(24 ffi and 0 ffi ) with a resolution of 256-by-256 pixels
and 8 bit (more details are given in appendix A).
In a leave-one-out procedure, a new "frontal"
view of a face was synthesized to a given "rotated"
view (24 ffi ). In each case the remaining 99 pairs
of face images were used to build the linear 2D-
shape and texture model of faces. Figure 4 shows
the results for six faces for three different implementations
of the algorithm (center rows A,B,C).
The left column shows the test image given to the
algorithm. The true "frontal" view to each test
face from the data base is shown in the right col-
umn. The implementation used for generating the
images in column A was identical to the method
already described in (Vetter and Poggio, 1996),
the linear object class approach was applied to the
shape and texture vector as a whole, no partitioning
of the reference face or texture mapping across
the viewpoints was applied. The method used in
was identical to A, except that the linear object
class approach was applied separately to the
different parts of a face. The three-dimensional
head model was divided into four parts (see figure
2B) the eye, nose, mouth region, and the remaining
part of the face. To segment the two reference
images correctly, it was clearly necessary to render
both of them from the same three-dimensional
model of a head. Based on this segmentation,
the texture and 2D-shape vectors for the different
parts were separated and for each part a separate
linear texture and 2D-shape model was ap-
plied. The final image was rendered after merging
the new shape and texture vectors of the parts.
The images shown in column C are the result of a
combination of the technique described in B and
texture mapping across the viewpoint change. After
mapping a given "rotated" face image onto the
"rotated" reference image, this normalized texture
can be mapped onto the "frontal" reference face
since the correspondence between the two images
of the reference face is given through the three-dimensional
model. The part of the "frontal" texture
not visible in the "rotated" view is substituted
by the texture obtained by the linear texture
model as described under B.
The quality of the synthesized "frontal" views was
tested in a simple simulated recognition experi-
ment. For each synthetic image, the most similar
frontal face image in the data base of 130 faces was
computed. For the image comparison, two common
similarity measures were used: a) the correlation
coefficient, also known as direction cosine;
and b) the Euclidean distance (L 2 ). Both measures
were applied to the images in pixel representation
without further processing.
The recognition rate of the synthesized images
(type A,B,C) was 100 % correct, both similarity
measures independently evaluated the true
"frontal" view to a given "rotated" view of a face
as the most similar image. This result holds for all
three different methods applied for the image syn-
thesis. The similarity of the synthetic images to
the real face image improved by applying the linear
object class approach separately to the parts
and improved again adding the correspondence between
the two reference images to the method.
This improvement is indicated in figure 5 where
decreases where as the correlation coefficients
increase for the different techniques.
INPUT
ROTATED
SYNTHESIZED IMAGES

Figure

4: Synthetic new frontal views (center columns) to a single given rotated (24 ffi ) image of a face (left column)
are shown. The prior knowledge about faces was given through a training set of 99 pairs of images of different
faces (not shown) in the two orientations. Column A shows the result based purely on the linear object class
approach. Adding a single 3D-head model, the linear object class approach can be applied separately to the nose,
mouth and eye region in a face (column B). The same 3D-model allows the texture mapping across the viewpoint
change (column C). The frontal image of the real face is shown in the right column.
Average Image Distance to Nearest Neighbor
Real Face Images 4780.3 0.9589
Synthetic Images Type A 3131.9 0.9811
Synthetic Images Type B 3039.3 0.9822
Synthetic Images Type C 2995.0 0.9827

Figure

5: Comparing the different image synthesis techniques using Direction Cosines and L2-Norm as distance
measures. First, for all real frontal face images the average distance to its nearest neighbor (an image of a
different computed over an images test set of 130 frontal face images. Second, for all synthetic images
(type A,B,C) the average value to its nearest neighbor was computed for both distance measures. For all synthetic
images the real face image was found as nearest neighbor. Switching from technique A to B and from B to C the
average values of Direction Cosines increase whereas the values of the L2-Norm decrease, indicating an improved
image similarity.
A crucial test for the synthesis of images is a direct
comparison of real and synthetic images by human
observers. In a two alternative forced choice
subjects were asked to decide which of the
two frontal face images matches a given rotated
image (24 ffi ) best. One image was the "real" face
the other a synthetic image generated applying the
linear class method to the parts of the faces separately
(method B). The first five images of the
data set were used to familiarize the subjects with
the task, whereas the performance was evaluated
on the remaining 95 faces. Although there was no
time limit for a response and all three images were
shown simultaneously, there were only 6 faces classified
correctly by all 10 subjects (see figure 6). In
all other cases the synthetic image was at least by
one subject classified as the true image and in one
case the synthetic image was found to match the
rotated image better as the real frontal image. In
average each observer was 74% correct whereas the
chance level was at 50%. The subjects responded
in average after 12 seconds.
The results demonstrate clearly an improvement
in generating new synthetic images of a human
face from only a single given example view, over
techniques proposed previously (Beymer and Pog-
gio, 1995; Vetter and Poggio, 1996). Here a single
three-dimensional model of a human head was
added to the linear class approach. Using this
model the reference images could be segmented
into corresponding parts and additionally any texture
on the reference image could be mapped precisely
across the view point change. The information
used from the three-dimensional model is
equivalent to the addition of a single correspondence
field across the viewpoint change. This addition
increased the similarity of the synthesized
image to the image of the real face for the shape
as well as for the texture. The improvement could
be demonstrated in automated image comparison
as well as in perceptual experiments with human
observers.
The results of the automated image comparison indicate
the importance of the proposed face model
for viewpoint independent face recognition sys-
tems. Here the synthetic rotated images were compared
with the real frontal face image. It should
also be noted, that coefficients, which result from
the decomposition of shape and texture into example
shapes and textures, already give us a representation
which is invariant under any 3D affine
transformation, supposing of course the linear face
model holds a good approximation of the target
The difficulties experienced by human observers in
distinguishing between the synthetic images and
the real face images indicate, that a linear face
model of 99 faces segmented into parts gives a
good approximation of a new face, it also indicates
possible applications of this method in computer
graphics. Clearly, the linear model depends on the
given example set, so in order to represent
from a different race or a different age group, the
model would clearly need examples of these, an
effect well known in human perception (cf. e.g.
(O'Toole et al., 1994)).
The key step in the proposed technique is a dense
correspondence field between images of faces seen
from the same view point. The optical flow technique
used for the examples shown worked well,
however, for images obtained under less controlled
conditions a more sophisticated method for finding
the correspondence might be necessary. New
Classification of Synthetic Versus Real Face Images
Number
of 6 17 22
Faces

Figure

For 95 different faces a rotated image (24 ffi ) and two frontal images were shown to human observers
simultaneously. They had to decide which of the frontal images was the synthesized image (type B) and which
one was the real image. The table shows the error rate for 10 observers and the related number of faces. In
average each observer was correct in 74% of the trails (chance level was 50%) and the average response time was
seconds.
correspondence techniques based on active shape
models (Cootes et al., 1995; Jones and Poggio,
are more robust against local occlusions and
larger distortions when applied to a known object
class. There shape parameters are optimized actively
to model the target image.
Several open questions remain for a fully automated
implementation. The separation of parts
of an object to form separated subspaces could
be done by computing the covariance between the
pixels of the example images. However, for images
at high resolution, this may need thousands of example
images. The linear object class approach
assumes that the orientation of an object in an
image is known. The orientation of faces can be
approximated computing the correlation of a new
image to templates of faces in various orientations
(Beymer, 1993). It is not clear jet how precisely
the orientation should be estimated to yield satisfactory
results.


Appendix


A Face Images.
pairs of images of caucasian faces, showing a
frontal view and a view taken 24 ffi from the frontal
were available. The images were originally rendered
for psychophysical experiments under ambient
illumination conditions from a data base of
three-dimensional human head models recorded
with laser scanner (Cyberware TM ). All faces were
without makeup, accessories, and facial hair. Ad-
ditionally, the head hair was removed digitally
(but with manual editing), via a vertical cut behind
the ears. The resolution of the grey-level images
was 256-by-256 pixels and 8 bit.
Preprocessing: First the faces were segmented
from the background and aligned roughly by automatically
adjusting them to their two-dimensional
centroid. The centroid was computed by evaluating
separately the average of all x; y coordinates
of the image pixels related to the face independent
of their intensity value.
A single three-dimensional model of a human
head, recorded with a laser scanner
(Cyberware TM ), was used to render the two reference
images.
B Computation of the
To compute the 2D-shape vectors s r
used
in equations (1) and (2), which are the vectors of
the spatial distances between corresponding points
in the face images, the correspondence of these
points has to be established first. That means we
have to find for every pixel location in an image,
e.g. a pixel located on the nose, the corresponding
pixel location on the nose in the other image.
This is in general a hard problem. However, since
all face images compared are in the same orienta-
tion, one can assume that the images are quite
similar and occlusions are negligible. The simplified
condition of a single view make it feasible
to compare the images of the different faces
with automatic techniques. Such algorithms are
known from optical flow computation, in which
points have to be tracked from one image to the
other. We use a coarse-to-fine gradient-based gradient
method (Bergen et al., 1992) and follow an
implementation described in (Bergen and Hingo-
rani, 1990). For every point x; y in an image I,
the error term
for y, with I x ; I y being the spatial image
derivatives and ffi I the difference of intensity
of the two compared images. The coarse-to-fine
strategy refines the computed displacements when
finer levels are processed. The final result of this
computation (ffix; ffi y) is used as an approximation
of the spatial displacement vector s in equation
(1)and (2). The correspondence is computed towards
the reference image from the example and
test images. As a consequence, all vector fields
have a common origin at the pixel locations of the
reference image.
C Linear shape and texture
synthesis.
First the optimal linear decomposition of a given
shape vector in equation (1) and a given texture
vector in equation (3) was computed. To compute
the coefficients ff i (or similar fi i ) the "initial" vector
T r of the new image is decomposed (in the
sense of least square) to the q training image vectors
given through the training images by min-
imizing
The numerical solution for ff i and fi i was obtained
by an standard SVD-algorithm (Press and Flan-
nery, 1992). The new shape and texture vectors
for the "frontal" view were obtained through simple
summation of the weighted "frontal" vectors
(equations( 2) and (4)).
D Blending of patches.
Blending of patches is used at different steps in
the proposed algorithm. It is applied for merging
different regions of texture as well as for merging
regions of correspondence fields which were computed
separately for different parts of the face.
Such a patch work might have little discontinuities
at the borders between the different patches.
It is known that human observers are very sensitive
to such effects and the overall perception of
the image might be dominated by these.
For images Burt and Adelson (Burt and Adel-
son, 1983; Burt and Adelson, 1985) proposed a
multiresolution approach for merging images or
components of images. First, each image patch is
decomposed into bandpass filtered component im-
ages. Secondly, this component images are merged
separately for each band to form mosaic images by
weighted averaging in the transition zone. Finally,
these bandpass mosaic images are summed to obtain
the desired composite image. This method
was applied to merge the different patches for the
texture construction as well as to combine the
texture mapped across the viewpoint change with
the missing part taken from the constructed one.
Originally this merging method was only described
for an application to images, however, the application
to patches of correspondence fields eliminates
visible discontinuities in the warped images.
Taking a correspondence field as an image with a
vector valued intensity, the merging technique was
applied to the x and y components of the correspondence
vectors separately.
Synthesis of the New Image.
The final step is image rendering. The new image
can be generated combining the texture and shape
vector generated in the previous steps. Since both
are given in the coordinates of the reference image,
for every pixel in the reference image the pixel intensity
and coordinates to the new location are
given. The new location generally does not coincide
with the equally spaced grid of pixels of the
destination image. The final pixel intensities of
the new image are computed by linear interpola-
tion, a commonly used solution of this problem
known as forward warping (Wolberg, 1990).

Acknowledgments

I am grateful to H.H. B-ulthoff and T. Poggio for
useful discussions and suggestions. Special thanks
to Alice O'Toole for editing the manuscript and for
her endurance in discussing the paper. I would like
to thank Nikolaus Troje for providing the images
and the 3D-model.



--R




Automatic creation of 3D facial mod- els

Hierarchical motion-based frame rate conversion
Face recognition under varying pose.
Face recognition from one model view.



Merging images through pattern decomposition.
Active shape models - their training and application
Parameterizing images for recognition and reconstruc- tion
Robot vision.
Motion and structure from orthographic projections.
Synthetizing a color algorithm from examples.

A novel approach to graphics.

Extracting projective structure from single perspective views of 3D point sets.
Analysis and synthesis of facial image sequences using physical and anatomical models.
Digital actors for interactive television.
Recognition by linear combinations of models.
Symmetric 3D objects are an easy case for 2D object recog- nition
Image synthesis from a single example image.

The importance of symmetry and virtual views in three-dimensional object recogni- tion
Image Warping.
--TR

--CTR
Xiaoyang Tan , Songcan Chen , Zhi-Hua Zhou , Fuyan Zhang, Face recognition from a single image per person: A survey, Pattern Recognition, v.39 n.9, p.1725-1745, September, 2006
Philip L. Worthington, Reillumination-driven shape from shading, Computer Vision and Image Understanding, v.98 n.2, p.326-344, May 2005
A. Criminisi , A. Blake , C. Rother , J. Shotton , P. H. Torr, Efficient Dense Stereo with Occlusions for New View-Synthesis by Four-State Dynamic Programming, International Journal of Computer Vision, v.71 n.1, p.89-110, January   2007
Martin A. Giese , Tomaso Poggio, Morphable Models for the Analysis and Synthesis of Complex Motion Patterns, International Journal of Computer Vision, v.38 n.1, p.59-73, June 2000
Bernd Heisele , Thomas Serre , T. Poggio, A Component-based Framework for Face Detection and International Journal of Computer Vision, v.74 n.2, p.167-181, August    2007
Yongmin Li , Shaogang Gong , Heather Liddell, Constructing Facial Identity Surfaces for Recognition, International Journal of Computer Vision, v.53 n.1, p.71-92, June
Athinodoros S. Georghiades , Peter N. Belhumeur , David J. Kriegman, From Few to Many: Illumination Cone Models for Face Recognition under Variable Lighting and Pose, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.23 n.6, p.643-660, June 2001
face recognition;flexible templates;image synthesis;rotation invariance