Person Identification Using Multiple Cues.
AbstractThis paper presents a person identification system based on acoustic and visual features. The system is organized as a set of non-homogeneous classifiers whose outputs are integrated after a normalization step. In particular, two classifiers based on acoustic features and three based on visual ones provide data for an integration module whose performance is evaluated. A novel technique for the integration of multiple classifiers at an hybrid rank/measurement level is introduced using HyperBF networks. Two different methods for the rejection of an unknown person are introduced. The performance of the integrated system is shown to be superior to that of the acoustic and visual subsystems. The resulting identification system can be used to log personal access and, with minor modifications, as an identity verification system.
Introduction
The identification of a person interacting with
computers represents an important task for automatic
systems in the area of information retrieval,
automatic banking, control of access to security ar-
eas, buildings and so on. The need for a reliable
identification of interacting users is obvious. At the
same time it is well known that the security of such
systems is too often violated in every day life. The
possibility to integrate multiple identification cues,
such as password, identification card, voice, face,
fingerprints and the like will, in principle, enhance
the security of a system to be used by a selected set
of people.
This paper describes in detail the theoretical
foundations and design methodologies of a person
recognition system that is part of MAIA, the integrated
AI project under development at IRST [26].
Previous works about speaker recognition [30],
[16] have proposed methods for classifying and combining
acoustic features and for normalizing [27],
[22] the various classifier scores. In particular, score
normalization is a fundamental step when a system
is required to confirm or reject the identity given
by the user (user verification): in this case, in fact,
the identity is accepted or rejected according to a
comparison with a preestimated threshold. Since
the integration of voice and images in an identification
system is a new concept, new methods for both
classifier normalization and integration were inves-
tigated. Effective ways for rejecting an unknown
person by considering score and rank information
and for comparing images with improved similarity
measures are proposed. A simple method for
adapting the acoustic models of the speakers to a
real operating environment also was developed.
The speaker and face recognition systems are decomposed
into two and three single feature classifiers
respectively. The resulting five classifiers produce
non-homogeneous lists of scores that are combined
using two different approaches. In the first
approach, the scores are normalized through a robust
estimate of the location and scale parameters
of the corresponding distributions. The normalized
scores are then combined using a weighted geometric
average and the final identification is accepted
or rejected according to the output of a linear clas-
sifier, based on score and rank information derived
from the available classifiers. Within the second
approach, the problem of combining the normalized
outputs of multiple classifiers and of accept-
ing/rejecting the resulting identification is considered
a learning task. A mapping from the scores
and ranks of the classifiers into the interval (0; 1)
is approximated using an HyperBF network. A
final threshold is then introduced based on cross-
validation. System performance is evaluated and
discussed for both strategies. Because of the novelty
of the problem, standard data-bases for system
training and test are not yet available. For
this reason, the experiments reported in this paper
are based on data collected at IRST. A system implementation
operating in real-time is available and
was tested on a variety of IRST researchers and vis-
itors. The joint use of acoustic and visual features
proved effective in increasing system performance
and reliability.
The system described here represents an improvement
over a recently patented identification
system based on voice and face recognition [6], [9].
The two systems differ in many ways: in the latter
the speaker and face recognition systems are not
further decomposed into classifiers, the score normalization
does not rely on robust statistical techniques
and, finally, the rejection problem is not addressed

The next sections will introduce the speaker and
face recognition systems. The first approach to the
integration of classifiers and the linear accept/reject
rule for the final system identification are then dis-
cussed. Finally, the novel rank/measurement level
integration strategy using an HyperBF network is
introduced with a detailed report on system performance

2. Speaker recognition
The voice signal contains two types of informa-
tion: individual and phonetic. They have mutual
effects and are difficult to separate; this represents
one of the main problems in the development of
automatic speaker and speech recognition systems.
The consequence is that speaker recognition systems
perform better on speech segments having specific
phonetic contents while speech recognition systems
provide higher accuracy when tuned on the
voice of a particular speaker. Usually the acoustic
parameters for a speech/speaker recognizer are derived
by applying a bank of band-pass filters to adjacent
short time windows of the input signal. The
energy outputs of the filters, for various frames, provide
a good domain representation. Figure 1 gives
an example of such an analysis. The speech waveforms
correspond to utterances of the Italian digit 4
(/kwat:ro/) by two different speakers. The energy
outputs of a 24 triangular band-pass filter bank are
represented below the speech waveforms (darker regions
correspond to higher energy values).
Fig. 1. Acoustic analysis of two utterances of the digit 4
(/kwat:ro/) by two different speakers.
In the past years several methods and systems
for speaker identification [13], [16] were proposed
that perform more or less efficiently depending on
the text the user is required to utter (in general,
systems can be distinguished into text dependent or
text independent), the length of the input utterance,
the number of people in the reference database and,
finally, the time interval between test and training
recordings.
For security applications, it is desirable that the
user utter a different sentence during each inter-
action. The content of the utterance can then be
verified to ensure that the system is not cheated by
prerecorded messages. For this work, a text independent
speaker recognition system based on Vector
Quantization (VQ) [28] was built. While it cannot
yet verify the content of the utterance, it can
be modified (using supervised clustering or other
techniques) to obtain this result.
A block diagram of the system is depicted in

Figure

2. In the system, each reference speaker is
represented by means of two sets of vectors (code-
books) that describe his/her acoustic characteris-
tics. During identification, two sets of acoustic features
(static and dynamic), derived from the short
time spectral analysis of the input speech signal,
are classified by evaluating their distances from the
prototype vectors contained in the speaker code-book
couples. In this way, two lists of scores are
sent to the integration module. In the following
both the spectral analysis and vector quantization
techniques will be described in more detail (see also
[21] and a reference book such as [23]).
Since the power spectrum of the speech signal decreases
as frequency increases a preemphasis filter
that enhances the higher frequencies is applied to
the sampled input signal. The transfer function of
the filter is
The preemphasized signal, x(n), 1 - n - N , is
subdivided into frames y t (n), 1 - t - T , having
length L. Each frame is obtained by multiplying
x(n) by an Hamming window h t (n):
(1)
(2)
Voice
Signal
Static Parameters
Static Score List
Dynamic Score List
Analysis
Acoustic
Distance Computation
Distance Computation
Dynamic Codebooks
Static Codebooks
Dynamic Parameters
Fig. 2. The speaker recognition system based on Vector
Quantization.
In the equation above L represents the length,
in samples, of the Hamming window and S is the
analysis step (also expressed in samples). For the
system L and S were chosen to correspond to 20
ms and 10 ms respectively.
The signal is multiplied by an Hamming window
(raised cosine) to minimize the sidelobe effects on
the spectrum of the resulting sequence y t (n).
The acoustic analysis of each frame is performed
as follows:
1. the power spectrum of the sequence y t (n) is
evaluated;
2. a bank of spaced
according to a logarithmic scale (Mel scale), is
applied to the power spectrum and the energy
outputs s tq , 1 - q - Q, from each filter are
evaluated;
3. the Mel Frequency Cepstrum Coefficients
are com-
puted, from the filterbank outputs, according
to the following equation:
the MFCCs are arranged into a vector, ' t ,
which is called static, since it refers to a single
speech frame;
4. to account for the transitional information
contained in the speech signal a linear fit
is applied to the components of 7 adjacent
MFCCs; the resulting regression coefficients
are arranged into a vector that is called dynamic

5. a binary variable is finally evaluated that allows
marking the frame as speech or background
noise; this parameter is computed by means of
the algorithm described in [12].
The Mel scale is motivated by auditory analysis
of sounds. The inverse Fourier transform of the
log-spectrum (cepstrum) provides parameters that
improves performance at both speech and speaker
recognition [23], [29]. Furthermore, the Euclidean
distance between two cepstral vectors represents
a good measure for comparing the corresponding
speech spectra. The static and dynamic 8-
dimensional vectors related to windows marked as
background noise are not considered during both
system training and testing. As previuosly said,
VQ is used to design the static and dynamic codebooks
of a given reference speaker, say the th one.
Starting from a set of training vectors (static or dy-
namic) \Theta g, derived from a certain
number of utterances, the objective is to find a new
set that represents
well the acoustic characteristics of the given
speaker. To do this a clustering algorithm, similar
to that described in [21], is applied to the \Theta i
set. The algorithm makes use of an iterative procedure
that allows determination of codebook cen-
troids, \Psi i , by minimizing their average distance,
from the training vectors:
min m=1
The distance d(' ik ; / im ) is defined as follows:
In the equation above t denotes transpose and
W is the covariance matrix of the training vectors.
The matrix W is estimated from the training data
of all the speakers in the reference database. This
matrix was found to be approximately diagonal, so
that only the diagonal elements are used to evaluate
distances.
In the recognition phase the distances, D Si ; DDi ,
between the static and dynamic vector sequences,
derived from the input signal, and the corresponding
speaker codebooks are evaluated and sent to the
integration module.
If is the static (or dynamic) input
sequence and \Psi i is the i th static (or dynamic)
codebook, then the total static (or dynamic) distance
will be:
min m=1
where I is the total number of speakers in the reference
database.
To train the system, 200 isolated utterances of
the Italian digits (from 0 to were collected for
each reference user. The recordings were realized by
means of a Digital Audio Tape (DAT): the signal on
the DAT tape, sampled at 48 kHz, was downsampled
to manually end-pointed, and stored
on a computer disk. The speech training material
was analyzed and clustered as previously described.
As demonstrated in [28], system performance depends
on both input utterance length and codebook
preliminary experiments have suggested that
the speaker, to be identified, should utter a string
of at least 7 digits in a continuous way and in whatever
order. In the reported experiments the number
of digits was kept equal to 7 and the codebook size
was set to higher values did not
improve recognition accuracy. Furthermore, if input
signal duration is too short, the system requires
the user to repeat the digit string.
To evaluate integrated system performance (see
section 4.1) the reference users interacted 3 times
with the system during 3 different sessions. The
test sessions were carried out in an office environment
using an ARIEL board as acquisition channel.
Furthermore the test phase was performed about
five months after the training recordings. Due
to both the different background noise and acquisition
conditions between training and test, the codebooks
must be adapted.
Adaptation means designing a new codebook,
starting from a given one, that better resembles the
acoustic characteristics of both the operating environment
and the acquisition channel. Adaptation
should also take into account variations in time of
the speaker's voice (intraspeaker variations). Adaptation
requires the use of few utterances to modify
the codebook as it is not necessary to design it from
scratch (this would require at least 30-40 seconds
of speech). In our case, the adaptation vectors are
derived from the digit strings uttered by the users
during a single test session. The dynamic codebooks
were not adapted since they represent temporal
variations of the speech spectra and therefore
they are less sensitive to both intraspeaker voice
variability and acquisition channel variations.
The adaptation process of the i th codebook, C i
can be summarized as follows:
1. the mean vectors - i and - i of the adaptation
vectors and of the given codebook respectively
are evaluated;
2. the difference vector
3. the vectors of C i are shifted by a quantity
equal to \Delta i obtaining a new set C 0
i is placed in the
region of the adaptation vectors;
4. the adaptation vectors are clustered using
the set C 0
i as initial estimate of the cen-
troids; therefore a new set of centroids O
and the corresponding cell occupancies
are evaluated;
5. the adapted codebook \Psi i is obtained according
to the following equation:
In the equation above the parameter nim determines
the fraction of deviation vector
im ), that has to be summed to the initial
centroid c 0
im . Eqn. 7 is a simple method
to modify the centroids of a codebook according
to the number of data available for their
estimates. ffi im can be zero when the utterance
used for adaptation does not contain sounds
whose spectra are related to the m-th centroid.
For the system, ff was chosen equal to 0:1. The
two shifts applied by the adaptation procedure can
be interpreted as follows:
1. the major shift, accounts for environment
and channel variations with respect to training;
2. ffi im , the minor shift, accounts for intra-speaker
voice variations in time.
3. Face recognition
Person identification through face recognition is
the most familiar among the possible identification
strategies. Several automatic or semiautomatic systems
were realized since the early seventies - albeit
with varying degree of success. Different techniques
were proposed, ranging from the geometrical description
of salient facial features to the expansion
of a digitized image of the face on an appropriate
Fig. 3. The highlighted regions represent the templates used
for identification.
basis of images (see [8] for references). The strategy
used by the described system is essentially based
on the comparison, at the pixel level, of selected regions
of the face [8]. A set of regions, respectively
encompassing the eyes, nose, and mouth of the user
to be identified are compared with the corresponding
regions stored in the database for each reference
user (see Figure 3). The images should represent a
frontal view of the user face without marked ex-
pressions. As will be clear from the detailed de-
scription, these constraints could be relaxed at the
cost of storing a higher number of images per user
in the database. The fundamental steps of the face
recognition process are the following:
1. acquisition of a frontal view of the user
2. geometrical normalization of the digitized image

3. intensity normalization of the image;
4. comparison with the images stored in the
database.
The image of the user face is acquired with a CCD
camera and digitized with a frame grabber.
To compare the resulting image with those stored
in the database, it is necessary to register the im-
age: it has to be translated, scaled, and rotated so
that the coordinates of a set of reference points take
corresponding standard values. As frontal views are
considered, the centers of the pupils represent a natural
set of control points that can be located with
good accuracy. Eyes can be found through the following
steps:
1. locate the (approximate) symmetry axis of the
2. locate the left/right eye by using an eye template
for which the location of the pupil is
known; if the confidence of the eye location is
not sufficiently high, declare failure (the identification
system will use only acoustic infor-
3. achieve translation, scale and rotation invariance
by fixing the origin of the coordinate system
at the midpoint of the interocular segment
and the interocular distance and the direction
of the eye-to-eye axis at predefined values.
Under the assumption that the user face is approximately
vertical in the digitized image, a good estimate
of the coordinate S of the symmetry axis is
given by
where   represent convolution, I the image, K V the
convolution kernel [\Gamma1; 0; 1] t , P V the vertical projection
whose index i runs over the columns of the
image. The face can then be split vertically into
two, slightly overlapping parts containing the left
and right eye respectively. The illumination under
which the image is taken can impair the template
matching process used to locate the eye. To minimize
this effect a filter, N (I), is applied to image
I:
where
I   KG(oe)
and KG(oe) is a Gaussian kernel whose oe is related to
the expected interocular distance \Delta ee . The arithmetic
operations act on the values of corresponding
pixels. The process mapping I into N reduces the
influence of ambient lighting while keeping the necessary
image details. This is mainly due to the removal
of linear intensity gradients that are mapped
to the constant value 1. Extensive experiments, using
ray-tracing and texture-mapping techniques to
generate synthetic images under a wide range of
lighting directions have shown that the local contrast
operator of eqn. (9) exhibits a lower illumination
sensitivity than other operators such as the
laplacian, the gradient magnitude or direction [5]
and that there is an optimal value of the parameter
oe (approximately equal to the iris radius).
The same filter is applied to the eye templates.
The template matching process is based on the
algorithm of hierarchical correlation proposed by
Burt [11]. Its final result is a map of correlation
values: the center of gravity of the pixels with maximum
value representing the location of the eye.
Once the two eyes have been located, the confidence
of the localization is expressed by a coefficient, CE ,
that measures the symmetry of the eye positions
with respect to the symmetry axis, the horizontal
alignment and the scale relative to that of the eye
templates:
where C l and C r represent the (maximum) correlation
value for the left/right eye, s the interocular
distance expressed as a multiple of the interocular
distance of the eyes used as templates, \Delta' represents
the angle of the interocular axis with respect
to the horizontal axis while oe ' and oe s represent tolerances
on the deviations from the prototype scale
and orientation.
The first factor in the RHS of eqn. (11) is the
average correlation value of the left and right eye:
the higher it is the better the match with the eye
templates. The second factor represents the symmetry
of the correlation values and equals 1 when
the two values are identical. The third and fourth
factors allow weighing the deviation from both the
assumed scale and (horizontal) orientation of the
interocular axis, respectively. The parameters of
the Gaussians, oe s and oe ' , were determined by the
analysis of a set of interactions.
If the value of CE is too low, the face recognition
system declares failure and the identification proceeds
using the acoustic features alone. Otherwise,
the image is translated, scaled and rotated to match
the location of the pupils to that of the database
images. In the reported experiments the interocular
distance was set equal to 28 pixels. Alternative
techniques for locating eyes are reported in [17],
[32]. Due to the geometrical standardization, the
subimages containing the eyes, nose, and mouth
are approximately characterized by the same coordinates
in every image. These regions are extracted
from the image of the user face and compared in
turn to the corresponding regions extracted from
the database entries, previously filtered according
to eqns. (9)(10). Let us introduce a similarity
measure C based on the computation of the L 1 norm
of a vector
and on the corresponding
distance dL1 (x;
The L 1 distance of two vectors is mapped by
C(\Delta; \Delta) into the interval [0; 1], higher values representing
smaller distances. This definition can be
easily adapted to the comparison of images. For
the comparison to be useful when applied to real
images, it is necessary to normalize the images so
that they have the same average intensity - and
standard deviation (or scale) oe. The latter is particularly
sensitive to values far from the average -
so that the scale of the image intensity distribution
can be better estimated by the following quantity:
oe L1 =n
where the image is considered as a one dimensional
vector x. The matching of an image B to an image
A can then be quantified by the maximum value
of C(A; B), obtained by sliding the smaller of the
Fig. 4. The distribution of the correlation values for corresponding
features of the same person and of different
people.
two images over the larger one. A major advantage
of the image similarity computed according to
eqn. (12) over the more common estimate given by
the cross-correlation coefficient [1], based on the L 2
norm, is its reduced sensitivity to small amounts
of unusually high differences between corresponding
pixels. These differences are often due to noise
or image specularities such as iris highlights. A detailed
analysis of the similarity measure defined in
eqn. (12) is given in [7]. An alternative technique
for face identification is reported in [31]. Let us
denote with fU km gm=1;:::;pk the set of images available
for the k th user. A comparison can now be
made between a set of regions of the unknown image
N and the corresponding regions of the database
images. The regions currently used by the system
correspond to the eyes, nose and mouth. A list of
similarity scores is obtained for each region F ff of
image
fs kff
where R ff (N ) represents a region of N containing
F ff with a frame whose size is related to the interocular
distance. The lists of matching scores corresponding
to eyes, nose, and mouth are then available
for further processing. The distribution of the
correlation values for corresponding features of the
same person and of different people are reported in

Figure

4.
Integration with the scores derived from the
acoustic analysis can now be performed with a single
or double step process. In the first case, the two
acoustic and the three visual scores are combined
simultaneously, while in the second the acoustic and
visual scores are first combined separately and the
final score is given by the integration of the outputs
of the speaker and face recognition systems (see [9]
for an example of the latter). The next section will
introduce two single-step integration strategies for
classifiers working at the measurement level.
4. Integration
The use of multiple cues, such as face and voice,
provides in a natural way the information necessary
to build a reliable, high performance system.
Specialized subsystems can identify (or verify) each
of the previous cues and the resulting outputs can
then be combined into a unique decision by some
integration process. The objective of this section
is to describe and evaluate some integration strate-
gies. The use of multiple cues for person recognition
proved beneficial for both system performance and
reliability 1 .
A simplified taxonomy of multiple classifier systems
is reported in [33]. Broadly speaking, a classifier
can output information at one of the following
levels:
the abstract level: the output is a subset of the
possible identification labels, without any qualifying
the rank level: the output is a subset of the possible
labels, sorted by decreasing confidence
(which is not supplied);
the measurement level: the output is a subset of
labels qualified by a confidence measure.
The level at which the different classifiers of a
composite system work clearly constrains the ways
their responses can be merged. The first of the following
sections will address the integration of the
speaker/face recognition systems at the measurement
level. The possibility of rejecting a user as
unknown will then be discussed. Finally, a novel,
hybrid level approach to the integration of a set of
classifiers will be presented.
4.1 Measurement level integration
The acoustic and visual identification systems already
constitute a multiple classifier system. How-
ever, both the acoustic and visual classifiers can
be further split into several subsystems, each one
based on a single type of feature. In our system,
five classifiers were considered (see secs. 2, working
on the static, dynamic acoustic features, and on
the eyes, nose and mouth regions.
A critical point in the design of an integration
procedure at the measurement level is that of measurement
normalization. In fact, the responses of
the different classifiers usually have different scales
(and possibly offsets), so that a sensible combination
of the outputs can proceed only after the scores
are properly normalized. As already detailed, the
outputs of the identification systems are not ho-
mogeneous: the acoustic features provide distances
while the visual ones provide correlation values. A
Two aspects of reliability are critical for a person identification
system: the first is the ability of rejecting a user
as unknown, the second is the possibility of working with
a reduced input, such as only the speech signal or the face
image.
first step towards the normalization of the scores
is to reverse the sign of distances, thereby making
them concordant with the correlation values: the
higher the value, the more similar the input pat-
terns. Inspection of the score distributions shows
them to be markedly unimodal and roughly sym-
metrical. A simple way to normalize scores is to
estimate their average values and standard deviations
so that distributions can be translated and
rescaled in order to have zero average and unit vari-
ance. The values can then be forced into a standard
interval, such as (0; 1), by means of an hyperbolic
tangent mapping. The normalization of the scores
can rely on a fixed set of parameters, estimated
from the score distributions of a certain number of
interactions, or can be adaptive, estimating the parameters
from the score distribution of the current
interaction. The latter strategy was chosen mainly
because of its ability to cope with variations such as
different speech utterance length without the need
to re-estimate the normalization parameters.
The estimation of the location and scale parameters
of the distribution should make use of robust
statistical techniques [18], [20]. The usual arithmetic
average and standard deviation are not well
suited to the task: they are highly sensitive to outlier
points and could give grossly erroneous esti-
mates. Alternative estimators exist that are sensitive
to the main bulk of the scores (i.e. the central
part of a unimodal symmetric distribution) and are
not easily misled by points in the extreme tails of
the distribution. The median and the Median Absolute
Deviation (MAD) are examples of such location
and scale estimators and can be used to reliably
normalize the distribution of the scores. However,
the median and the MAD estimators have a low
efficiency relative to the usual arithmetic average
and standard deviation. A class of robust estimators
with higher efficiency was introduced by Hampel
under the name of tanh-estimators and is used
in the current implementation of the system (see
[18] for a detailed description). Therefore each list
of scores fS ij g i=1;:::;I from classifier j, being I the
number of people in the reference database, can be
transformed into a normalized list by the following
mapping:
tanh
oe tanh
where - tanh and oe tanh are the average and standard
deviation estimates of the scores fS ij g i=1;:::;I
as given by the Hampel estimators. An example
of distributions of the resulting normalized scores
is reported in Figure 5 for each of the five features
used in the classification.
In the following formulas, a subscript index i m indicates
the m th entry within the set of scores sorted
Fig. 5. The density distribution of the normalized scores for
each of the classifiers: S1, S2 represent the static and
dynamic speech scores while F 1, F2 and F3 represent
the eyes, nose and mouth scores respectively.
by decreasing value. The normalized scores can be
integrated using a weighted geometric average:
ijA
1=
where the weights w j represent an estimate of the
score dispersion in the right tail of the corresponding
distributions:
The main reason suggesting the use of geometric average
for the integration of scores relies on probabil-
ity: if we assume that the features are independent
the probability that a feature vector corresponds
to a given person can be computed by taking the
product of the probabilities of each single feature.
The normalized scores could then be considered as
equivalent to probabilities. Another way of looking
at the geometric average is that of predicate
conjunction using a continuous logic [2], [3]. The
weights reflect the importance of the different features
(or predicates). As defined in eqn. (17), each
feature is given an importance proportional to the
separation of the two best scores. If the classification
provided by a single feature is ambiguous, it is
given low weight. A major advantage of eqn. (16)
is that it does not require a detailed knowledge of
how each feature is distributed (as would be necessary
when using a Bayes approach). This eases
the task of building a system that integrates many
features.
The main performance measure of the system is the
percentage of persons correctly recognized. Performance
can be further qualified by the average value
of the following ratio
I
Feature Recognition
Voice 88 1.14
Dynamic 71 1.08
Face 91 1.56
Eyes
Nose 77 1.25
Mouth


I
The recognition performance and average
separation ratio R for each single feature and for
their integration. Data are based on 164 real
interactions and a database of 89 users.
The ratio R x measures the separation of the correct
match S 0
x from the wrong ones. This ratio
is invariant against the scale and location parameters
of the integrated score distribution and can
be used to compare different integration strategies
(weighted/unweighted geometric average, adap-
tive/fixed normalization). The weighted geometric
average of the scores adaptively normalized exhibits
the best performance and separation among
the various schemes on the available data.
Experiments have been carried out using data acquired
during 3 different test sessions. Of the 89
persons stored in the database, 87 have interacted
with the system in one or more sessions. One of the
three test sessions was used to adapt the acoustic
and visual databases (in the last case the images of
the session were simply added to those available);
therefore, session 1 was used to adapt session 2 and
session 2 to adapt session 3. As the number of interactions
for each adapted session is 82 the total
number of test interactions was 164. As each session
consisted of 82 interactions, the system was tested
on 164 interactions. The recognition performance
and the average value of R x for the different separate
features and for their integration are reported
in

Table

I.
4.2 Rejection
An important capability of a classifier is to reject
input patterns that cannot be classified in any of
the available classes with a sufficiently high degree
of confidence. For a person verification system, the
ability to reject an impostor is critical. The following
paragraphs introduce a rejection strategy that
takes into account the level of agreement of all the
different classifiers in the identification of the best
candidate.
A simple measure of confidence is given by the
integrated score itself: the higher the value, the
higher the confidence of the identification. Another
is given by the difference of the two best scores: it
is a measure of how sound the ranking of the best
candidate is. The use of independent features (or
feature sets) also provides valuable information in
the form of the rankings of the identification labels
across the classifier ouputs: if the pattern does not
belong to any of the known classes, its rank will
vary significantly from classifier to classifier. On
the contrary, if the pattern belongs to one of the
known classes, rank agreement will be consistently
high. The average rank and the rank dispersion
across the classifiers can then be used to quantify
the agreement of the classifiers in the final identi-
fication. The confidence in the final identification
can then be quantified through several measures.
The decision about whether the confidence is sufficient
to accept the system output can be based
on one or several of them. In the proposed system,
a linear classifier, based on absolute and relative
scores, ranks and their dispersion, will be used to
accept/reject the final result. The following issues
will be discussed:
1. degree of dependence of the features used;
2. choice of the confidence measures to be used
in the accept/reject rule;
3. training and test of the linear classifier used
to implement the accept/reject rule.
As a preliminary step, the independence of the
features used in the identification process will be
evaluated. It is known that the higher the degree
of independence, the higher the information provided
to the classifier. Let us consider a couple of
features X and Y . Let f(x
the corresponding normalized scores. They can be
considered as random samples from a population
with a bivariate distribution function. Let A i be
the rank of x i among x when they are arranged
in descending order, and B i the rank of y i
among defined similarly to A i . Spear-
man's rank correlation [19] is defined by:
A and -
B are the average values of fA i g and
respectively. An important characteristic of
rank correlation is its non-parametric nature. To
assess the independence of the features it is not
necessary to know the bivariate distribution from
which the (X are drawn, since the distribution
of their ranks is known, under the assumption of
independence. It turns out that
s


II
The rank correlation value of the couples of
features. The parenthesized values represent the
significance of the correlation. S1 and S2 represent
the dynamic and static acoustic features
nose and
mouth.
is distributed approximately as a Student's distribution
with I \Gamma 2 degrees of freedom [19]. It is then
possible to assess the dependence of the different
features used by computing the rank correlation of
each couple and by testing the corresponding signif-
icance. Results for the features used in the system
developed are given in Table II.
The acoustic features are clearly correlated, as
well as the nose and mouth features. The latter
correlation is due to the overlapping of the nose and
mouth regions, which was found to be necessary
in order to use facial regions characterized by the
same coordinates for the whole database. Acoustic
and visual features are independent, as could be
expected.
The feasibility of using a linear classifier was investigated
by looking at the distribution of acceptable
and non-acceptable 2 best candidates in a 3D
space whose coordinates are the integrated score,
a normalized ratio of the first to second best score
and the standard deviation of the rankings. As can
be seen in Figure 6 a linear classifier seems to be
appropriate.
The full vector d 2 R used as input to the linear
classifier is given by:
1. the integrated score, S 1 , of the best candidate;
2. the normalized ratio of the first to the second
best integrated score:
3. the minimum and maximum ranks of the first
and second final best candidates (4 entries);
4. the rank standard deviation of the first and
second final best candidates (2 entries);
5. the individual ranks of the first and second
final best candidates (10 entries).
To train the linear classifier the following procedure
was used. A set of positive examples fp i g is derived
Non-acceptable best candidates derive from two sources:
misclassified users from real interactions and best candidates
from virtual interactions, characterized by the removal of the
user entry from the data base.
Fig. 6. Let us represent the match with the database entries
by means of the integrated score, the standard deviation
of the rankings across the different features and the normalized
ratio of the first to second best integrated score.
The resulting three dimensional points are plotted and
marked with a 2 if they represent a correct match or
with a \Theta if the match is incorrect. Visual inspection
of the resulting point distribution shows that the two
classes of points can be separated well by using a plane.
from the data relative to the persons correctly classified
by the system. A set of negative examples
is given by the data relative to the best candidate
when the system did not classify the user
correctly. The set of negative examples can be augmented
by the data of the best candidate when the
correct entry is removed from the database, thereby
simulating the interaction with a stranger. The linear
discriminant function defined by the vector w
can be found by minimizing the following error:
)Awhere ff and fi represent the weight to be attributed
to false negatives and to false positives respectively
and is the dimensionality of the input vec-
tors. When represents the output
error of a linear perceptron with a symmetric sigmoidal
unit.
Final acceptance or rejection of an identification,
associated to a vector d, is done according to the
simple rule:
l
l
reject (24)
Fig. 7. System performance when false positives and false
negatives are weighted differently.
Note that the LHSs of eqns. (23)(24) represent
the signed distance, in arbitrary units, of point d
from the plane defined by w that divides the space
into two semispaces. Points lying in the correct
semispace contribute to E inversely to their distance
from plane w. Points lying near the plane
contribute with ff or fi while points lying in the
wrong semispace and at great distance from the discriminating
plane contribute with 2ff or 2fi. If the
two classes of points are linearly separable it is possible
to drive E to zero (see [14], [24]). A stochastic
minimization algorithm [4], [10] was used to minimize
When the system is required to work in a strict
mode (no errors allowed, that is, no strangers ac-
cepted), fi ?? ff should be considered in the training
phase. Note that a similar discriminant function
can be computed for each of the recognition
subsystems (i.e. face recognition and voice recog-
nition), thereby enabling the system to reject an
identification when it is not sufficiently certain even
when not all of the identification cues are available.
The training/test of the classifier followed a leave-
one-out strategy to maximize the number of data
available in the training phase [15]. The classifier is
trained by using all but one of the available samples
and tested on the excluded one. The performance
of the classifier can be evaluated by excluding in
turn each of the available samples and averaging
the classification error.
In the reported experiments, the available examples
were grouped per interacting user. The leave-
one-out method was then applied to the resulting
87 sets (the number of users that interacted with
the system) to guarantee the independence of the
training and test sets.
Each set was used in turn for testing, leaving the
remaining 86 for training. The results are reported
in

Table

III. A complete operating characteristic
curve for the integrated performance shown
in

Table

III is reported in Figure 7 where the
stranger-accepted and familiar-rejected rates at different
fi=ff ratios are plotted.
Face
Stranger accepted 4.0
Familiar rejected 8.0
Familiar misrecog. 0.5
Voice
Stranger accepted 14.0
Familiar rejected 27.0
Familiar misrecog. 1.0
Integrated
Stranger accepted 0.5
Familiar rejected 1.5
Familiar misrecog. 0.0


III
rates of the subsystems and of the complete
system when a rejection threshold is introduced.
Data are based on the subset of interactions for
which both face and speech data were available (155
out of 164).
Similar experiments were run on the acoustic and
visual features separately and are also reported in

Table

III. The results show that the use of the complete
set of features provides a relevant increase in
reliable performance over the separate subsystems.
4.3 Hybrid level integration
In this sub-section, a hybrid rank/measurement
level at which multiple classifiers can be combined
will be introduced. The approach is to reconstruct a
mapping from the sets of scores, and corresponding
ranks, into the set f0; 1g. The matching to each
of the database entries, as described by a vector
of five scores and the corresponding ranks should
be mapped to 1, if it corresponds to the correct
label, and to 0 otherwise. The reconstruction of
the mapping proceeds along the following steps:
1. find a set of positive and negative examples;
2. choose a parametric family of mappings;
3. choose the set of parameters for which the corresponding
mapping minimizes a suitable error
measure over the training examples.
Another way to look at the reconstruction of the
mapping is to consider the problem as a learning
task, where, given a set of acceptable and non acceptable
inputs, the system should be able to appropriately
classify unseen data.
Let fC j g be the set of classifiers. Each of them
associates to each person X some numerical data
that can be considered a vector. By comparison
with the i th database entry, a normalized similarity
can be computed. Each score
can be associated to its rank r ij in the list of scores
produced by classifier C j . The output of each classifier
can then be regarded as a list of couples
I represents the number
of people in the reference database. A mapping is
sought L 01 such that:
If, after mapping the list of scores, more than a
label qualifies, the system rejects the identification.
It is possible to relax the definition of L 01 by letting
the value of the mapping span the whole interval
[0; 1]. In this way the measurement level character
of the classification can be retained. The new
mapping L can be interpreted as a fuzzy predicate.
The following focuses on the fuzzy variant, from
which the original formulation can be obtained by
introducing a threshold !:
where '(\Delta) is the Heavyside unit-step function and
dimensional
vector containing the feature normalized matching
scores and corresponding ranks. The goal is to
approximate the characteristic function of the correct
matching vectors as a sum of Gaussian bumps.
Therefore the search for L is conducted within the
following family of functions:
ff
where
being a diagonal matrix with positive entries,
R. The approximating function
can be represented as an HyperBF network [25]
whose topology is reported in Figure 8.
The sigmoidal mapping is required to ensure that
the co-domain be restricted to the interval (0; 1).
The location t ff , shape \Sigma and height c ff of each
bump are chosen by minimizing the following error
measure:
ff
is a set of examples (points at
which the value of the mapping to be recovered is
known). The first subscript i denotes the database
c
x xS
Fig. 8. The function used to approximate the mapping
from the score/rank domain into the interval (0; 1) can
be represented as an HyperBF network.
entry from which x ij is derived and the second subscript
j represents the example.
The required value of the mapping at x ij is 1
when i is the correct label (class) for the j-th example
and 0 otherwise. The error measure E is
minimized over the parameter space f(c
by means of a stochastic algorithm with adaptive
memory [10]. The number of free parameters involved
in the minimization process dictates the use
of a large set of examples. As a limited number
of real interactions was available, a leave-one-
out strategy was used for training and testing the
system as for the linear classifier previously de-
scribed. From each of the available user-system
interactions, a virtual interaction was derived by
removing from the database the entry of the interacting
user, thereby simulating an interaction with
a stranger. For each interaction j
1. the vector corresponding to the correct
database entry provides a positive example;
2. the vectors of the first ten, non correct, entries
of the real interaction (as derived from sorting
the integrated scores of Section 4.1) and the
vectors of the first ten entries of the virtual
interaction provide the negative examples.
The reason for using only the first ten non correct
entries is that the matching scores decay quickly
with rank position in the final score list and additional
examples would not provide more informa-
tion. Data from different interactions of the same
user were then grouped. The resulting set of examples
was used to generate an equal number of
different training/testing set pairs. Each set was
used in turn for testing, leaving the remaining ones
for training. The problem of matching the number
of free parameters in the approximation function to
the complexity of the problem was solved by testing
the performance of networks with increasing size.
For each network size, a value for threshold ! of
eqn. (26) was computed to minimize the total error
defined as the sum of the percentage of accepted
Fig. 9. The total error achieved by networks with different
number of units. The total error is computed by
summing the percentage of accepted strangers, misrecognized
and rejected database people. For each net size
a threshold was chosen to minimize the cumulative error.
accepted
Familiar rejected (%) 3.0 3.0 3.5
Familiar misrecog. (%


IV
The performance of the system when using an
HyperBF network with 21 units to perform score
integration.
strangers, misrecognized and rejected database per-
sons. In Figure 9, the total error is reported as a
function of the network size. Note that the threshold
is computed on the test set, so that it gives an
optimistic estimate.
To obtain a correct estimate of system perfor-
mance, a cross-validation approach was used for the
net giving the best (optimistic) total error estimate.
be the interval over which the total error
assumes its minimumvalue (see Figure 10). The
threshold value can be chosen as:
favouring acceptance over rejection;
favouring rejection over acceptance.
The resulting performance is reported in Table IV.
Note that using ! 1 the system was able to reject
all of the strangers, which is the ultimate requirement
for a reliable system, missing only 3.5% of the
known users.
5. Conclusions
A system that combines acoustic and visual
cues in order to identify a person has been de-
scribed. The speaker recognition sub-system is
based on Vector Quantization of the acoustic parameter
space and includes an adaptation phase of
the codebooks to the test environment. A different
method to perform speaker recognition, which
makes use of the Hidden Markov Model technique
and pitch information, is under investigation.
Fig. 10. Error percentages as a function of the rejection
threshold for a Gaussian based expansion.
A face recognition sub-system also was described.
It is based on the comparison of facial features at
the pixel level using a similarity measure based on
the L 1 norm.
The two sub-systems provide a multiple classifier
system. In the implementation described, 5 classifiers
acoustic and 3 visual) were considered. The
multiple classifier operates in two steps. In the first
one, the input scores are normalized using robust
estimators of location and scale. In the second step,
the scores are combined using a weighted geometric
average. The weights are adaptive and depend on
the score distributions. While normalization is fundamental
to compensate for input variations (e.g.
variations of illumination, background noise con-
ditions, utterance length and of speaker voices),
weighting emphasizes the classification power of the
most reliable classifiers. The use of multiple cues,
acoustic and visual, proved to be effective in improving
performance. The correct identification
rate of the integrated system is 98% which represents
a significant improvement with respect to the
88% and 91% rates provided by the speaker and
face recognition systems respectively. Future use
of the Hidden Markov Model technique is expected
to improve performance of the VQ based speaker
recognizer.
An important capability of the multiple classifier
itself is the rejection of the input data when they
can not be matched with sufficient confidence to
any of the database entries.
An accept/reject rule is introduced by means of a
linear classifier based on measurement and rank information
derived from the five recognition systems.
A novel, alternative, approach to the integration of
multiple classifiers at the hybrid rank/measurement
level is also presented. The problem of combining
the outputs of a set of classifiers is considered as
a learning task. A mapping from the scores of the
classifiers and their ranks into the interval (0; 1) is
approximated using an HyperBF network. A final
rejection/acceptance threshold is then introduced
using the cross-validation technique. System performance
is evaluated on data acquired during real
interactions of the users in the reference database.
Performance of the two techniques is similar.
The current implementation of the system is
working on an HP 735 workstation with a Matrox
Magic frame grabber. In order to optimize system
throughput, it relies on a hierarchical match with
the face database.
The incoming picture, represented by a set of features
is compared at low resolution with the complete
database. For each person in the database,
the most similar feature, among the set of available
images, is chosen and the location of the best
matching position stored. The search is then continued
at the upper resolution level by limiting the
search to the most promising candidates at the previous
level.
These candidates are selected by integrating their
face scores according to the procedure described in
section 4.1. All available data must be used to secure
a reliable normalization of the scores. How-
ever, new scores at higher resolution are computed
only for a selected subset of persons and this constitutes
a problem for the integration procedure. In
fact, scores from image comparisons at different levels
would me mixed, similarity values deriving from
lower resolutions being usually higher. To overcome
this difficulty, the scores from the previous level are
reduce (scaled) by the highest reduction factor obtained
comparing the newly computed scores to the
corresponding previous ones.
The performance, measured on the datasets used
for the reported experiments, does not decrease and
the overall identification time (face and voice pro-
cessing) is approximately 5 seconds.
The same approach, using codebooks of reduced
size could be applied to the speaker identification
system, thereby increasing system throughput.
Adding a subject to the database is a simple task
for both subsystems. This is due to the modularity
of the databases, each subject being described
independently of the others. The integration strategy
itself does not require any update. The rejection
and the combined identification/rejection procedures
do require updating. However, the training
of the linear perceptron and of the HyperBF
network can be configured more as a refinement of
a suboptimal solution (available from the previous
database) than as the computation of a completely
unknown set of optimal parameters. While the sys-
tem, as presented, is mainly an identification sys-
tem, a small modification transforms it into a verification
system. For each person in the database
it is possible to select a subset containing the most
similar people (as determined by the identification
system). When the user must be verified the identification
system can be used using the appropriate
subset, thereby limiting the computational effort,
and verifying the identity of the user using the techniques
reported in the paper.
Future work will have the purpose of further improving
the global efficiency of the system with the
investigation of more accurate and reliable rejection
methods.

Acknowledgement

The authors would like to thank Dr. L. Stringa,
Prof. T. Poggio and Prof. R. de Mori for valuable
suggestions and discussions. The authors are grateful
to the referees for many valuable comments.



--R

Computer Vision.
Selecting uncertainty calculi and granularity: an experiment in trading-off precision and complexity
Rum: A layered architecture for reasoning with uncertainty.
On Training Neural Nets through Stochastic Minimization.
Estimation of Pose and Illuminant Direction for Face Processing.
A recognition system
Robust Estimation of Correlation: an Application to Computer Vision.
Face Recognition: Features versus Templates.
Automatic Person Recognition by Using Acoustic and Geometric Features.
Stochastic minimization with adaptive memory.
Smart sensing within a pyramid vision ma- chine
A start-end point detection algorithm for a real-time acoustic front-end based on dsp32c vme board
Speaker Recognition
Pattern Recognition and Scene Analysis.
Introduction to Statistical Pattern Recog- nition
Cepstrum Analysis Technique for Automatic Speaker Verification.
Recognizing Human Eyes.
Robust statistics: the approach based on influence functions.
Introduction to mathematical statistics.
Robust Statistics.

Similarity normalization method for speaker verification based on a posteriori probability.
Speech Communication.
Adaptive Pattern Recognition and Neural Networks.
Regularization algorithms for learning that are equivalent to multilayer networks.
A Project for an Intelligent System: Vision and Learning.
The use of cohort normalized scores for speaker verification.
Evaluation of a Vector Quantization Talker Recognition System in Text Independent and Text Dependent Modes.

On the Use of Istantaneous and Transitional Spectral Information in Speaker Recognition.
Automatic Face Recognition using Directional Derivatives.
Eyes Detection for Face Recognition.
Methods of Combining Multiple Classifiers and Their Applications to Handwriting Recognition.
--TR

--CTR
Jian-Gang Wang , Hui Kong , Eric Sung , Wei-Yun Yau , Eam Khwang Teoh, Fusion of appearance image and passive stereo depth map for face recognition based on the bilateral 2DLDA, Journal on Image and Video Processing, v.2007 n.2, p.6-6, August 2007
Anil K. Jain , Arun Ross, Multibiometric systems, Communications of the ACM, v.47 n.1, January 2004
Arun Ross , Anil Jain, Information fusion in biometrics, Pattern Recognition Letters, v.24 n.13, p.2115-2125, September
Jian-Gang Wang , Eng Thiam Lim , Xiang Chen , Ronda Venkateswarlu, Real-time Stereo Face Recognition by Fusing Appearance and Depth Fisherfaces, Journal of VLSI Signal Processing Systems, v.49 n.3, p.409-423, December  2007
Niall A. Fox , Ralph Gross , Philip de Chazal , Jeffery F. Cohn , Richard B. Reilly, Person identification using automatic integration of speech, lip, and face experts, Proceedings of the ACM SIGMM workshop on Biometrics methods and applications, November 08, 2003, Berkley, California
Julian Fierrez-Aguilar , Daniel Garcia-Romero , Javier Ortega-Garcia , Joaquin Gonzalez-Rodriguez, Adapted user-dependent multimodal biometric authentication exploiting general information, Pattern Recognition Letters, v.26 n.16, p.2628-2639, December 2005
R. Brunelli, verification through finger matching: A comparison of Support Vector Machines and Gaussian Basis Functions classifiers, Pattern Recognition Letters, v.27 n.16, p.1905-1915, December, 2006
Maycel-Isaac Faraj , Josef Bigun, Audio-visual person authentication using lip-motion from orientation maps, Pattern Recognition Letters, v.28 n.11, p.1368-1382, August, 2007
H. Vajaria , T. Islam , P. Mohanty , S. Sarkar , R. Sankar , R. Kasturi, Evaluation and analysis of a face and voice outdoor multi-biometric system, Pattern Recognition Letters, v.28 n.12, p.1572-1580, September, 2007
Robert Snelick , Umut Uludag , Alan Mink , Michael Indovina , Anil Jain, Large-Scale Evaluation of Multimodal Biometric Authentication Using State-of-the-Art Systems, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.27 n.3, p.450-455, March 2005
R. Brunelli, verification through finger matching: a comparison of support vector machines and Gaussian basis functions classifiers, Pattern Recognition Letters, v.27 n.16, p.1905-1915, December 2006
Doroteo T. Toledano , Rubn Fernndez Pozo , lvaro Hernndez Trapote , Luis Hernndez Gmez, Usability evaluation of multi-modal biometric verification systems, Interacting with Computers, v.18 n.5, p.1101-1122, September, 2006
Hakan Altinay , Mbeccel Demirekler, Undesirable effects of output normalization in multiple classifier systems, Pattern Recognition Letters, v.24 n.9-10, p.1163-1170, 01 June
Aleix M. Martnez, Recognizing Imprecisely Localized, Partially Occluded, and Expression Variant Faces from a Single Sample per Class, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.6, p.748-763, June 2002
Rodrigo de Luis-Garca , Carlos Alberola-Lpez , Otman Aghzout , Juan Ruiz-Alzola, Biometric identification systems, Signal Processing, v.83 n.12, p.2539-2557, December
Seong G. Kong , Jingu Heo , Faysal Boughorbel , Yue Zheng , Besma R. Abidi , Andreas Koschan , Mingzhong Yi , Mongi A. Abidi, Multiscale Fusion of Visible and Thermal IR Images for Illumination-Invariant Face Recognition, International Journal of Computer Vision, v.71 n.2, p.215-233, February  2007
K. Srinivasa Rao , A. N. Rajagopalan, A probabilistic fusion methodology for face recognition, EURASIP Journal on Applied Signal Processing, v.2005 n.1, p.2772-2787, 1 January 2005
Harini Veeraraghavan , Paul Schrater , Nikos Papanikolopoulos, Robust target detection and tracking through integration of motion, color, and geometry, Computer Vision and Image Understanding, v.103 n.2, p.121-138, August 2006
Sharon Oviatt, Advances in Robust Multimodal Interface Design, IEEE Computer Graphics and Applications, v.23 n.5, p.62-68, September
H. E. etingl , E. Erzin , Y. Yemez , A. M. Tekalp, Multimodal speaker/speech recognition using lip motion, lip texture and audio, Signal Processing, v.86 n.12, p.3549-3558, December 2006
Slobodan Ribaric , Ivan Fratric, A Biometric Identification System Based on Eigenpalm and Eigenfinger Features, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.27 n.11, p.1698-1709, November 2005
classification;learning;face recognition;correlation;template matching;robust statistics;speaker recognition