Recognizing Action Units for Facial Expression Analysis.
AbstractMost automatic expression analysis systems attempt to recognize a small set of prototypic expressions, such as happiness, anger, surprise, and fear. Such prototypic expressions, however, occur rather infrequently. Human emotions and intentions are more often communicated by changes in one or a few discrete facial features. In this paper, we develop an Automatic Face Analysis (AFA) system to analyze facial expressions based on both permanent facial features (brows, eyes, mouth) and transient facial features (deepening of facial furrows) in a nearly frontal-view face image sequence. The AFA system recognizes fine-grained changes in facial expression into action units (AUs) of the Facial Action Coding System (FACS), instead of a few prototypic expressions. Multistate face and facial component models are proposed for tracking and modeling the various facial features, including lips, eyes, brows, cheeks, and furrows. During tracking, detailed parametric descriptions of the facial features are extracted. With these parameters as the inputs, a group of action units (neutral expression, six upper face AUs and 10 lower face AUs) are recognized whether they occur alone or in combinations. The system has achieved average recognition rates of 96.4 percent (95.4 percent if neutral expressions are excluded) for upper face AUs and 96.7 percent (95.6 percent with neutral expressions excluded) for lower face AUs. The generalizability of the system has been tested by using independent image databases collected and FACS-coded for ground-truth by different research teams.
Introduction
Recently facial expression analysis has attracted attention in the computer vision literature [3, 5, 6, 9,
11, 13, 17, 19]. Most automatic expression analysis systems attempt to recognize a small set of prototypic
expressions (i.e. joy, surprise, anger, sadness, fear, and disgust) [11, 17]. In everyday life, however,
such prototypic expressions occur relatively infrequently. Instead, emotion is communicated by changes
in one or two discrete facial features, such as tightening the lips in anger or obliquely lowering the lip
corners in sadness [2]. Change in isolated features, especially in the area of the brows or eyelids, is
typical of paralinguistic displays; for instance, raising the brows signals greeting. To capture the subtlety
of human emotion and paralinguistic communication, automated recognition of fine-grained changes in
facial expression is needed.
Ekman and Friesen [4] developed the Facial Action Coding System (FACS) for describing facial
expressions. The FACS is a human-observer-based system designed to describe subtle changes in
facial features. FACS consists of 44 action units, including those for head and eye positions. AUs
are anatomically related to contraction of specific facial muscles. They can occur either singly or in
combinations. AU combinations may be additive, in which case combination does not change the
appearance of the constituents, or nonadditive, in which case the appearance of the constituents changes
(analogous to co-articulation effects in speech). For action units that vary in intensity, a 5-point ordinal
scale is used to measure the degree of muscle contraction. Although the number of atomic action units
is small, more than 7,000 combinations of action units have been observed [12]. FACS provides the
necessary detail with which to describe facial expression.
Automatic recognition of action units is a difficult problem. AUs have no quantitative definitions and
as noted can appear in complex combinations. Several researchers have tried to recognize AUs [1, 3, 9].
The system of Lien et al. [9] used dense-flow, feature point tracking and edge extraction to recognize 6
upper face AUs or AU combinations (AU1+2, AU1+4, AU4, AU5, AU6, and AU7) and 9 lower face AUs
and AU combinations (AU12, AU25, AU26, AU27, AU12+25, AU20+25, AU15+17, AU17+23+24,
AU9+17). Bartlett et al. [1] recognized 6 individual upper face AUs (AU1, AU2, AU4, AU5, AU6, and
but none occurred in combinations. The performance of their feature-based classifier on novel
was 57%; on new images of faces used for training, the rate was 85.3%. By combining holistic
spatial analysis and optical flow with local features in a hybrid system, Bartlett et al. increased accuracy
to 90.9% correct. Donato et al. [3] compared several techniques for recognizing action units including
optical flow, principal component analysis, independent component analysis, local feature analysis, and
Gabor wavelet representation. Best performances were obtained by Gabor wavelet representation and
independent component analysis which achieved a 95% average recognition rate for 6 upper face AUs
and 6 lower face AUs.
In this report, we developed a feature-based AU recognition system. This system explicitly analyzes
appearance changes in localized facial features. Since each AU is associated with a specific set of facial
muscles, we believe that accurate geometrical modeling of facial features will lead to better recognition
results. Furthermore, the knowledge of exact facial feature positions could benefit the area-based [17],
holistic analysis [1], or optical flow based [9] classifiers. Figure 1 depicts the overview of the analysis
system. First, the head orientation and face position are detected. Then, subtle changes in the facial
components are measured. Motivated by FACS action units, these changes are represented as a collection
of mid-level feature parameters. Finally, action units are classified by feeding these parameters to a neural
network.
Because the appearance of facial features is dependent upon head orientation, we develop a multi-state
model-based system for tracking facial features. Different head orientations and corresponding variation
in the appearance of face components are defined as separate states. For each state, a corresponding
description and one or more feature extraction methods are developed.
We separately represent all the facial features into two parameter groups for upper face and lower face
because facial actions in the upper and lower face are relatively independent [4]. Fifteen parameters
are used to describe eye shape, motion, and state, and brow and cheek motion, and upper face furrows
for upper face. Nine parameters are used to describe the lip shape, lip motion, lip state, and lower face
furrows for lower face.
After the facial features are correctly extracted and suitably represented, we employ a neural network
to recognize the upper face AUs (Neutral, AU1, AU2, AU4, AU5, AU6, and AU7) and lower face AUs
(Neutral, AU9, AU 10, AU 12, AU 15, AU 17, AU 20, AU 25, AU 26, AU 27, and AU23+24) respectively.
Seven basic upper face AUs and eleven basic lower face AUs are identified regardless of whether they
occurred singly or in combinations. For the upper face AU recognition, compared to Bartlett's [1] results
by using the same database, our system achieves recognition accuracy with an average recognition rate
of 95% with fewer parameters and in the more difficult case in which AUs may occur either individually
or in additive and nonadditive combinations. For the lower face AU recognition, a previous attempt for a
similar task [9] recognized 6 lower face AUs and combinations(AU 12, AU12+25, AU20+25, AU9+17,
AU17+23+24, and AU15+17) with 88% average recognition rate by separate hidden Markov Models
for each action unit or action unit combination. Compared to the previous results, our system achieves
recognition accuracy with an average recognition rate of 96.71%. Difficult cases in which AUs occur
either individually or in additive and nonadditive combinations are handled also.

Figure

1. Feature based action unit recognition system.
2. Multi-State Models for Face and Facial Components
2.1. Multi-state face model
Head orientation is a significant factor that affects the appearance of a face. Based on the head orien-
tation, seven head states are defined in Figure 2. To develop more robust facial expression recognition
system, head state will be considered. For the different head states, facial components, such as lips,
appear very differently, requiring specific facial component models. For example, the facial component
models for a front face include F rontLips, F rontEyes (left and right), F rontCheeks(left and
right), NasolabialFurrows, and Nosewrinkles. The right face includes only the component models
SideLips, Righteye, Rightbrow, and Rightcheek. In our current system, we assume the face images
are nearly front view with possible in-plane head rotations.
2.2. Multi-state face component models
Different face component models must be used for different states. For example, a lip model of the
front face doesn't work for a profile face. Here, we give the detailed facial component models for the
nearly front-view face. Both the permanent components such as lips, eyes, brows, cheeks and the transient
components such as furrows are considered. Based on the different appearances of different components,
different geometric models are used to model the component's location, shape, and appearance. Each
(a) Head state.
(b) Different facial components used for each head state.

Figure

2. Multiple state face model. (a) The head state can be left, left-front, front, right-front, right,
down, and up. (b) Different facial component models are used for different head states.
component employs a multi-state model corresponding to different component states. For example, a
three-state lip model is defined to describe the lip states: open, closed, and tightly closed. A two-state
eye model is used to model open and closed eye. There is one state for brow and cheek. Present and
absent are use to model states of the transient facial features. The multi-state component models for
different components are described in Table 1.

Table

1. Multi-state facial component models of a front face
Component State Description/Feature
Opened
Closed
(xc, yc)
Tightly closed Lip corner1 Lip corner2
Eye Open
(xc, yc) h2
Closed corner2
corner1
Brow Present
Cheek Present
Furrow Present
Eye's inner corner line
furrows
nasolabial
Absent
3. Facial Feature Extraction
Contraction of the facial muscles produces changes in both the direction and magnitude of the motion
on the skin surface and in the appearance of permanent and transient facial features. Examples of
permanent features are the lips, eyes, and any furrows that have become permanent with age. Transient
features include any facial lines and furrows that are not present at rest. We assume that the first frame is
in a neutral expression. After initializing the templates of the permanent features in the first frame, both
permanent and transient features can be tracked and detected in the whole image sequence regardless of
the states of facial components. The tracking results show that our method is robust for tracking facial
features even when there is large out of plane head rotation.
3.1. Permanent features
features: A three-state lip model is used for tracking and modeling lip features. As shown in

Table

1, we classify the mouth states into open, closed, and tightly closed. Different lip templates are
used to obtain the lip contours. Currently, we use the same template for open and closed mouth. Two
parabolic arcs are used to model the position, orientation, and shape of the lips. The template of open
and closed lips has six parameters: lip center (xc, yc), lip shape (h1, h2 and w), and lip orientation (').
For a tightly closed mouth, the dark mouth line connecting lip corners is detected from the image to
model the position, orientation, and shape of the tightly closed lips.
After the lip template is manually located for the neutral expression in the first frame, the lip color is
obtained by modeling as a Gaussian mixture. The shape and location of the lip template for the image
sequence is automatically tracked by feature point tracking. Then, the lip shape and color information
are used to determine the lip state and state transitions. The detailed lip tracking method can be found in
paper [15].
Eye features: Most eye trackers developed so far are for open eyes and simply track the eye locations.
However, for recognizing facial action units, we need to recognize the state of eyes, whether they are
open or closed, and the parameters of an eye model, the location and radius of the iris, and the corners
and height of the open eye. As shown in Table 1, the eye model consists of "open" and "closed".
The iris provides important information about the eye state. If the eye is open, part of the iris normally
will be visible. Otherwise, the eye is closed. For the different states, specific eye templates and different
algorithms are used to obtain eye features.
For an open eye, we assume the outer contour of the eye is symmetrical about the perpendicular
bisector to the line connecting two eye corners. The template, illustrated in Table 1, is composed of a
circle with three parameters and two parabolic arcs with six parameters
This is the same eye template as Yuille's except for two points located at the center of the whites [18].
For a closed eye, the template is reduced to 4 parameters for each of the eye corners.
The default eye state is open. Locating the open eye template in the first frame, the eye's inner corner
is tracked accurately by feature point tracking. We found that the outer corners are hard to track and
less stable than the inner corners, so we assume the outer corners are on the line that connects the inner
corners. Then, the outer corners can be obtained by the eye width, which is calculated from the first
frame.
Intensity and edge information are used to detect an iris because the iris provides important information
about the eye state. A half-circle iris mask is used to obtain correct iris edges. If the iris is detected, the
eye is open and the iris center is the iris mask center In an image sequence, the eyelid contours
are tracked for open eyes by feature point tracking. For a closed eye, we do not need to track the eyelid
contours. A line connects the inner and outer corners of the eye is used as the eye boundary. The detailed
eye feature tracking techniques can be found in paper [14].
Brow and cheek features: Features in the brow and cheek areas are also important to facial expression
analysis. For the brow and cheek, one state is used respectively, a triangular template with six parameters
is used to model the position of brow or cheek. Both brow and cheek
are tracked by feature point tracking. A modified version of the gradient tracking algorithm [10] is
used to track these points for the whole image sequence. Some permanent facial feature tracking results
for different expressions are shown in Figure 3. More facial feature tracking results can be found in
http://www.cs.cmu.edu/-face.
3.2. Transient features
Facial motion produces transient features. Wrinkles and furrows appear perpendicular to the motion
direction of the activated muscle. These transient features provide crucial information for the recognition
of action units. Contraction of the corrugator muscle, for instance, produces vertical furrows between
the brows, which is coded in FACS as AU 4, while contraction of the medial portion of the frontalis
muscle (AU 1) causes horizontal wrinkling in the center of the forehead.
Some of these lines and furrows may become permanent with age. Permanent crows-feet wrinkles
around the outside corners of the eyes, which is characteristic of AU 6 when transient, are common in
adults but not in infants. When lines and furrows become permanent facial features, contraction of the
corresponding muscles produces changes in their appearance, such as deepening or lengthening. The
presence or absence of the furrows in a face image can be determined by geometric feature analysis [9, 8],
or by eigen-analysis [7, 16]. Kwon and Lobo [8] detect furrows by snake to classify pictures of people
into different age groups. Lien [9] detected whole face horizontal, vertical and diagonal edges for face
expression recognition.
In our system, we currently detect nasolabial furrows, nose wrinkles, and crows feet wrinkles. We
define them in two states: present and absent. Compared to the neutral frame, the wrinkle state is present
if the wrinkles appear, deepen, or lengthen. Otherwise, it is absent. After obtaining the permanent facial
features, the areas with furrows related to different AUs can be decided by the permanent facial feature
locations. We define the nasolabial furrow area as the area between eye's inner corners line and lip
corners line. The nose wrinkle area is a square between two eye inner corners. The crows feet wrinkle
areas are beside the eye outer corners.
We use canny edge detector to detect the edge information in these areas. For nose wrinkles and crows
feet wrinkles, we compare the edge pixel numbers E of current frame with the edge pixel numbers E 0
of the first frame in the wrinkle areas. If E=E 0 large than the threshold T , the furrows are present.
Otherwise, the furrows are absent. For the nasolabial furrows, we detect the continued diagonal edges.
The nasolabial furrow detection results are shown in Fig. 4.
4. Facial Feature Representation
Each action unit of FACS is anatomically related to contraction of a specific facial muscle. For
instance, AU 12 (oblique raising of the lip corners) results from contraction of the zygomaticus major
muscle, AU 20 (lip stretch) from contraction of the risorius muscle, and AU 15 (oblique lowering of the
lip corners) from contraction of the depressor anguli muscle. Such muscle contractions produce motion
in the overlying skin and deform shape or location of the facial components. In order to recognize the
(a) (b) (c) (d)

Figure

3. Permanent feature tracking results for different expressions. (a) Narrowing eyes and opened
smiled mouth. (b) Large open eye, blinking and large opened mouth. (c) Tight closed eye and eye
blinking. (4) Tightly closed mouth and blinking.

Figure

4. Nasolabial furrow detection results. For the same subject, the nasolabial furrow angle(between
the nasolabial furrow and the line connected eye inner corners) is different for different expressions.
subtle changes of face expression, we represent the upper face features and lower face features into a
group of suitable parameters respectively because facial actions in the upper face have little influence on
facial motion in lower face, and vice versa [4].
For defining these parameters, we first define the basic coordinate system. Because the eye's inner
corners are the most stable features in the face and are relatively insensitive to deformation by facial
expressions, we define the x-axis as the line connecting two inner corners of eyes and the y-axis as
perpendicular to x-axis. In order to remove the effects of the different size of face images in different
image sequences, all the parameters except those about wrinkles' states are calculated in ratio scores by
comparison to the neutral frame.
4.1. Upper Face Feature Representation
We represent the upper face features as 15 parameters. Of these, 12 parameters describe the motion
and shape of eyes, brows, and cheeks. 2 parameters describe the state of crows feet wrinkles, and 1
parameter describes the distance between brows. Figure 5 shows the coordinate system and the parameter
meanings. The definitions of upper face parameters are listed in Table 2.

Table

2. Upper face feature representation for AU recognition
Permanent features (Left and right)
Inner brow Outer brow Eye height
motion (r binner ) motion (r bouter ) (r eheight )
r binner r bouter r eheight
If r binner >0, If r bouter >0, If r eheight >0,
Inner brow Outer brow Eye height
move up. move up. increases.
Eye top lid Eye bottom lid Cheek motion
motion (r top ) motion (r btm ) (r cheek )
r top r btm r cheek
. =\Gamma h2\Gammah2 0
. =\Gamma c\Gammac 0
If r top ? 0, If r btm ? 0, If r cheek ? 0,
Eye top lid Eye bottom lid Cheek
move up. move up. move up.
Other features
Distance Left crows Right crows
of brows feet wrinkles feet wrinkles
(D brow
D brow If W lef
. Left crows feet Right crows feet
wrinkle present. wrinkle present.

Figure

5. Upper face features. hl(hl1 are the height of left eye and right
eye; D is the distance between brows; cl and cr are the motion of left cheek and right cheek. bli and
bri are the motion of the inner part of left brow and right brow. blo and bro are the motion of the
outer part of left brow and right brow. f l and fr are the left and right crows feet wrinkle areas.
4.2. Lower Face Feature Representation
We define nine parameters to represent the lower face features from the tracked facial features. Of
these, 6 parameters describe the permanent features of lip shape, lip state and lip motion, and 3 parameters
describe the transient features of the nasolabial furrows and nose wrinkles.
We notice that if the nasolabial furrow is present, there are different angles between the nasolabial
furrow and x-axis for different action units. For example, the nasolanial furrow angle of AU9 or AU10
is larger than that of AU12. So we use the angle to represent its orientation if it is present. Although
the nose wrinkles are located in the upper face, but we classify the parameter of them in the lower face
feature because it is related to the lower face AUs.
The definitions of lower face parameters are listed in Table 3. These feature data are affine aligned
by calculating them based on the line connected two inner corners of eyes and normalized for individual
differences in facial conformation by converting to ratio scores. The parameter meanings are shown in

Figure

6.

Figure

6. Lower face features. h1 and h2 are the top and bottom lip heights; w is the lip width; D lef t
is the distance between the left lip corner and eye inner corners line; D right is the distance between the
right lip corner and eye inner corners line; n1 is the nose wrinkle area.
5. Facial Action Unit Definitions
Ekman and Friesen [4] developed the Facial Action Coding System (FACS) for describing facial
expressions by action units (AUs) or AU combinations. are anatomically related to

Table

3. Representation of lower face features for AUs recognition
Permanent features
Lip height Lip width Left lip corner
r height r width r lef t
. =\Gamma D left \GammaD left0
If r height >0, If r width >0, If r lef t >0,
lip height lip width left lip corner
increases. increases. move up.
Right lip corner Top lip motion Bottom lip
right
r right r top r btm
right \GammaD right0
D right0
. =\Gamma Dtop \GammaD top0
. =\Gamma D btm \GammaD btm0
If r right >0, If r top >0, If r btm >0,
right lip corner top lip bottom lip
move up. move up. move up.
Transient features
Left nasolibial Right nasolibial State of nose
furrow angle furrow angle wrinkles
(Ang
Left nasolibial Left nasolibial If S nosew
furrow present furrow present nose wrinkles
with angle Ang lef t . with angle present.
Ang right .
contraction of a specific set of facial muscles. Of thses, 12 are for upper face, and are for lower
Action units can occur either singly or in combinations. The action unit combinations may be
additive such as AU1+5, in which case combination does not change the appearance of the constituents,
or nonadditive, in which case the appearance of the constituents does change such as AU1+4. Although
the number of atomic action units is small, more than 7,000 combinations of action units have been
observed [12]. FACS provides the necessary detail with which to describe facial expression.

Table

4. Basic upper face action units or AU combinations
Inner portion of Outer portion of Brows lowered
the brows is the brows is and drawn
raised. raised. together
Upper eyelids Cheeks are Lower eyelids
are raised. raised. are raised.
AU 1+4 AU 4+5 AU 1+2
Medial portion Brows lowered Inner and outer
of the brows is and drawn portions of the
raised and pulled together and brows are raised.
together. upper eyelids
are raised.
AU 1+2+4 AU1+2+5+6+7 AU0(neutral)
Brows are pulled Brow, eyelids, and Eyes, brow, and
together and cheek are raised. cheek are
upward. relaxed.
5.1. Upper Face Action Units

Table

4 shows the definitions of 7 individual upper face AUs and 5 non-additive combinations involving
these action units. As an example of a non-additive effect, AU4 appears differently depending on whether
it occurs alone or in combination with AU1, as in AU1+4. When AU1 occurs alone, the brows are drawn
together and lowered. In AU1+4, the brows are drawn together but are raised by the action of AU 1. As
another example, it is difficult to notice any difference between the static images of AU2 and AU1+2
because the action of AU2 pulls the inner brow up, which results in a very similar appearance to AU1+2.
In contrast, the action of AU1 alone has little effect on the outer brow.
5.2. Lower face action units

Table

5 shows the definitions of 11 lower face AUs or AU combinations.
6. Image Database
6.1. Image Database for Upper Face AU Recognition
We use the database of Bartlett et al. [1] for upper face AUs recognition. This image database was
obtained from 24 Caucasian subjects, consisting of 12 males and 12 females. Each image sequence
consists of 6-8 frames, beginning with a neutral or with very low magnitude facial actions and ending
with a high magnitude facial actions. For each sequence, action units were coded by a certified FACS
coder.
For this investigation, 236 image sequences from 24 subjects were processed. Of these, 99 image
sequences contain only individual upper face AUs, and 137 image sequences contain upper-face AU
combinations. Training and testing are performed on the initial and final two frames in each image
sequence. For some of the image sequences, lighting normalizations were performed.
To test our algorithm on the individual AUs, we randomly generate training and testing sets from the
image sequences, as shown in Table 6. In T rainS3 and TestS3, we ensure that the subjects do not
appear in both training and testing sets.
To test our algorithm on the both individuall AUs and AU combinations, we generate a training set
(T rainC1) and a testing set (T estC1) as shown in Table 6.

Table

5. Basic lower face action units or AU combination
The infraorbital The infraorbital The lips and the
triangle and triangle is lower portion of
center of the pushed upwards. the nasolabial
upper lip are Upper lip is furrow are pulled
pulled upwards. raised. Nose pulled back
Nose wrinkling wrinkle is absent. laterally. The
is present. mouth is
elongated.
The corner of The chin boss Lip corners are
the lips are is pushed pulled obliquely.
pulled down. upwards.
AU 25 AU 26 AU27
Lips are relaxed Lips are relaxed Mouth stretched,
and parted. and parted; open and the
mandible is mandible pulled
lowered. downwards.
AU 23+24 neutral
Lips tightened, Lips relaxed
narrowed, and and closed.
pressed together.

Table

6. Data distribution of each data set for upper face AU recognition.
Single AU Data Sets
AUs AU0 AU1 AU2 AU4 AU5 AU6 AU7 Total
TrainS1
TrainS2 76 20 28 228
TrainS3 52
AU Combination Data Sets
6.2. Image Database for Lower Face AU Recognition
We use the data of Pitt-CMU AU-Coded Face Expression Image Database for lower face AU recogni-
tion. The database currently includes 1917 image sequences from 182 adult subjects of varying ethnicity,
performing multiple tokens of 29 of primary FACS action units. Subjects sat directly in front of the
camera and performed a series of facial expressions that included single action units (e.g., AU 12, or
smile) and combinations of action units (e.g., AU 6+12+25). Each expression sequence began from a
neutral face. For each sequence, action units were coded by a certified FACS coder.
Total 463 image sequences from 122 adults (65% female, 35% male, 85% European-American, 15%
African-American or Asian, ages to 35 years) are processed for lower face action unit recognition.
Some of the image sequences are with more action unit combinations such as AU9+17, AU10+17,
AU12+25, AU15+17+23, AU9+17+23+24, and AU17+20+26. For each image sequence, we use the
neutral frame and two peak frames. 400 image sequences are used as training data and 63 different image
sequences are used as test data. The training and testing data sets are shown in Table 7.

Table

7. Training data set for lower face AU recognition
neutral AU9 AU10 AU12 AU15 AU17 AU20 AU25 AU26 AU27 AU23+24 Total
Train Set 400 38
Test
7. Face Action Units Recognition
7.1. Upper Face Action Units Recognition
We used three-layer neural networks with one hidden layer. The inputs of the neural networks are the
parameters shown in Table 2. Three separate neural networks were evaluated. For comparison with
Bartlett's results, the first NN is for recognizing individual AUs only. The second NN is for recognizing
AU combinations when only modeling 7 individual upper face AUs. The third NN is for recognizing AU
combinations when separately modeling nonadditive AU combinations. The desired number of hidden
units to achieve a good recognition was also investigated.
7.1.1 Upper Face Individual AU Recognition
The NN outputs are 7 individual upper face AUs. Each output unit gives an estimate of the probability
of the input image consisting of the associated action units. From experiments, we have found 6 hidden
units are sufficient.
In order to recognize individual action units, we used the training and testing data that include individual
AUs only. Table 8 shows results of our NN on the T rainS1, TestS1 training and testing sets. A 92.3%
recognition rate was obtained. When we increase the training data by using T rainS2 and test by using
TestS2, a 92% recognition rate was obtained.
For detecting the system's robustness to new faces, we tested our algorithm on the T rainS3/TestS3
training/testing sets. The recognition results are shown in Table 9. The average recognition rate is 92.9%
with zero false alarms. For the misidentifications between AUs, although the probability of the output
units of the labeled AU is very close to the highest probability, it was treated as an incorrect result. For

Table

8. AU recognition for single AUs on T rainS1 and TestS1. The rows correspond to NN outputs,
and columns correspond to human labels.
Average Recognition rate: 92.3%
example, if we obtain the probability of AU1 and AU2 with AU1=0.59 and AU2=0.55 for a labeled
AU2, it means that AU2 was misidentified as AU1. When we tested the NN trained on single AU image
sequences on data set containing AU combinations, we found the recognition rate decreases to 78.7%.

Table

9. AU recognition for single AUs when all test data come from new subjects who were not used
for training.
Average Recognition rate: 92.9%
7.1.2 Upper Face AU Combination Recognition When Modeling 7 Individual AUs
This NN is similar to the one used in the previous section, except that more than one output units could
fire. We also restrict the output to be the first 7 individual AUs. For the additive and nonadditive AU
combinations, the same value is given for each corresponding individual AUs in training data set. For
example, for AU1+2+4, the outputs are AU1=1.0, AU2=1.0, and AU4=1.0. From experiments, we found
we need to increase the number of hidden units from 6 to 12.

Table

shows the results of our NN on the (T rainC1)/(TestC1) training/testing set. A 95% average
recognition rate is achieved, with a false alarm rate of 6.4%. The higher false alarm rate comes from
the AU combination. For example, if we obtained the recognition results with AU1=0.59 and AU2=0.55
for a labeled AU2, it was treated as AU1+AU2. This means AU2 is recognized but with AU1 as a false
alarm.

Table

10. AU recognition for AU combinations when modeling 7 single AUs only.
AU No. Correct false Missed Confused Recognition
rate
Total 94
False alarm: 6.4%
7.1.3 Upper Face AU Combination Recognition When Modeling Nonadditive Combinations
For this NN,we separately model the nonadditive AU combinations. The 11 outputs consist of 7 individual
upper face AUs and 4 non-additive AU combinations (AU1+2, AU1+4, AU4+5, and AU1+2+4). The
non-additive AU combinations and the corresponding individual AUs strongly depend on each other.

Table

11 shows the correlations between AU1, AU2, AU4, AU5, AU1+2, AU1+2+4, AU1+4, and AU4+5
used in the training set. We set the values based on the appearances of these AUs or combinations.

Table

12 shows the results of our NN on the (T rainC1)/(T estC1) training/testing set. An average
recognition rate of 93.7% is achieved, with a slightly lower false alarm rate of 4.5%. In this case,
modeling separately the nonadditive combinations does not improve recognition rate due to the fact that

Table

11. The correlation of AU1, AU2, AU4, AU5, AU1+2, AU1+2+4, AU1+4, and AU4+5.
the AUs in these combinations strongly depend on each other.

Table

12. AU recognition for AU combinations by modeling the non-additive AU combinations as separate
AUs.
AU No. Correct false Missed Confused Recognition
rate
Total 111 104 5 7 - 93.7%
False alarm: 4.5%
7.2. Lower Face Action Units Recognition
We used a three-layer neural network with one hidden layer to recognize the lower face action units.
The inputs of the neural network are the lower face feature parameters shown in Table 3. 7 parameters
are used except two parameters of the nasolabial furrows. We don't use the angles of the nasolabial
furrows because they are varied much for the different subjects. Generally, we use them to analyze the
different expressions of same subject.
Two separate neural networks are trained for lower face AU recognition. The outputs of the first NN
ignore the nonadditive combinations and only models 11 basic single action units which are shown in

Table

5. We use AU 23+24 instead of AU23 and AU24 because they almost occur together. The outputs
of the second one separately models some nonadditive combinations such as AU9+17 and AU10+17
besides the basic single action units.
The recognition results for modeling basic lower face AUs only are shown in Table 13 with recognition
rate of 96.3%. The recognition results for modeling non-additive AU combinations are shown in Table 14
with average recognition rate of 96.71%. We found that separately model the nonadditive combinations
slightly increase lower action unit recognition accuracy.
All the misidentifications come from AU10, AU17, and AU26. All the mistakes of AU26 are confused
by AU25. It is reasonable because both AU25 and AU26 are with parted lips. But for AU26, the mandible
is lowered. We did not use the jaw motion information in current system. All the mistakes of AU10 and
AU17 are caused by the image sequences with AU combination AU10+17. Two combinations AU10+17
are classified to AU10+12. One combination of AU10+17 is classified as AU10 (missing AU17). The
combination AU 10+17 modified the single AU's appearance. The neural network needs to learn the
modification by more training data of AU 10+17. There are only ten examples of AU10+17 in 1220
training data in our current system. More data about AU10+17 is collecting for future training. Our
system is able to identify action units regardless of whether they occurred singly or in combinations. Our
system is trained with the large number of subjects, which included African-Americans and Asians in
addition to European-Americans, thus providing a sufficient test of how well the initial training analyses
generalized to new image sequences.
For evaluating the necessity of including the nonadditive combinations, we also train a neural network
using 11 basic lower face action units as the outputs. For the same test data set, the average recognition
rate is 96.3%.
8. Conclusion and Discussion
We developed a feature-based facial expression recognition system to recognize both individual AUs
and AU combinations. To localize the subtle changes in the appearance of facial features, we developed
a multi-state method of tracking facial features that uses convergent methods of feature analysis. It has

Table

13. Lower face action unit recognition results for modeling basic lower face AUs only.
AU No. Correct false Missed Confused Recognition
rate
9
Total

Table

14. Lower face action unit recognition results for modeling non-additive AU combinations.
AU No. Correct false Missed Confused Recognition
rate
9
26 14 9 - 5 (AU25) 64.29%
Total
high sensitivity and specificity for subtle differences in facial expressions. All the facial features are
represented in a group of feature parameters.
The network was able to learn the correlations between facial feature parameter patterns and specific
action units. Although often correlated, these effects of muscle contraction potentially provide unique
information about facial expression. Action units 9 and 10 in FACS, for instance, are closely related
expressions of disgust that are produced by variant regions of the same muscle. The shape of the nasolabial
furrow and the state of nose wrinkles distinguishe between them. Changes in the appearance of facial
features also can affect the reliability of measurements of pixel motion in the face image. Closing of
the lips or blinking of the eyes produces occlusion, which can confound optical flow estimation. Unless
information about both motion and feature appearance are considered, accuracy of facial expression
analysis and, in particular, sensitivity to subtle differences in expression may be impaired. A recognition
rate of 95% was achieved for seven basic upper face AUs. Eleven basic lower face action units are
recognized and 96.71% of action units were correctly classified.
Unlike previous methods [9] which build a separate model for each AU and AU combination, we build
a single model that recognizes AUs whether they occur singly or in combinations. This is an important
capability since the number of possible AU combinations is too large (over 7000) for each combination
to be modeled separately.
Using the same database, Bartlett et al. [1] recognized only 6 single upper face action units but no
combinations. The performance of their feature-based classifier on novel faces was 57%; on new images
of a face used for training, the rate was 85.3%. After they combined holistic spatial analysis, feature
measures and optical flow, they obtained their best performance at 90.9% correct. Compared to their
system, our feature-based classifier obtained a higher performance rate about 92.5% on both novel faces
and new images of a face used for training for individual AU recognition. Moreover, our system works
well for a more difficult case in which AUs occur either individually or in additive and nonadditive
AU combinations. 95% of upper face AUs or AU combinations are correctly classified regardless of
whether these action units occur singly or in combination. Those disagreements that did occur were from
nonadditive AU combinations such as AU1+2, AU1+4, AU1+2+4, AU4+5, and AU6+7. As a result,
more analysis of the nonadditive AU combinations should be done in future.
From the experimental results, we have the following observations:
1. The recognition performance from facial feature measurements is comparable to holistic analysis
and Gabor wavelet representation for AU recognition.
2. 5 to 7 hidden units are sufficient to code 7 individual upper face AUs. 10 to 16 hidden units are
needed when AUs may occur either singly or in complex combinations.
3. For upper face AU recognition, separately modeling nonadditive AU combinations affords no
increase in the recognition accuracy. In contrast, separately modeling nonadditive AU combinations
affords slightly increase in the recognition accuracy for lower face AU recognition.
4. After using sufficient data to train the NN, recognition accuracy is stable for recognizing AUs of
new faces.
In summary, the face image analysis system demonstrated concurrent validity with manual FACS
coding. The multi-state model based convergent-measures approach was proved to capture the subtle
changes of facial features. In the test set, which included subjects of mixed ethnicity, average recognition
accuracy for 11 basic action units in the lower face was 96.71%, for 7 basic action units in the upper
face was 95%, regardless of these action units occur singly or in combinations. This is comparable to
the level of inter-observer agreement achieved in manual FACS coding and represents advancement over
the existing computer-vision systems that can recognize only a small set of prototypic expressions that
vary in many facial regions.

Acknowledgements

The authors would like to thank Paul Ekman, Human Interaction Laboratory, University of California,
San Francisco, for providing the database. The authors also thank Zara Ambadar, Bethany Peters, and
Michelle Lemenager for processing the images. This work is supported by NIMH grant R01 MH51435.



--R

Measuring facial expressions by computer image analysis.
Facial expression in hollywood's portrayal of emotion.
Classifying facial actions.
The Facial Action Coding System: A Technique For The Measurement of Facial Movement.

Facial feature point extraction method based on combination of shape extraction and pattern matching.
Application of the k-l procedure for the characterization of human faces
Age classification from facial images.

An interative image registration technique with an application in stereo vision.
Recognition of facial expression from optical flow.
Handbook of methods in nonverbal behavior research.
Analysis of facial images using physical and anatomical models.

Robust lip tracking by combining shape
face recognition using eigenfaces.
Recognizing human facial expression from long image sequences using optical flow.
Feature extraction from faces using deformable templates.

--TR

--CTR
Chao-Fa Chuang , Frank Y. Shih, Rapid and Brief Communication: Recognizing facial action units using independent component analysis and support vector machine, Pattern Recognition, v.39 n.9, p.1795-1798, September, 2006
Jia-Jun Wong , Siu-Yeung Cho, Facial emotion recognition by adaptive processing of tree structures, Proceedings of the 2006 ACM symposium on Applied computing, April 23-27, 2006, Dijon, France
Robust feature detection for facial expression recognition, Journal on Image and Video Processing, v.2007 n.2, p.5-5, August 2007
Jiatao Song , Zheru Chi , Jilin Liu, A robust eye detection method using combined binary edge and intensity information, Pattern Recognition, v.39 n.6, p.1110-1125, June, 2006
Seong G. Kong , Jingu Heo , Besma R. Abidi , Joonki Paik , Mongi A. Abidi, Recent advances in visual and infrared face recognition: a review, Computer Vision and Image Understanding, v.97 n.1, p.103-135, January 2005
Matthew Turk, Computer vision in the interface, Communications of the ACM, v.47 n.1, January 2004
Alice J. O'Toole , Joshua Harms , Sarah L. Snow , Dawn R. Hurst , Matthew R. Pappas , Janet H. Ayyad , Herve Abdi, A Video Database of Moving Faces and People, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.27 n.5, p.812-816, May 2005
Hatice Gunes , Massimo Piccardi , Tony Jan, Face and body gesture recognition for a vision-based multimodal analyzer, Proceedings of the Pan-Sydney area workshop on Visual information processing, p.19-28, June 01, 2004
Mu-Chun Su , Yi-Jwu Hsieh , De-Yuan Huang, A simple approach to facial expression recognition, Proceedings of the 2007 annual Conference on International Conference on Computer Engineering and Applications, p.456-461, January 17-19, 2007, Gold Coast, Queensland, Australia
Dong Liang , Jie Yang , Zhonglong Zheng , Yuchou Chang, A facial expression recognition system based on supervised locally linear embedding, Pattern Recognition Letters, v.26 n.15, p.2374-2389, November 2005
Congyong Su , Li Huang, Spatio-temporal graphical-model-based multiple facial feature tracking, EURASIP Journal on Applied Signal Processing, v.2005 n.1, p.2091-2100, 1 January 2005
Yongmian Zhang , Qiang Ji, Active and Dynamic Information Fusion for Facial Expression Understanding from Image Sequences, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.27 n.5, p.699-714, May 2005
Shyi-Chyi Cheng , Ming-Yao Chen , Hong-Yi Chang , Tzu-Chuan Chou, Semantic-based facial expression recognition using analytical hierarchy process, Expert Systems with Applications: An International Journal, v.33 n.1, p.86-95, July, 2007
Philipp Michel , Rana El Kaliouby, Real time facial expression recognition in video using support vector machines, Proceedings of the 5th international conference on Multimodal interfaces, November 05-07, 2003, Vancouver, British Columbia, Canada
Benjamn Hernndez , Gustavo Olague , Riad Hammoud , Leonardo Trujillo , Eva Romero, Visual learning of texture descriptors for facial expression recognition in thermal imagery, Computer Vision and Image Understanding, v.106 n.2-3, p.258-269, May, 2007
Iain Matthews , Jing Xiao , Simon Baker, 2D vs. 3D Deformable Face Models: Representational Power, Construction, and Real-Time Fitting, International Journal of Computer Vision, v.75 n.1, p.93-113, October   2007
Yan Tong , Yang Wang , Zhiwei Zhu , Qiang Ji, Robust facial feature tracking under varying face pose and facial expression, Pattern Recognition, v.40 n.11, p.3195-3208, November, 2007
Maria Shugrina , Margrit Betke , John Collomosse, Empathic painting: interactive stylization through observed emotional state, Proceedings of the 4th international symposium on Non-photorealistic animation and rendering, June 05-07, 2006, Annecy, France
Haisong Gu , Yongmian Zhang , Qiang Ji, Task oriented facial behavior recognition with selective sensing, Computer Vision and Image Understanding, v.100 n.3, p.385-415, December 2005
Yanxi Liu , Karen L. Schmidt , Jeffrey F. Cohn , Sinjini Mitra, Facial asymmetry quantification for expression invariant human identification, Computer Vision and Image Understanding, v.91 n.1-2, p.138-159, July
Tao Xiang , Shaogang Gong, Model Selection for Unsupervised Learning of Visual Context, International Journal of Computer Vision, v.69 n.2, p.181-201, August    2006
Chuang , Christoph Bregler, Mood swings: expressive speech animation, ACM Transactions on Graphics (TOG), v.24 n.2, p.331-347, April 2005
Zhang , Zicheng Liu , Dennis Adler , Michael F. Cohen , Erik Hanson , Ying Shan, Robust and Rapid Generation of Animated Faces from Video Images: A Model-Based Modeling Approach, International Journal of Computer Vision, v.58 n.2, p.93-119, July 2004
R. W. Picard , S. Papert , W. Bender , B. Blumberg , C. Breazeal , D. Cavallo , T. Machover , M. Resnick , D. Roy , C. Strohecker, Affective Learning  A Manifesto, BT Technology Journal, v.22 n.4, p.253-269, October 2004
Aleix M. Martnez, Recognizing Imprecisely Localized, Partially Occluded, and Expression Variant Faces from a Single Sample per Class, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.24 n.6, p.748-763, June 2002
Scott Brave , Clifford Nass, Emotion in human-computer interaction, The human-computer interaction handbook: fundamentals, evolving technologies and emerging applications, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 2002
neural network;facial expression analysis;action units;multistate face and facial component models;computer vision;facial action coding system;AU combinations