Soft Locality Preserving Map (SLPM) for Facial Expression
Recognition
Cigdem Turana,*, Kin-Man Lama, Xiangjian Heb
a

Centre for Signal Processing, Department of Electronic and Information Engineering, The Hong
Kong Polytechnic University, Kowloon, Hong Kong
b
Computer Science, School of Electrical and Data Engineering, University of Technology, Sydney,
Australia
*Corresponding author.
E-mail addresses: cigdem.turan@connect.polyu.hk (C. Turan), enkmlam@polyu.edu.hk (K.-M. Lam),
xiangjian.he@uts.edu.au (X. He)

ABSTRACT
For image recognition, an extensive number of methods have been proposed to overcome the highdimensionality problem of feature vectors being used. These methods vary from unsupervised to
supervised, and from statistics to graph-theory based. In this paper, the most popular and the state-ofthe-art methods for dimensionality reduction are firstly reviewed, and then a new and more efficient
manifold-learning method, named Soft Locality Preserving Map (SLPM), is presented. Furthermore,
feature generation and sample selection are proposed to achieve better manifold learning. SLPM is a
graph-based subspace-learning method, with the use of k-neighbourhood information and the class
information. The key feature of SLPM is that it aims to control the level of spread of the different
classes, because the spread of the classes in the underlying manifold is closely connected to the
generalizability of the learned subspace. Our proposed manifold-learning method can be applied to
various pattern recognition applications, and we evaluate its performances on facial expression
recognition. Experiments on databases, such as the Bahcesehir University Multilingual Affective Face
Database (BAUM-2), the Extended Cohn-Kanade (CK+) Database, the Japanese Female Facial
Expression (JAFFE) Database, and the Taiwanese Facial Expression Image Database (TFEID), show
that SLPM can effectively reduce the dimensionality of the feature vectors and enhance the
discriminative power of the extracted features for expression recognition. Furthermore, the proposed
feature-generation method can improve the generalizability of the underlying manifolds for facial
expression recognition.

1. Introduction
Dimensionality reduction, which aims to find the distinctive features to represent high-dimensional data
in a low-dimensional subspace, is a fundamental problem in classification. Many real-world computervision and pattern-recognition applications, e.g. facial expression recognition, are involved with large
volumes of high-dimensional data. Principal Component Analysis (PCA) [6, 7] and Linear Discriminant
Analysis (LDA) [7, 10] are two notable linear methods for dimensionality reduction. PCA aims to find
principal projection vectors, which are those eigenvectors associated with the largest eigenvalues of the
covariance matrix of training samples, to project the high-dimensional data to a low-dimensional
subspace. Unlike PCA, which is an unsupervised method that considers common features of training
samples, LDA employs the Fisher criteria to maximize the between-class scattering and to minimize
the within-class scattering. Although LDA is superior to PCA for pattern recognition, it suffers from
the small-sample-size (SSS) problem [13] because the number of training samples available is much
smaller than the dimension of the feature vectors in most of the real-world applications. To overcome
the SSS problem, Li et al. [15] proposed the Maximum Margin Criterion (MMC) method, which utilizes
the difference between the within-class and the between-class scatter matrices as the objective function.

In [16], it is shown that intra-class scattering has an important effect when dealing with overfitting in
training a model. Unlike the conventional wisdom, too much compactness within each class decreases
the generalizability of the manifolds. Since LDA and MMC are too “harsh”, they need to be softened.
Liu et al. [16] proposed the Soft Discriminant Map (SDM), which tries to control the spread of the
different classes. MMC can be considered as a special case of SDM, where the softening parameter 𝛽 =
1.
Linear methods, like PCA, LDA and SDM, may fail to find the underlying nonlinear structure of
the data under consideration, and they may lose some discriminant information of the manifolds during
the linear projection. To overcome this problem, some nonlinear dimensionality reduction techniques
have been proposed. In general, the nonlinear dimensionality reduction techniques are divided into two
categories: kernel-based and manifold-learning-based approaches. Kernel-based methods, as well as the
linear methods mentioned above, only employ the global structure while ignoring the local geometry of
the data. However, manifold-learning-based methods can explore the intrinsic geometry of the data.
Popular nonlinear manifold-learning methods include ISOMAP [17], LLE [18], and Laplacian
Eigenmaps [19], which can be considered as special cases of the general framework for dimensionality
reduction named “graph embedding”, proposed by Yan et al. [5]. Although these methods can represent
the local structure of the data, they suffer from the out-of-sample problem. Locality Preserving
Projection (LPP) [1] was proposed as a linear approximation of the nonlinear Laplacian Eigenmaps [19]
to overcome the out-of-sample problem. LPP considers the manifold structure via the adjacency graph.
The manifold-learning methods presented so far are unsupervised methods, i.e. they do not consider the
class information. Several supervised-based methods [2, 20, 21] have been proposed, which utilize the
discriminant structure of the manifolds. With the Marginal Fisher Analysis (MFA) [5], which uses the
Fisher criterion and constructs two adjacency graphs to represent the within-class and the between-class
geometry of the data, several other methods have been proposed with similar ideas, such as LocalityPreserved Maximum Information Projection (LPMIP) [9], Constrained Maximum Variance Mapping
(CMVM) [8], and Locality Sensitive Discriminant Analysis (LSDA) [4]. In real-life applications,
unlabeled data can exist because of various reasons. To deal with this problem, various semi-supervised
learning algorithms have also been proposed [22-24].
In this paper, we propose a new graph-based subspace-learning method to solve the various
problems of the existing methods by combining their best components to form a better method. The
proposed method, named “Soft Locality Preserving Map (SLPM)” can be outlined as follows:
1. SLPM constructs a within-class graph matrix and a between-class graph matrix using the k-nearest
neighborhood and the class information to discover the local geometry of the data.
2. To overcome the SSS problem and to decrease the computational cost of computing the inverse of
a matrix, SLPM defines its objective function as the difference between the between-class and the
within-class Laplacian matrices.
3. Inspired by the idea of SDM on the importance of the intra-class spread, a parameter β is added to
control the penalty on the within-class Laplacian matrix so as to avoid the overfitting problem and
to increase the generalizability of the underlying manifold.
Although subspace-learning methods have demonstrated promising performances by increasing the
discriminative power of training data after transformation, they might fail to exhibit a similar
performance on testing data. To improve the generalizability of the manifolds generated by the
subspace-analysis methods, more training samples, which are located near the boundaries of the
respective classes, are desirable. In this paper, we apply our proposed SLPM method to facial expression
recognition, and propose an efficient way to enhance the generalizability of the manifolds of the
different expression classes by feature generation. An expression video sequence, which ranges from a
neutral-expression face to the highest intensity of an expression, allows us to select appropriate samples
for learning a better and more representative manifold for the expression class. For the optimal manifold
of an expression class, its center should represent those samples that best represent the facial expression
concerned, i.e. those expression face images with the highest intensities. When moving away from the
manifold center, the corresponding expression intensity should be reducing. Those samples near the

boundary of a manifold are important for describing the expression, which also defines the shape of the
manifold. To describe a manifold boundary, images with low-intensity expressions should be
considered. Since the feature vectors used to represent facial expressions usually have high
dimensionality, many training samples near the manifold boundary are required, so as to represent it
completely. However, we usually have a limited number of weak-intensity expression images, so
feature generation is necessary to learn more complete manifolds.
In other applications, additional samples have also been generated for manifold learning. In [25],
faces are morphed between two people with different percentages so as to generate face images near
the manifold boundaries. By generating more face images and extracting their feature vectors, the
manifold for each face subject can be learned more accurately. Therefore, the decision region for each
subject can be determined for watch-list surveillance. In our algorithm, rather than morphing faces and
extracting features from the synthesized face, we generate features for low-intensity expressions
directly in the feature domain. Generating features in this way should be more accurate than extracting
features from distorted faces generated by morphing. Several fields of research, such as text
categorization [26], handwritten digit recognition [27], facial expression recognition [28, 29], etc., have
also employed feature generation to achieve better learning. Unlike these methods, which generate
features in the image domain, the proposed method generates features in the feature domain.
In Section 2, we further explain the graph-embedding techniques, and give a detailed comparison
of those existing subspace-learning approaches similar to our proposed method. In Section 3, the
proposed method, SLPM, is formulated and its relation to SDM is further explored. In Section 4, we
explain the local descriptors used in our experiments and the feature-generation algorithm, and describe
how to enhance the manifold learning with low-intensity images. In Section 5, we present the databases
used in our experiments, and the preprocessing of the face images. Then, experiment results are
represented, with a discussion. We conclude this paper in Section 6.

2. Literature review of subspace learning
In this section, a review of the graph-embedding techniques is presented in detail, with the different
variants. Then, graph-based subspace-learning methods are described in two parts: 1) how the adjacency
matrices are constructed, and 2) how their objective functions are defined.
2.1. Graph embedding
Given 𝑚 data points {𝒙1 , 𝒙2 , … , 𝒙𝑚 } ∈ ℝ𝐷 , the graph-based subspace-learning methods aim to find a
transformation matrix A that maps the training data points to a new set of points {𝒚1 , 𝒚2 , … , 𝒚𝑚 } ∈ ℝ𝑑
(𝑑 ≪ 𝐷), where 𝒚𝑖 = 𝐀𝑇 𝒙𝑖 and A is the projection matrix. After the transformation, the data points 𝒙𝑖
and 𝒙𝑗 , which are close to each other, will have their projections in the manifold space 𝒚𝑖 and 𝒚𝑗 close
to each other. This goal can be achieved by minimizing the following objective function:
2

∑𝑖𝑗(𝒚𝑖 − 𝒚𝑗 ) 𝑤𝑖𝑗 ,
(1)
where 𝑤𝑖𝑗 represents the similarity between the training data xi and xj. If 𝑤𝑖𝑗 is non-zero, 𝒚𝑖 and 𝒚𝑗
must be close to each other, in order to minimize (1). Taking the data points in the feature space as
nodes of a graph, an edge between nodes 𝑖 and 𝑗 has a weight of 𝑤𝑖𝑗 , which is not zero, if they are close
to each other. In the literature, we have found three different ways to determine the local geometry of a
data point:
1. ε-neighbourhood: This uses the distance to determine the closeness. Given ε (ε ∈ ℝ), εneighbourhood chooses the data points that fall within the circle around 𝑥𝑖 with a radius ε.
Those data points fall within the ε-neighbourhood of 𝒙𝑖 can be defined as
𝑂(𝒙𝑖 , ε) = {𝒙 | ‖𝒙 − 𝒙𝑖 ‖2 < ε}.
(2)
2. k-nearest neighbourhood: Another way of determining the local structure is to use the nearest
neighbourhood information. Presuming that the closest k points of 𝒙𝑖 would still be the closest

data points of 𝒚𝑖 in the projected manifold space, we can define a function 𝑁(𝒙𝑖 , 𝑘), which
outputs the set of k-nearest neighbours of 𝒙𝑖 . Two types of neighbourhood, with label
information incorporated, are considered: 𝑁(𝒙𝑖 , 𝑘 + ) and 𝑁(𝒙𝑖 , 𝑘 − ), which represent the sets
of k-nearest neighbours of 𝒙𝑖 of the same label and of different labels, respectively.
3. The class information: The class or label information is often used in supervised subspace
methods. In a desired manifold subspace, the data points belonging to the class of 𝒙𝑖 are to be
projected such that they are close to each other, so as to increase the intra-class compactness.
The data points belonging to other classes are projected, such that they will become farther
apart and have larger inter-class separability. The class label information is often combined
with either the ε-neighbourhood or the k-nearest neighbourhood.
The similarity graph is constructed by setting up edges between the nodes. There are different ways of
determining the weights of the edges, considering the fact that the distance between two neighbouring
points can also provide useful information about the manifold. Given a sparse symmetric similarity
matrix 𝐖, two variations have been proposed in the literature.
1. Binary weights: 𝑤𝑖𝑗 = 1 if, and only if, the nodes 𝑖 and 𝑗 are connected by an edge, otherwise
𝑤𝑖𝑗 = 0.
2. Heat kernel (𝑡 ∈ ℝ): If the nodes 𝑖 and 𝑗 are connected by an edge, the weight of the edge is
defined as
2

−‖𝒙𝑖 − 𝒙𝑗 ‖ ⁄
𝑤𝑖𝑗 = exp (
𝑡) .

(3)

After constructing the similarity matrix with the weights, the minimization problem defined in (1) can
be solved by using the spectral graph theory. Defining the Laplacian matrix 𝐋 = 𝐃 − 𝐖, where 𝐃 is the
diagonal matrix whose entries are the column sum of 𝐖, i.e. 𝑑𝑖𝑖 = ∑𝑗 𝑤𝑖𝑗 , the objective function is
reduced to
2

2

min ∑(𝒚𝑖 − 𝒚𝑗 ) 𝑤𝑖𝑗 = min ∑(𝐀T 𝒙𝑖 − 𝐀T 𝒙𝑗 ) 𝑤𝑖𝑗
𝑖𝑗

𝑖𝑗

= min 𝐀T 𝐗𝐋𝐗 T 𝐀,
(4)
where 𝐗 = [𝒙1 , 𝒙2 , … , 𝒙𝑚 ]. To avoid the trivial solution of the objective function, the constraint
𝐀T 𝐗𝐃𝐗 T 𝐀 = 1 is often added. After specifying the objective function, the optimal projection matrix A
can be obtained by choosing the eigenvectors corresponding to the 𝑑 (𝑑 ≪ 𝐷) smallest non-zero
eigenvalues computed by solving the standard eigenvalue decomposition or generalized eigenvalue
problem, depending on the objective function being considered.
2.1.1. Constructing the within-class and the between-class graph matrices
As mentioned in the last section, one of the most popular graph-based subspace-learning methods is
LPP [1], which uses an intrinsic graph to represent the locality information of the dataset, i.e. the
neighbourhood information. The idea behind LPP is that if the data points 𝒙𝑖 and 𝒙𝑗 are close to each
other in the feature space, then they should also be close to each other in the manifold subspace. The
similarity matrix 𝑤𝑖𝑗 for LPP can be defined as follows:
1, if 𝒙𝑖 ∈ 𝑁(𝒙𝑗 , 𝑘) or 𝒙𝑗 ∈ 𝑁(𝒙𝑖 , 𝑘),
𝑤𝑖𝑗 = {
(5)
0, otherwise,
where 𝑁(𝒙𝑖 , 𝑘) represents the set of k-nearest neighbors of 𝒙𝑖 . One shortfall of the above formulation
for wij is that it is an unsupervised method, i.e. not using any class-label information. Thinking that the
label information can help to find a better separation between different class manifolds, Supervised
Locality Preserving Projections (SLPP) was introduced in [2]. Denote 𝑙(𝒙𝑖 ) as the corresponding class
label of the data point 𝒙𝑖 . SLPP uses either one of the following formulations:
1, if 𝑙(𝒙𝑖 ) = 𝑙(𝒙𝑗 ),
𝑤𝑖𝑗 = {
(6)
0, otherwise,

1, if (𝒙𝑖 ∈ 𝑁(𝒙𝑗 , 𝑘) or 𝒙𝑗 ∈ 𝑁(𝒙𝑖 , 𝑘)) and 𝑙(𝒙𝑖 ) = 𝑙(𝒙𝑗 ),
𝑤𝑖𝑗 = {
(7)
0, otherwise.
Note that (6) does not include the neighbourhood information to the adjacency graph, and the similarity
matrices defined above can be constructed using the heat kernel. Orthogonal Locality Preserving
Projection (OLPP) [3] whose eigenvectors are orthogonal to each other is an extension of LPP. Please
note that, in our experiments we applied Supervised Orthogonal Locality Preserving Projections
(SOLPP), which is OLPP with its adjacency matrix including class information.
Yan et al. [5] proposed a general framework for dimensionality reduction, named Marginal Fisher
Analysis (MFA). MFA, which is based on graph embedding as LPP, uses two graphs, the intrinsic and
penalty graphs, to characterize the intra-class compactness and the interclass separability, respectively.
In MFA, the intrinsic graph 𝑤𝑖𝑗𝑤 , i.e. the within-class graph, is constructed using the neighbourhood and
class information as follows:
1, if 𝒙𝑖 ∈ 𝑁(𝒙𝑗 , 𝑘1+ ) or 𝒙𝑗 ∈ 𝑁(𝒙𝑖 , 𝑘1+ ),
𝑤𝑖𝑗𝑤 = {
(8)
0, otherwise.
Similarly, the penalty graph 𝑤𝑖𝑗𝑏 , i.e. the between-class graph, is constructed as follows:
1, if 𝒙𝑖 ∈ 𝑁(𝒙𝑗 , 𝑘2− ) or 𝒙𝑗 ∈ 𝑁(𝒙𝑖 , 𝑘2− ),
𝑤𝑖𝑗𝑏 = {
(9)
0, otherwise.
Locality Sensitive Discriminant Analysis (LSDA) [4] and Improved Locality Sensitive
Discriminant Analysis (ILSDA) [14] are subspace-learning methods proposed in 2007 and 2015,
respectively. They construct the similarity matrices in the same way, but LSDA uses binary weights,
while ILSDA sets the weight of the edges using the heat kernel. The similarity matrices of LSDA are
defined as follows:
1, if 𝒙𝑖 ∈ 𝑁(𝒙𝑗 , 𝑘) and 𝑙(𝒙𝑖 ) = 𝑙(𝒙𝑗 ),
𝑤𝑖𝑗𝑤 = {
(10)
0, otherwise.
1, if 𝒙𝑖 ∈ 𝑁(𝒙𝑗 , 𝑘) and 𝑙(𝒙𝑖 ) ≠ 𝑙(𝒙𝑗 ),
𝑤𝑖𝑗𝑏 = {
(11)
0, otherwise.
It can be observed that the intrinsic and the penalty graphs of MFA, LSDA, and ILSDA are similar to
each other. In MFA, the numbers of neighboring points for both the similarity matrices are known, i.e.
k1 and k2. In LSDA and ILSDA, the k neighbors of 𝒙𝑖 are selected, which are then divided for
constructing the within-class (𝑘 + samples the same class as 𝒙𝑖 ) and the between-class matrices (𝑘 −
samples of other classes), i.e. 𝑘 = 𝑘 + + 𝑘 − . Let 𝑘1 and 𝑘2 be the numbers of samples belonging to the
same class and different classes, respectively, for MFA. The following relation is not always true:
𝑁(𝒙𝑖 , 𝑘1 ) ∪ 𝑁(𝒙𝑖 , 𝑘2 ) = 𝑁(𝒙𝑖 , 𝑘),
(12)
because it is not necessarily true that 𝑘 + = 𝑘1 and 𝑘 − = 𝑘2 . Therefore, the neighboring points of 𝑥𝑖 in
LSDA and ILSDA are not the same as MFA, even if 𝑘 = 𝑘𝑡 = 𝑘1 + 𝑘2 . However, the adjacency
matrices constructed in the manifold learning methods are similar to each other. The main difference
between the existing methods in the literature is in their definitions of the objective functions. We will
elaborate on the differences in the objective functions in the next section.
Locality-Preserved Maximum Information Projection (LPMIP) [9], proposed in 2008, uses the εneighbourhood condition, i.e. 𝑂(𝒙𝑖 , ε). Although it was originally applied as an unsupervised learning
method, the class labels were used to construct the locality and non-locality information for facial
expression recognition. In 2008, Li et al. [8] proposed Constrained Maximum Variance Mapping
(CMVM), which aims to keep the local structure of the data, while separating the different manifolds,
i.e. different classes, farther apart. The local-structure graphs, i.e. the between-class graph and the
dissimilarities graph, are defined as follows:
2

−‖𝒙𝑖 − 𝒙𝑗 ‖ ⁄
1 or exp (
𝑡) , if 𝒙𝑖 ∈ 𝑂(𝒙𝑗 , ε),
𝑤𝑖𝑗𝑤 = {
0, otherwise,

(13)

1, if 𝑙(𝒙𝑖 ) ≠ 𝑙(𝒙𝑗 ),
𝑤𝑖𝑗𝑏 = {
(14)
0, otherwise.
As (13) and (14) show, the within-class matrix of CMVM only preserves the local structure of the whole
data, while the between-class matrix only uses the class label to increase the separability of different
class manifolds. In 2015, an extension of CMVM, namely CMVM+ [12], was proposed to overcome
the obstacles of CMVM. CMVM+ adds the class information and neighbourhood information to the
similarity matrices. The updated version of the graphs can be written as follows:
1, 𝒙𝑖 ∈ 𝑁(𝒙𝑗 , 𝑘) and 𝑙(𝒙𝑖 ) = 𝑙(𝒙𝑗 ),
𝑤𝑖𝑗𝑤 = {
(15)
0, otherwise,
1, if 𝑙(𝒙𝑗 ) ∈ 𝐶𝑖𝑛𝑐 (𝒙𝑖 ),
𝑤𝑖𝑗𝑏 = {
(16)
0, otherwise,
where 𝐶𝑖𝑛𝑐 (𝒙𝑖 ) is a set of neighboring points belonging to different classes, i.e. 𝑙(𝒙𝑖 ) ≠ 𝑙(𝒙𝑗 ). More
details of the function 𝐶𝑖𝑛𝑐 (𝒙𝑖 ) can be found in [12].
In 2011, Multi-Manifolds Discriminant Analysis (MMDA) [11] was proposed for image feature
extraction, and applied to face recognition. The idea behind MMDA is to keep the points from the same
class as close as possible in the manifold space, with the within-class matrix defined as follows:
2

−‖𝒙𝑖 − 𝒙𝑗 ‖ ⁄
exp (
𝑡) , if 𝑙(𝒙𝑖 ) = 𝑙(𝒙𝑗 ),
𝑤𝑖𝑗𝑤 = {

(17)

0, otherwise.
MMDA also constructs a between-class matrix in order to separate the different classes from each other.
The difference between the between-class matrix of MMDA and the other subspace methods is that its
graph matrix is constructed by not taking all the data points as nodes, but rather calculating the weighted
centres of different classes by averaging all the data points belonging to the classes under consideration.
̃ 1, 𝒎
̃ 2, … , 𝒎
̃ 𝑐 ] be the class-weighted centres, where c is the number of classes. Then, the
Let 𝐌 = [𝒎
between-class matrix of MMDA can be written as:
2

̃𝑖 −𝒎
̃ 𝑗‖ ⁄
−‖𝒎
𝑤𝑖𝑗𝑏 = exp (
𝑡).

(18)

In Table 1, a summary is given of the within-class graph and between-class graph for the subspacelearning methods, reviewed in this paper.
2.1.2. Defining the objective functions
Table 2 summarizes the objective functions of the approaches reviewed in the previous section, as well
as the constraints used. We can see that SLPP has only one Laplacian matrix defined in its objective
function, because it constructs one similarity matrix only, while all the other methods have two matrices:
one is based on the intrinsic graph, and the other on the penalty graph.
In general, there are two ways of defining the objective functions with the intrinsic and the penalty
matrices. The first one utilizes the Fisher criterion to maximize the ratio between the scattering of the
between-class and that of the within-class Laplacian matrices. MFA, MMDA, and CMVM+ employ
the Fisher criterion. Although the application of the Fisher criterion shows its robustness, it involves
taking the inverse of a high-dimensional matrix to solve a generalized eigenvalue problem. To solve
this problem, LSDA, LPMIP, and our proposed method define the objective functions as the difference
between the intrinsic and the penalty-graph matrices, while MMC and SDM use the difference between
the inter-class and the intra-class scatter matrices.
As shown in Table 2, ILSDA adopts a similar objective function to LSDA, but with a difference
that the within-class scatter matrix is included in the objective function. The within-class scatter matrix
𝐒𝑤 — as used in LDA — indicates the compactness of the data point in each class. ILSDA uses the

Table 1. A comparison of the within-class graph and the between-class graph for different subspacelearning methods. (bn: binary weights, hk: heat kernel, knn: k-nearest neighborhood)
Subspace
Learning
Methods
LPP [1]/
OLPP [3]
SLPP [2]/
SOLPP
LSDA [4]
MFA [5]
CMVM [8]
LPMIP [9]
MMDA [11]
CMVM+
[12]
ILSDA [14]
SLPM
(proposed)

The within-class graph
Class
Neighbourhood
Weight
Info

The between-class graph
Class
Neighbourhood
Weight
Info

optional

No

optional

n/a

n/a

n/a

optional

Yes

optional

n/a

n/a

n/a

knn
knn
ϵ-ball
ϵ-ball
n/a

Yes
Yes
No
No
Yes

bn
bn
bn/hk
hk
hk

knn
knn
n/a
ϵ-ball
class centers

Yes
Yes
Yes
No
Yes

bn
bn
bn
hk
hk

knn

Yes

bn

knn

Yes

bn

knn

Yes

hk

knn

Yes

hk

knn

Yes

bn/hk

knn

Yes

hk

scatter matrix to project outliers closer to the class centers under consideration. The objective function
of ILSDA is defined as follows:
max 𝐀𝑇 (𝐏 − 𝛼𝐒𝑤 )𝐀,
(19)
𝑇
where 𝐏 = 𝐗(𝐋𝑏 − 𝐋𝑤 )𝐗 , as defined in the objective function of LSDA. CMVM, unlike other methods
which aim to minimize the within-class spread, intends to maintain the within-class structure for each
class by defining a constraint, i.e. 𝐀𝑇 𝐗𝐋𝑤 𝐗 𝑇 𝐀 = 𝐗𝐋𝑤 𝐗 𝑇 , while increasing the inter-class separability
with the following objective function:
max 𝐀𝑇 𝐗𝐋𝑏 𝐗 𝑇 𝐀,
(20)
where 𝐋𝑤 and 𝐋𝑏 are the within-class and the between-class Laplacian matrices, respectively.

3. Soft locality preserving map
In this section, we introduce the proposed method, Soft Locality Preserving Map (SLPM), with its
formulation and connection to the previous works. Then, we will also describe the local descriptors
used for facial expression recognition in our experiments.
3.1. Formulation of the SLPM
Similar to other manifold-learning algorithms, two graph-matrices, the between-class matrix 𝐖𝑏 and
the within-class matrix 𝐖𝑤 , are constructed to characterize the discriminative information, based on
the locality and class-label information. Given 𝑚 data points {𝒙1 , 𝒙2 , … , 𝒙𝑚 } ∈ ℝ𝐷 and their
𝑤
𝑤
𝑤
corresponding class labels {𝑙(𝒙1 ), 𝑙(𝒙2 ), … , 𝑙(𝒙𝑚 )}, we denote 𝑁𝑤 (𝒙𝑖 , 𝑘𝑤 ) = {𝒙𝑖 1 , 𝒙𝑖 2 , … , 𝒙𝑖 𝑘𝑤 } as
𝑤

𝑤

the set of 𝑘𝑤 nearest neighbours with the same class label as 𝒙𝑖 , i.e. 𝑙(𝒙𝑖 ) = 𝑙(𝒙𝑖 1 ) = 𝑙(𝒙𝑖 2 ) = ⋯ =
𝑤𝑘 𝑤

𝑙(𝒙𝑖

𝑏

𝑏

𝑏𝑘𝑏

), and 𝑁𝑏 (𝒙𝑖 , 𝑘𝑏 ) = {𝒙𝑖 1 , 𝒙𝑖 2 , … , 𝒙𝑖
𝑤

} as the set of its 𝑘𝑏 nearest neighbours with different

class labels from 𝒙𝑖 , i.e. 𝑙(𝒙𝑖 ) ≠ 𝑙(𝒙𝑖 𝑗 ), where 𝑗 = 1, 2, … , 𝑘𝑏 . Then, the inter-class weight matrix 𝐖𝑏
and the intra-class weight matrix 𝐖𝑤 can be defined as below:

Table 2. A comparison of the objective functions used by different subspace methods.
Objective functions

Constraints (s.t.)

LPP [1]/ SLPP [2]

max 𝐀𝑇 𝐗𝐋𝐗 𝑇 𝐀

𝐀𝑇 𝐗𝐃𝐗 𝑇 𝐀 = 𝐈

LSDA [4]

max 𝐀𝑇 𝐗(𝑎𝐋𝑏 + (1 − 𝛼)𝐖𝑤 )𝐗 𝑇 𝐀

𝐀𝑇 𝐗𝐃𝑤 𝐗 𝑇 𝐀 = 𝐈

𝐀

A

min

MFA [5]

𝐀

𝐀𝑇 𝐗𝐋𝑤 𝐗 𝑇 𝐀
𝐀𝑇 𝐗𝐋𝑏 𝐗 𝑇 𝐀

n/a

CMVM [8]

max 𝐀𝑇 𝐗𝐋𝑏 𝐗 𝑇 𝐀

𝐀𝑇 𝐗𝐋𝑤 𝐗 𝑇 𝐀 = 𝐗𝐋𝑤 𝐗 𝑇

LPMIP [9]

max 𝐀𝑇 𝐗(𝑎𝐋𝑏 − (1 − 𝛼)𝐋𝑤 )𝐗 𝑇 𝐀

𝐀𝑇 𝐀 − 𝐈 = 0

MMDA [11]
CMVM+ [12]

𝐀

𝐀𝑇 𝐗𝐋𝑏 𝐗 𝑇 𝐀
𝐀 𝐀𝑇 𝐗𝐋𝑤 𝐗 𝑇 𝐀
𝐀𝑇 𝐗𝐋𝑏 𝐗 𝑇 𝐀
max 𝑇
𝐀 𝐀 𝐗𝐋𝑤 𝐗 𝑇 𝐀

n/a

max

n/a

ILSDA [14]

max 𝐀𝑇 (𝐏 − 𝛼𝐒𝑤 )𝐀 where 𝐏 = 𝐗(𝐋𝑏 − 𝐋𝑤 )𝐗 𝑇

𝐀𝑇 𝐀 = 𝐈

SDM [16]

max 𝐒𝑏 − 𝛼𝐒𝑤

n/a

SLPM

max 𝐀𝑇 (𝐗𝐋𝑏 𝐗 𝑇 − 𝛽𝐗𝐋𝑤 𝐗 𝑇 )𝐀
or
max 𝐀𝑇 𝐗(𝐋𝑏 − 𝛽𝐋𝑤 )𝐗 𝑇 𝐀

𝐀𝑇 𝐀 − 𝐈 = 0

2

−‖𝒙𝑖 − 𝒙𝑗 ‖ ⁄
exp (
𝑏
𝑡) , 𝒙𝑗 ∈ 𝑁𝑏 (𝒙𝑖 , 𝑘𝑏 ),
𝑤𝑖𝑗 = {

(21)

0, otherwise.
2

−‖𝒙𝑖 − 𝒙𝑗 ‖ ⁄
exp (
𝑤
𝑡) , 𝒙𝑗 ∈ 𝑁𝑤 (𝒙𝑖 , 𝑘𝑤 ),
𝑤𝑖𝑗 = {

(22)

0, otherwise.
SLPM is a supervised manifold-learning algorithm, which aims to maximize the between-class
separability, while controlling the within-class spread with a control parameter β used in the objective
function. Consider the problem of creating a subspace, such that data points from different classes, i.e.
represented as edges in 𝐖𝑏 , stay as distant as possible, while data points from the same class, i.e.
represented as edges in 𝐖𝑤 , stay close to each other. To achieve this, two objective functions are defined
as follows:
1
2

2

1

2

max ∑𝑖𝑗(𝒚𝑖 − 𝒚𝑗 ) 𝑤𝑖𝑗𝑏 ,
min 2 ∑𝑖𝑗(𝒚𝑖 − 𝒚𝑗 ) 𝑤𝑖𝑗𝑤 .

(23)
(24)

Eq. (23) ensures that the samples from different classes will stay as far as possible from each other,
while Eq. (24) is to make samples from the same class stay close to each other after the projection.

However, as shown in [30] and [16], small variations in the manifold subspace can lead to overfitting
in training. To overcome this problem, we add the parameter β to control the intra-class spread. Note
that, the method SDM in [16] uses the within-class scatter matrix 𝐒𝑤 – as defined for LDA – to control
the intra-class spread. In our proposed method, we adopt the graph-embedding method, which uses the
locality information about each class, in addition to the class information. Hence, the two objective
functions Eq. (23) and Eq. (24) can be combined as follows:
2

1

2

max 2 (∑𝑖𝑗(𝒚𝑖 − 𝒚𝑗 ) 𝑤𝑖𝑗𝑏 − 𝛽 ∑𝑖𝑗(𝒚𝑖 − 𝒚𝑗 ) 𝑤𝑖𝑗𝑤 )
= max(𝐽𝑏 (𝐀) − 𝛽𝐽𝑤 (𝐀)),

(25)

where 𝐀 is a projection matrix, i.e. 𝐘 = 𝐀T 𝐗 and 𝐗 = [𝒙1 , 𝒙2 , … , 𝒙𝑚 ]. Then, the between-class objective
function 𝐽𝑏 (𝐀) can be reduced to
1

2

𝐽𝑏 (𝐀) = 2 ∑𝑖𝑗(𝒚𝑖 − 𝒚𝑗 ) 𝑤𝑖𝑗𝑏
= 𝐀T 𝐗𝐋𝑏 𝐗 T 𝐀

(26)

where 𝐋𝑏 = 𝐃𝑏 − 𝐖𝑏 is the Laplacian matrix of 𝐖𝑏 and 𝑑𝑏 𝑖𝑖 = ∑𝑗 𝑤𝑖𝑗𝑏 is a diagonal matrix. Similarly,
the within-class objective function 𝐽𝑤 (𝐀) can be written as
1

2

𝐽𝑤 (𝐀) = 2 ∑𝑖𝑗(𝒚𝑖 − 𝒚𝑗 ) 𝑤𝑖𝑗𝑤
= 𝐀T 𝐗𝐋𝑤 𝐗 T 𝐀

(27)

where 𝐋𝑤 = 𝐃𝒘 − 𝐖𝑤 and 𝑑𝑤 𝑖𝑖 = ∑𝑗 𝑤𝑖𝑗𝑤 . If 𝐽𝑤 and 𝐽𝑏 are substituted to Eq. (25), the objective
function becomes as follows:
max 𝐽𝑇 (𝐀) = max(𝐽𝑏 (𝐀) − β𝐽𝑤 (𝐀))
= max(𝐀T 𝐗𝐋𝑏 𝐗 T 𝐀 − β𝐀T 𝐗𝐋𝑤 𝐗 T 𝐀)
= max 𝐀T 𝐗(𝐋𝑏 − β𝐋𝑤 )𝐗 T 𝐀
(28)
which is subject to 𝐀T 𝐀 − 1 = 0 so as to guarantee orthogonality. By using Lagrange multiplier, we
obtain
L(𝐀) = 𝐀T 𝐗(𝐋𝑏 − β𝐋𝑤 )𝐗 T 𝐀 − 𝜆(𝐀T 𝐀 − 𝐈).

(29)

By computing the partial derivative of L(A), the optimal projection matrix A can be obtained, as
follows:
𝜕L(𝐀)
𝜕𝐀

= 𝐗(𝐋𝑏 − β𝐋𝑤 )𝐗 𝑇 𝐀 − 𝜆𝐀 = 0,

(30)

i.e. 𝐗(𝐋𝑏 − β𝐋𝑤 )𝐗 𝑇 𝐀 = 𝜆𝐀. The projection matrix 𝐀 can be obtained by computing the eigenvectors
of 𝐗(𝐋𝑏 − β𝐋𝑤 )𝐗 𝑇 . The columns of A are the d leading eigenvectors, where d is the dimension of the
subspace. LDA, LPP, MFA, and other manifold-learning algorithms, whose objective functions have a
similar structure, lead to a generalized eigenvalue problem. Such methods suffer from the matrixsingularity problem, because the solution involves computing the inverse of a singular matrix. The
proposed objective function is designed in such a way as to overcome this singularity problem.
However, in our algorithm, PCA is still applied to data, so as to reduce its dimensionality and to reduce
noise.
3.2. Intra-class spread
As we have mentioned before, the manifold spread of the different classes can affect the generalizability
of the learned classifier. To control the spread of the classes, the parameter β is adjusted in our proposed

Figure 1. The spread of the respective expression manifolds when the value of β increases
from 1 to 1,000: (1) Anger, (2) Disgust, (3) Fear, (4) Happiness, (5) Sadness, and (6)
Surprise.

method, like SDM. Figure 1 shows the change in the spread of the classes when β increases. We can
see that increasing β will also increase the separability of the data, e.g. the training data is located at
almost the same position in the subspace when 𝛽 = 1,000.
3.3. Relations to other subspace-learning methods
As discussed in Section 2, there have been extensive studies on manifold-learning methods. They share
the same core idea, i.e. using locality and/or label information to define an objective function, so that
the data can be represented in a specific way after projection.
There are two main differences between SLPM and LSDA. First, LSDA defines their objective
function as a subtraction of two objective functions like SLPM. However, LSDA imposes the
constraint 𝐀𝑇 𝐗𝐃𝑤 𝐗 𝑇 𝐀 = 𝐈, which results in a generalized eigenvalue problem. As we mentioned in
Section 2, the generalized eigenvalue problem suffers from the computational cost of calculating an
inverse matrix. SLPM only determines the orthogonal projections, with the constraint 𝐀𝑇 𝐀 − 𝐈 = 0.
Therefore, SLPM can still be computed by eigenvalue decomposition, without requiring computing any

Figure 2. The overall flow of our proposed method.
1. Extract features from face images: 𝐗 𝑑𝑒𝑠𝑐
2. Learn the projection matrix 𝐖𝑝𝑐𝑎 via PCA
3. Construct the within-class graph matrix 𝐖𝑤 and the between-class similarity
matrices 𝐖𝑏
4. Calculate the Laplacian matrices 𝐋𝑤 and 𝐋𝑏
5. Solve the eigenvalue decomposition of 𝐗(𝐋𝑏 − 𝛽𝐋𝑤 )𝐗 T
6. Choose the eigenvectors corresponding to the d largest eigenvalues, 𝐖𝑚𝐿
T
T
7. 𝐘𝑑𝑒𝑠𝑐 = 𝐖𝑚𝐿
𝐖𝑝𝑐𝑎
𝐗 𝑑𝑒𝑠𝑐
8. Add features obtained with either low-intensity images (𝐘𝑙 ) or feature
̅𝑙 ) to form the training data 𝐓𝑙 or 𝐓
̅𝑙 , respectively
generation (𝐘
9. Learn the nearest neighbor classifier
inverse matrix. Second, LSDA finds the neighboring points followed by determining whether the
considered neighboring points are of the same class or of different classes. This may lead to an
unbalanced and unwanted division of neighboring points, simply because of the fact that a sample point
may be surrounded by more samples belonging to the same class than samples with different class
labels. In order not to lose locality information in such a case, SLPM defines two parameters k1 and k2,
which are the numbers of neighboring points belonging to the same class and different classes,
respectively. In other words, the numbers of neighboring points belonging to the same class and
different classes can be controlled.
Both SDM and ILSDA also consider the intra-class spread when defining the objective function.
SDM controls the level of spread by applying a parameter to the within-class scatter matrix 𝐒𝑤 .
However, it only uses the label information about the training data – its scatter matrices do not consider
the local structure of the data. Our proposed SLPM aims to include the locality information by
employing graph embedding in our objective functions. Therefore, SLPM is a graph-based version of
SDM. ILSDA uses both the label and neighborhood information represented in the adjacent matrices,
and also aims to control the spread of the classes. However, ILSDA achieves this by adding the scatter
matrix 𝐒𝑤 to its objective function. In our algorithm, we propose controlling the spread with the withinclass Laplacian matrix 𝐋𝑤 , without adding a separate element to the objective function.

4. Feature descriptors and generation
In this section, we will first present the descriptors used for representing facial images for expression
recognition, then investigate the use of face images with low-intensity and high-intensity expressions
for manifold learning, which represent the corresponding samples at the core and boundary of the
manifold for an expression. After that, we will introduce our proposed feature-generation algorithm.
4.1. Descriptors
Recent research has shown that local features can achieve higher and more robust recognition
performance than by using global features, such as Eigenfaces and Fisherfaces, and intensity values.
Therefore, in order to show the robustness of our proposed method, four different commonly used local
descriptors for facial expression recognition, Local Binary Pattern (LBP) [31, 32], Local Phase
Quantization (LPQ) [33], Pyramid of Histogram of Oriented Gradients (PHOG) [34], and Weber Local
Descriptor (WLD) [35], are considered in our experiments. These descriptors can represent face images,
in terms of different aspects such as intensity, phase, shape, etc., so that they are complementary to each
other. As shown in Figure 2, features are extracted using one of the above-mentioned local descriptors,
followed by the subspace learning with SLPM and a feature-generation method.

4.2. Feature generation
Features in a projected subspace still have a high dimension. A large number of samples for each
expression is necessary in order to represent its corresponding manifold accurately. By generating more
features located near the manifold boundaries, more accurate decision boundaries can then be
determined for accurate facial expression.
Video sequences with face images, changing from neutral expression to a particular expression, are
used for learning. Let 𝑓𝑖,𝜃 denote the frame index of the face image of expression intensity 𝜃 (0 ≤ 𝜃 ≤
1, 0 = neutral expression and 1 = the highest intensity of an expression, i.e. the peak expression) of
the sequence 𝑆𝑖 in a dataset of 𝑚 video sequences. Let 𝒙𝜃𝑖 ∈ ℝ𝐷 be the feature vector extracted from
the 𝑓𝑖,𝜃 -th frame of the sequence 𝑆𝑖 . The frame index 𝑓𝑖,𝜃 can be calculated as follows:
𝑓𝑖,𝜃 = 𝑛𝑖 × 𝜃,
(33)
1
1
1 }
𝐷
{𝒙
where 𝑛𝑖 is the number of frames in the sequence 𝑆𝑖 . Therefore, 1 , 𝒙2 , … , 𝒙𝑚 ∈ ℝ are the feature
vectors extracted from the face images with high-intensity expressions, i.e. the last frames of the m






video sequences. Suppose that {𝒙1 , 𝒙2 , … , 𝒙𝑚 } are the feature vectors extracted from the corresponding
low-intensity images, and the corresponding frame number in the respective video sequences is 𝑓𝑖, . In
our algorithm, we use a different set of  values, where 0.6 ≤  ≤ 0.9, to learn the different expression
manifolds.
4.2.1. Manifold learning with high and low-intensity training samples
A projection matrix A that maps the feature vectors 𝐗1 = [𝒙11 , 𝒙12 , … , 𝒙1𝑚 ] to a new subspace is first
calculated using SLPM. The corresponding projected samples are denoted as 𝐘1 = [𝒚11 , 𝒚12 , … , 𝒚1𝑚 ] ∈
ℝ𝑑 (𝑑 ≪ 𝐷), i.e. 𝒚1𝑖 = 𝐀𝑇 𝒙1𝑖 . Then, the same projection matrix A is used to map the low-intensity










feature vectors 𝐗  = [𝒙1 , 𝒙2 , … , 𝒙𝑚 ], i.e. 𝒚𝑖 = 𝐀𝑇 𝒙𝑖 , which should lie on the boundary of the
Figure 3. The representation of the feature vectors (FV) of happiness (HA) on the CK+
database, after SLPM: (a) HA, i.e. high-intensity expression samples are applied to SLPM,
(b) HA + low intensity FV with  = 0.9, (c) HA + low intensity FV with  = 0.7, (d) HA +
generated FV with 𝜃𝑛𝑒 = 0.9, and (e) HA + generated FV with 𝜃𝑛𝑒 = 0.7.
(b)

(c)

(d)

(e)

(a)

corresponding expression manifold. The high-intensity and low-intensity samples in the subspace form
a training matrix, denoted as 𝐓 , as follows:
𝐓 = [𝐘1 𝐘  ] = [𝐀𝑇 𝐗1 𝐀𝑇 𝐗  ],
(34)
where  (0 ≤  ≤ 1) represents the intensity of the low-intensity images. Figures 3(b) and 3(c)
demonstrate the training data 𝐓 with two different values of  on the CK+ database.
Conventional manifold-learning methods map training samples, irrespective of how strong the
expressing images are, as close as possible after transformation. This results in limited performance in
terms of generalization. In our feature-generation algorithm, the subspace learning method, SLPM, is
first applied to features extracted from high-intensity expressions. Then, features extracted from lowintensity expressions are mapped to the learned subspace. As observed in Figure 4, features extracted
from low-intensity expressions are located farther from the core samples (formed by high-intensity
expressions) and near the boundary of the manifolds after the mapping.
The high-intensity samples are used to determine the centroid of an expression manifold, while those
low-intensity samples are for representing the manifold boundary. The feature vectors are multidimensional, so a large number of low-intensity samples are required to represent the manifold
boundary faithfully. However, only a small number of low-intensity images are available from the
training video sequences. Furthermore, most of the existing expression databases have static images
only. To solve this problem, we propose generating more low-intensity feature vectors for each
expression class, so that the manifold learned for each expression class will be more accurate. In this
paper, we consider the recognition of six facial expressions, i.e. anger, disgust, fear, happiness, sadness,
and surprise. In addition to these facial expressions, we also consider the neutral expression in the
proposed feature-generation method.
Let {𝒙0𝑠1 , 𝒙0𝑠2 , … , 𝒙0𝑠𝑝 } ∈ ℝ𝐷 be the set of feature vectors extracted from neutral face images, where
𝒙0𝑠𝑖 is the feature vector of the neutral face image belonging to the subject 𝑠𝑖 , and 𝑝 is the number of the
subjects in the dataset. The expression images of the subject 𝑠𝑖 are denoted as
𝐗1𝑠𝑖 = [𝒙1𝑠𝑖,1 , 𝒙1𝑠𝑖,2 , … , 𝒙1𝑠𝑖,𝑟 ],
(35)
1
where 𝑟 is the number of expression images belonging to 𝑠𝑖 and 𝒙𝑠𝑖,𝑗 is the feature vector extracted from
the 𝑗th expression image of 𝑠𝑖 . Then, the feature matrix for all the expressions is formed as follows:
𝐗1𝑠 = [𝐗1𝑠1 , 𝐗1𝑠2 , … , 𝐗1𝑠𝑝 ].
(36)
Figure 4. The subspace learned using SLPM, with local descriptors “LPQ”, based on the dataset
named CK+: (a) the mapped features extracted from high-intensity expression images and neutral
face images, (b) the mapped features extracted from high-intensity and low-intensity ( = 0.7)
images, and (c) the mapped features extracted from high-intensity and low-intensity ( =
{0.9, 0.8, 0.7, 0.6, 0.5, 0.4}) images.

(a)

(b)

(c)

The proposed sample-generation method operates in the learned subspace. Thus, the feature vectors
extracted from the neutral face images and the expression images are all mapped to the learned subspace
using the projection matrix A learned from 𝐗1𝑠 , as follows:
𝐘1 = 𝐀𝑇 𝐗1𝑠 = [𝐘𝑠11 , 𝐘𝑠12 , … , 𝐘𝑠1𝑝 ], and

(37)

𝐘 0 = 𝐀𝑇 𝐗 0𝑠 = [𝒚0𝑠1 , 𝒚0𝑠2 , … , 𝒚0𝑠𝑝 ].

(38)

Equations (37) and (38) represent the set of feature vectors of high-intensity expressions and neutral
expressions of all subjects, respectively, in the subspace.
The proposed feature-generation method generates low-intensity feature vectors based on vectorpairs selected from two different sets: (1) vector-pairs from 𝐘𝑠1𝑖 , (2) vector-pairs from 𝐘𝑠1𝑖 and 𝒚0𝑠𝑖 . In the
following sections, we will describe the feature-generation method with respect to two different vectorpairs.
4.2.2. Vector-pairs from 𝒀𝟏𝒔𝒊 and 𝒚𝟎𝒔𝒊
𝜃

𝜃

𝜃

𝜃

̅𝑠 𝑛𝑒 = [𝒚𝑠 𝑛𝑒 , 𝒚𝑠 𝑛𝑒 , … , 𝒚𝑠 𝑛𝑒 ] be the feature matrix of possible low-intensity expressions with
Let 𝐘
𝑖
𝑖,1→0
𝑖,2→0
𝑖,𝑟→0
𝜃

𝑛𝑒
an intensity of 𝜃𝑛𝑒 (0 < 𝜃𝑛𝑒 < 1) belonging to the subject 𝑠𝑖 , where 𝒚𝑠𝑖,𝑗→0
is the corresponding low-

Figure 5. The representation of the sample-generation process based on (a) feature vectors extracted
from high-intensity images and neutral-face images, and (b) feature vectors extracted from highintensity images.
(a)

(b)

.
.
.

.
.
.

intensity feature vector generated using 𝒚1𝑠𝑖,𝑗 and 𝒚0𝑠𝑖 . In the rest of the paper, the arrow “→” indicates
the direction of the feature vectors to be generated, with 0 and 1 being a neutral face image and a face
𝜃

𝑛𝑒
image with the highest intensity, respectively. 𝒚𝑠𝑖,𝑗→0
means that the feature vector is generated in the

direction from 𝒚1𝑠𝑖,𝑗 to 𝒚0𝑠𝑖 where 𝒚1𝑠𝑖,𝑗 is the mapped feature vector extracted from the jth expression
image of 𝑠𝑖 .
A set of feature vectors extracted from an expression video sequence, which starts from a neutralexpression face to the highest intensity of an expression, can be perceived as a path from the reference
center, i.e. the neutral manifold, to a particular expression manifold wherein the distance of an
expression manifold from the center is directly proportional to the intensity of the expression [36].
Therefore, for databases consisting of only static expression images, the feature matrix of possible low𝜃𝑛𝑒
intensity expressions can be obtained by assuming that the relation between the distance from 𝒚𝑠𝑖,𝑗→0
to

𝒚0𝑠𝑖 and the expression intensity is linear. As illustrated in Fig. 5(a), the low-intensity feature vector
𝜃

𝑛𝑒
𝒚𝑠𝑖,𝑗→0
, belonging to 𝑠𝑖 , can be computed as follows:

𝜃

𝑛𝑒
𝒚𝑠𝑖,𝑗→0
= 𝜃𝑛𝑒 𝒚1𝑠𝑖,𝑗 + (1 − 𝜃𝑛𝑒 )𝒚0𝑠𝑖 .

(39)

Figs 3(d) and 3(e) outline the training data with the feature generation using neutral images when
𝜃𝑛𝑒 = 0.9 and 𝜃𝑛𝑒 = 0.7, respectively. As seen in Fig. 3, both the absolute low-intensity feature vectors
and the possible low-intensity feature vectors generated by linear interpolation have a similar structure.
4.2.3. Vector-pairs from 𝒀𝟏𝒔𝒊
The respective expression manifolds can be far from each other in the learned subspace. For this reason,
more features between expression manifolds are also needed. In the previous section, we proposed the
idea that the feature vectors extracted from low-intensity expression images should be distant from the
corresponding manifold centre, thus, this can enhance the generalizability of the learned manifold.
Using a similar idea, more features that are distant from the manifold centres can be generated using
vector-pairs from the feature matrix of high-intensity expressions of the same subject, 𝐘𝑠1𝑖 , as illustrated
𝜃

𝑒𝑥𝑝
in Fig. 5(b). A feature vector 𝒚𝑠𝑖,𝑗→𝑘
, which lies on the line from the jth expression-vector of 𝑠𝑖 , 𝒚1𝑠𝑖,𝑗 , to

the kth expression-vector of 𝑠𝑖 , 𝒚1𝑠𝑖,𝑘 , with a weight 𝜃𝑒𝑥𝑝 (0 < 𝜃𝑒𝑥𝑝 < 1) can be computed as follows:
𝜃

𝑒𝑥𝑝
𝒚𝑠𝑖,𝑗→𝑘
= 𝜃𝑒𝑥𝑝 𝒚1𝑠𝑖,𝑗 + (1 − 𝜃𝑒𝑥𝑝 )𝒚1𝑠𝑖,𝑘 ,

(40)

Suppose that 𝑐𝑗 = 𝑙(𝒚1𝑠𝑖,𝑗 ) and 𝑐𝑘 = 𝑙(𝒚1𝑠𝑖,𝑘 ) are the expression classes of the jth and the kth expression
vectors, respectively, and 𝑛𝑖,𝑐𝑗 and 𝑛𝑖,𝑐𝑘 are the number of expression-vectors of expression classes 𝑐𝑗
and 𝑐𝑘 , respectively, belonging to subject 𝑠𝑖 . Then, a total of 𝑛𝑖,𝑐𝑗 𝑛𝑖,𝑐𝑘 feature vectors can be generated.
The feature matrix consisting of the generated features using the pairs from 𝐘𝑠1𝑖 can be denoted as
follows:
𝜃𝑒𝑥𝑝
𝜃𝑒𝑥𝑝
𝜃𝑒𝑥𝑝
𝜃𝑒𝑥𝑝
𝜃𝑒𝑥𝑝
𝜃𝑒𝑥𝑝
̅𝑠𝜃𝑒𝑥𝑝
𝐘
= [𝒚𝑠𝑖,1→2
, 𝒚𝑠𝑖,1→3
, … , 𝒚𝑠𝑖,1→𝑟
, … , 𝒚𝑠𝑖,𝑟→1
, 𝒚𝑠𝑖,𝑟→2
, … , 𝒚𝑠𝑖,𝑟−1→𝑟
(41)
].
𝑖 ,𝑒𝑥𝑝
̅𝜃 , which is used as a static database, as follows:
The training matrix, 𝐓𝜃 , is updated to 𝐓
̅𝜃 = [𝐘1
𝐓

𝜃𝑛𝑒
̅𝑛𝑒
𝐘

𝜃𝑒𝑥𝑝
̅𝑒𝑥𝑝
],
𝐘

(42)

𝜃𝑒𝑥𝑝
𝜃𝑛𝑒
̅𝑛𝑒
̅𝑠𝜃𝑛𝑒 , 𝐘
̅𝑠𝜃𝑛𝑒 , … , 𝐘
̅𝑠𝜃𝑛𝑒 ] and 𝐘
̅𝑒𝑥𝑝
̅𝑠𝜃𝑒𝑥𝑝 , 𝐘
̅𝑠𝜃𝑒𝑥𝑝 , … , 𝐘
̅𝑠𝜃𝑒𝑥𝑝 ].
where 𝐘
= [𝐘
= [𝐘
1.𝑛𝑒
2,𝑛𝑒
𝑝,𝑛𝑒
1.𝑒𝑥𝑝
2,𝑒𝑥𝑝
𝑝,𝑒𝑥𝑝

In our experiments, we vary the 𝜃𝑛𝑒 and the 𝜃𝑒𝑥𝑝 values from 0.7 to 0.9. Figure 2 lists the overall
flow of the proposed algorithm.
When a feature vector is generated, it is checked whether or not it is closest to its corresponding
manifold class. Furthermore, the feature vectors are generated solely for the pairs of clusters that are
in close proximity to each other in the learned subspace.
Table 3. A comparison of the number of images for different expression
classes in the databases used in our experiments

Disgust
Fear

BAUM-2
80
32
35

CK+
45
59
25

JAFFE
30
29
32

TFEID
34
40
40

Happiness
Sadness

139
68

69
28

31
31

40
39

Surprise
Neutral
TOTAL

83
99
536

82
106
414

30
30
213

36
39
268

Anger

5. Experimental set-up and results
5.1. Experimental setup
In our experiments, four facial-expression databases were used to show the robustness and
performances of the proposed methods: 1) Bahcesehir University Multilingual Affective Face Database
(BAUM-2) [37], 2) Extended Cohn-Kanade (CK+) [38] database, 3) Japanese Female Facial Expression
(JAFFE) [39] database, and 4) Taiwanese Facial Expression Image Database (TFEID) [40].
The BAUM-2 multilingual database consists of short videos extracted from movies. In our
experiments, an image dataset, namely BAUM-2i, consisting of images with peak expressions from the
videos in BAUM-2, is considered. There are 829 face images from 250 subjects in the BAUM-2i static
expression dataset, which express 6 basic emotions. However, only 536 of them, which have their facialfeature points provided, are considered in our experiments. Since the BAUM-2 database was created
by extracting images from movies, the images are close to real-life conditions (i.e. under pose, age, and
illumination variations, etc.), and are more challenging than those in acted databases.
The CK+ dataset contains a total of 593 posed sequences across 123 subjects, of which 304 of the
sequences have been labelled with one of the six discrete emotions, which are anger, disgust, fear,
happiness, sadness, and surprise. Each sequence starts with a neutral face and ends with a frame of peak
Figure 6. The recognition rates of the different subspace methods, with different
local descriptors, based on a combined dataset of BAUM-2, CK+, JAFFE &
TFEID.
0.80
0.75
0.70
0.65

0.60
0.55
0.50
0.45

'NaN'

'LDA'

'MMC'
LBP

'SLPP'
LPQ

'MFA'
PHOG

'LSDA'

'SDM'

'SLPM'

WLD

expression. The last frame of each sequence, and the first frames of the sequences that have unique
subject labels, as well as their landmarks provided, are used for recognition. There are a total of 414
face images. Note that some of the first frames are also discarded because the expressed emotions are
of low intensity. JAFFE consists of 213 images from 10 Japanese females, which express 6 basic
emotions – anger, disgust, fear, happiness, sadness, surprise – and neutral. JAFFE is also a widely used
acted database, which means that it was recorded in a controlled environment. The TFEID database
contains 268 images, with the six basic expressions and the neutral expression, from 40 Taiwanese
subjects. Like CK+ and JAFFE, TFEID is also an acted database.
Each of the above-mentioned databases has its own characteristics. Table 3 shows the number of
images for each expression class for the different databases. Although some of the databases also have

Figure 7. Recognition rates of our proposed method in terms of different
dimensions.
0.9
0.8

Recognition rate

0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0

10

20

30

40

50

60

Dimension

the contempt expression, only the six basic prototypical facial expressions (i.e. anger, disgust, fear,
happiness, sadness, and surprise), as well as the neutral facial expression, are considered in our
experiments. Please note that neutral facial expression has been used only for creating feature vectors
Table 4. The comparison of recognition rates obtained by using lowintensity images with different 𝑙 values on the CK+ database, using the
LPQ feature.
METHODS

CK+

SLPM

94.81%

SLPM +  = 𝟎. 𝟗 95.45%
SLPM +  = 𝟎. 𝟖

94.16%

SLPM +  = 𝟎. 𝟕

93.51%

SLPM +  = 𝟎. 𝟔

91.88%

of low-intensity expressions.
Subspace-learning methods are often applied to feature vectors formed by the pixel intensities of
face images. In our method, features are first extracted using the state-of-the-art local descriptors, and
then a subspace-learning method is applied for manifold learning and dimensionality reduction. The
usual way of using local descriptors is to divide a face image into a number of overlapping or nonoverlapping regions, then extract features from these regions, and finally concatenate them to form a
single feature vector. In this way, local information, as well as spatial information, can be obtained.
Another way of using local descriptors is to consider only the regions that have more salient information
about the considered expression classes. Following this idea, features extracted from the eye and mouth
regions are used in [41], which showed that features extracted from these regions only can achieve
higher recognition rates than those extracted by dividing face images into sub-regions.
In our experiments, face images from the different databases are all scaled to the size of 126189
pixels, with a distance of 64 pixels between the two eyes. To determine the eye and mouth windows,

the facial landmarks, i.e. the eyes and mouth corners, are used. If facial landmarks are not provided for
a database, the required facial-feature points are marked manually. The eye window and the mouth
window are further divided into 12 and 8 sub-regions, respectively. The nearest neighbour classifier
Table 5. The comparison of subspace learning methods on different datasets, with the LPQ
descriptor being used with nearest neighbor classifier.
BAUM-2

CK+

JAFFE

TFEID

MFA

62.01%

93.83%

89.07%

91.70%

SDM

62.01%

93.51%

89.07%

92.58%

SLPM

62.93%

94.81%

90.71%

93.45%

SLPM + 𝜽𝒆𝒙𝒑 = 𝟎. 𝟗 + 𝜽𝒏𝒆 = 𝟎. 𝟗

63.62%

94.81%

91.26%

93.45%

SLPM + 𝜽𝒆𝒙𝒑 = 𝟎. 𝟖 + 𝜽𝒏𝒆 = 𝟎. 𝟖

62.93%

96.10%

91.26%

94.32%

SLPM + 𝜽𝒆𝒙𝒑 = 𝟎. 𝟕 + 𝜽𝒏𝒆 = 𝟎. 𝟕

63.16%

95.13%

91.80%

93.89%

SLPM + 𝜽𝒆𝒙𝒑 = 𝟎. 𝟔 + 𝜽𝒏𝒆 = 𝟎. 𝟔

62.01%

94.16%

90.71%

93.45%

and SVM with linear kernel are used in the experiments.
5.2. Experimental results
In this section, we evaluate the performances of our proposed method, using four different descriptors,
on the four different databases. We also compare our method with four subspace-learning methods, as
well as without using any subspace-learning method.
Firstly, the four acted databases, i.e. BAUM-2, CK+, JAFFE, and TFEID, are combined to form a
single dataset, called COMB4, so that we can better measure the general performances of the different
subspace-learning methods and the descriptors.
Figure 6 shows that MFA, SDM, and SLPM are the three best subspace-learning methods, which
outperform the other subspace-learning methods. The LPQ local descriptor achieves the highest
recognition rates, for the different subspace-learning methods, on COMB4. Therefore, the subspacelearning methods, MFA and SDM, and the local descriptor, LPQ, are chosen to further compare the
performance of the proposed method on each of the individual datasets. In Figure 6, we can also observe
that SDM outperforms most of the subspace-learning methods, except SLPM, because the intra-class
spread is adjustable. Furthermore, SDM is also computationally simpler than the other compared
methods, but it does not incorporate the local geometry of the data. In our proposed method, information
about local structure is incorporated into the objective function. Thus, SLPM can achieve higher
recognition rates than SDM.
Table 6. Comparison of the runtimes (in milliseconds) required by the different subspace learning
methods (MFA, SDM, and SLPM) on different datasets, with the LPQ descriptor used.
BAUM-2 CK+ JAFFE TFEID
MFA

96

69

45

51

SDM

151

133

120

118

SLPM

65

37

23

25

Figure 7 shows the recognition rates of SLPM on COMB4, with the dimensionality of the subspace
varied. The results show that SLPM has converged to its highest recognition rate, when the
dimensionality is lower than 10. In other words, our method is still very effective even at a low
dimensionality. Based on these results, we set the subspace dimensionality at 11 in the rest of the
experiments.
To investigate the effect of the use of images of expression with low intensities, several experiments
have been conducted on the CK+ database. As shown in Table 4, the recognition rate is the highest
when  = 0.9. Table 5 and Table 6 show the recognition rates of the three subspace-learning methods,
MFA, SDM and SLPM, as well as SLPM, using feature generation with different 𝜃𝑒𝑥𝑝 and 𝜃𝑛𝑒 values,
with the LPQ descriptor, on the four different databases using nearest neighbour classifier and SVM
classifier, respectively. It can be found that SLPM achieves the best classification performance again,
when compared to the other methods. The classification performance is further improved by up to 2%,
when feature generation is employed. Furthermore, as observed in Tables 5 and 6, the nearest neighbour
classifier outperforms the SVM classifier in most of the databases. Lastly, additional experiments were
conducted to validate the efficiency of the proposed subspace learning methods. Table 7 tabulates the
runtimes in milliseconds for each of the subspace learning methods. We can see that SLPM is twice as
fast as MFA, which solves the generalized eigenvalue problem instead of calculating eigenvalue
decomposition like SLPM. SDM is much slower than MFA and SLPM.

6. Conclusion
In this paper, we have proposed a subspace-learning method, named Soft Locality Preserving Map
(SLPM), which uses the neighbourhood and class information to construct a projection matrix for
mapping high-dimensional data to a meaningful low-dimensional subspace. The difference between the
within-class and between-class matrices is used to define the objective function, rather than the Fisher
criteria, in order to avoid the singularity problem. Also, a parameter β is added to control the withinclass spread, so that the overfitting problem can be solved. The robustness and the generalizability of
SLPM have been analysed on four different databases, using four different state-of-the-art descriptors
and two different classifiers, and SLPM has been compared with other subspace-learning methods.
Moreover, we have proposed using low-intensity expression images to learn a better manifold for each
expression class. By taking advantage of domain-specific knowledge, we have proposed two methods
of generating new low-intensity features in the subspace. Our experiment results have shown that SLPM
outperforms the other subspace-learning methods, and is a good alternative to performing

Table 7. The comparison of subspace learning methods on different datasets, with the LPQ
descriptor being used with SVM classifier.
BAUM-2

CK+

JAFFE

TFEID

MFA

61.10%

92.21%

91.26%

91.70%

SDM

60.18%

92.21%

89.62%

92.58%

SLPM

63.16%

92.53%

89.62%

93.01%

SLPM + 𝜽𝒆𝒙𝒑 = 𝟎. 𝟗 + 𝜽𝒏𝒆 = 𝟎. 𝟗

63.84%

92.86%

91.26%

94.76%

SLPM + 𝜽𝒆𝒙𝒑 = 𝟎. 𝟖 + 𝜽𝒏𝒆 = 𝟎. 𝟖

62.47%

93.83%

91.26% 95.20%

SLPM + 𝜽𝒆𝒙𝒑 = 𝟎. 𝟕 + 𝜽𝒏𝒆 = 𝟎. 𝟕

62.24%

94.48%

89.07%

94.32%

SLPM + 𝜽𝒆𝒙𝒑 = 𝟎. 𝟔 + 𝜽𝒏𝒆 = 𝟎. 𝟔

61.56%

94.48%

88.52%

94.32%

dimensionality reduction on high-dimensional datasets. Our experiment results, also, have shown that
the proposed feature-generation method can further increase the recognition rates.

References
[1] X. Niyogi, Locality preserving projections, in: Neural information processing systems, MIT, 2004,
pp. 153.
[2] C. Shan, S. Gong, P.W. McOwan, Appearance manifold of facial expression, in: International
Workshop on Human-Computer Interaction, Springer, 2005, pp. 221-230.
[3] D. Cai, X. He, Orthogonal locality preserving indexing, in: Proceedings of the 28th annual
international ACM SIGIR conference on Research and development in information retrieval, ACM,
2005, pp. 3-10.
[4] D. Cai, X. He, K. Zhou, J. Han, H. Bao, Locality Sensitive Discriminant Analysis, in: IJCAI, 2007,
pp. 708-713.
[5] S. Yan, D. Xu, B. Zhang, H.-J. Zhang, Q. Yang, S. Lin, Graph embedding and extensions: a general
framework for dimensionality reduction, IEEE transactions on pattern analysis and machine
intelligence, 29 (2007) 40-51.
[6] H. Hotelling, Analysis of a complex of statistical variables into principal components, Journal of
educational psychology, 24 (1933) 417.
[7] A.M. Martínez, A.C. Kak, Pca versus lda, IEEE transactions on pattern analysis and machine
intelligence, 23 (2001) 228-233.
[8] B. Li, D.-S. Huang, C. Wang, K.-H. Liu, Feature extraction using constrained maximum variance
mapping, Pattern Recognition, 41 (2008) 3287-3294.
[9] H. Wang, S. Chen, Z. Hu, W. Zheng, Locality-preserved maximum information projection, IEEE
Transactions on Neural Networks, 19 (2008) 571-585.
[10] R.A. Fisher, The use of multiple measurements in taxonomic problems, Annals of eugenics, 7
(1936) 179-188.
[11] W. Yang, C. Sun, L. Zhang, A multi-manifold discriminant analysis method for image feature
extraction, Pattern Recognition, 44 (2011) 1649-1657.
[12] Z. Zhao, J. Han, Y. Zhang, L.-f. Bai, A New Supervised Manifold Learning Algorithm, in:
International Conference on Image and Graphics, Springer, 2015, pp. 240-251.
[13] S.J. Raudys, A.K. Jain, Small sample size effects in statistical pattern recognition:
recommendations for practitioners, IEEE Transactions on pattern analysis and machine intelligence, 13
(1991) 252-264.
[14] Y. Yi, B. Zhang, J. Kong, J. Wang, An improved locality sensitive discriminant analysis approach
for feature extraction, Multimedia Tools and Applications, 74 (2015) 85-104.
[15] H. Li, T. Jiang, K. Zhang, Efficient and robust feature extraction by maximum margin criterion,
IEEE Transactions on Neural Networks, 17 (2006) 157-165.
[16] R. Liu, D.F. Gillies, Overfitting in linear feature extraction for classification of high-dimensional
image data, Pattern Recognition, 53 (2016) 73-86.
[17] J.B. Tenenbaum, V. De Silva, J.C. Langford, A global geometric framework for nonlinear
dimensionality reduction, science, 290 (2000) 2319-2323.
[18] S.T. Roweis, L.K. Saul, Nonlinear dimensionality reduction by locally linear embedding, Science,
290 (2000) 2323-2326.
[19] M. Belkin, P. Niyogi, Laplacian eigenmaps and spectral techniques for embedding and clustering,
in: NIPS, 2001, pp. 585-591.
[20] W.K. Wong, H. Zhao, Supervised optimal locality preserving projection, Pattern Recognition, 45
(2012) 186-197.
[21] W.-L. Chao, J.-J. Ding, J.-Z. Liu, Facial expression recognition based on improved local binary
pattern and class-regularized locality preserving projection, Signal Processing, 117 (2015) 1-10.
[22] S. Wang, J. Lu, X. Gu, H. Du, J. Yang, Semi-supervised linear discriminant analysis for dimension
reduction and classification, Pattern Recognition, 57 (2016) 179-189.
[23] J. Zhang, J. Yu, J. You, D. Tao, N. Li, J. Cheng, Data-driven facial animation via semi-supervised
local patch alignment, Pattern Recognition, 57 (2016) 1-20.

[24] R. Ptucha, A. Savakis, Manifold based sparse representation for facial understanding in natural
images, Image and Vision Computing, 31 (2013) 365-378.
[25] B. Kamgar-Parsi, W. Lawson, B. Kamgar-Parsi, Toward development of a face recognition system
for watchlist surveillance, IEEE Transactions on Pattern Analysis and Machine Intelligence, 33 (2011)
1925-1937.
[26] E. Gabrilovich, S. Markovitch, Feature generation for text categorization using world knowledge,
in: IJCAI, 2005, pp. 1048-1053.
[27] P.D. Gader, M.A. Khabou, Automatic feature generation for handwritten digit recognition, IEEE
Transactions on Pattern Analysis and Machine Intelligence, 18 (1996) 1256-1261.
[28] B. Abboud, F. Davoine, M. Dang, Facial expression recognition and synthesis based on an
appearance model, Signal processing: image communication, 19 (2004) 723-740.
[29] J. Yu, B. Bhanu, Evolutionary feature synthesis for facial expression recognition, Pattern
Recognition Letters, 27 (2006) 1289-1298.
[30] H.-W. Kung, Y.-H. Tu, C.-T. Hsu, Dual subspace nonnegative graph embedding for identityindependent expression recognition, IEEE Transactions on Information Forensics and Security, 10
(2015) 626-639.
[31] T. Ojala, M. Pietikäinen, D. Harwood, A comparative study of texture measures with classification
based on featured distributions, Pattern recognition, 29 (1996) 51-59.
[32] C. Shan, S. Gong, P.W. McOwan, Facial expression recognition based on local binary patterns: A
comprehensive study, Image and Vision Computing, 27 (2009) 803-816.
[33] V. Ojansivu, J. Heikkilä, Blur insensitive texture classification using local phase quantization, in:
International conference on image and signal processing, Springer, 2008, pp. 236-243.
[34] A. Bosch, A. Zisserman, X. Munoz, Representing shape with a spatial pyramid kernel, in:
Proceedings of the 6th ACM international conference on Image and video retrieval, ACM, 2007, pp.
401-408.
[35] J. Chen, S. Shan, C. He, G. Zhao, M. Pietikainen, X. Chen, W. Gao, WLD: A robust local image
descriptor, IEEE transactions on pattern analysis and machine intelligence, 32 (2010) 1705-1720.
[36] Y. Chang, C. Hu, M. Turk, Manifold of facial expression, in: AMFG, 2003, pp. 28-35.
[37] C.E. Erdem, C. Turan, Z. Aydin, BAUM-2: a multilingual audio-visual affective face database,
Multimedia Tools and Applications, 74 (2015) 7429-7459.
[38] T. Kanade, J.F. Cohn, Y. Tian, Comprehensive database for facial expression analysis, in:
Automatic Face and Gesture Recognition, 2000. Proceedings. Fourth IEEE International Conference
on, IEEE, 2000, pp. 46-53.
[39] M. Lyons, S. Akamatsu, M. Kamachi, J. Gyoba, Coding facial expressions with gabor wavelets,
in: Automatic Face and Gesture Recognition, 1998. Proceedings. Third IEEE International Conference
on, IEEE, 1998, pp. 200-205.
[40] L.-F. Chen, Y.-S. Yen, Taiwanese facial expression image database, Brain Mapping Laboratory,
Institute of Brain Science, National Yang-Ming University, Taipei, Taiwan, (2007).
[41] C. Turan, K.-M. Lam, Region-based feature fusion for facial-expression recognition, in: 2014
IEEE International Conference on Image Processing (ICIP), IEEE, 2014, pp. 5966-5970.

