Objective Classes for Micro-Facial Expression Recognition
Adrian K. Davison Walied Merghani Moi Hoon Yap








^1Centre for Imaging Sciences, University of Manchester, Manchester, United Kingdom

^2Sudan University of Science and Technology, Khartoum, Sudan

^3School of Computing, Mathematics and Digital Technology, Manchester Metropolitan University, Manchester, United Kingdom



computer vision, pattern recognition, feature descriptor





Moi Hoon Yap

M.Yap@mmu.ac.uk



Micro-expressions are brief spontaneous facial expressions that appear on a face when a person conceals an emotion, making them different to normal facial expressions in subtlety and duration. Currently, emotion classes within the CASME II dataset are based on Action Units and self-reports, creating conflicts during machine learning training. We will show that classifying expressions using Action Units, instead of predicted emotion, removes the potential bias of human reporting. The proposed classes are tested using LBP-TOP, HOOF and HOG 3D feature descriptors. The experiments are evaluated on two benchmark FACS coded datasets: CASME II and SAMM. The best result achieves 86.35% accuracy when classifying the proposed 5 classes on CASME II using HOG 3D, outperforming the result of the state-of-the-art 5-class emotional-based classification in CASME II. Results indicate that classification based on Action Units provides an objective method to improve micro-expression recognition.








§ INTRODUCTION



A micro-facial expression is revealed when someone attempts to conceal their true emotion <cit.>. When they consciously realise that a facial expression is occurring, the person may try to suppress the facial expression because showing the emotion may not be appropriate <cit.>. Once the suppression has occurred, the person may mask over the original facial expression and cause a micro-facial expression. In a high-stakes environment, these expressions tend to become more likely as there is more risk to showing the emotion.




Objective Classes for Micro-Facial Expression Recognition
    
Adrian K. Davison^1, Walied Merghani^2 and Moi Hoon Yap^3
    
==============================================================



The duration of a micro-expression is very short and is considered the main feature that distinguishes them from a facial expression <cit.>, with the general standard being a duration of no more than 500 ms <cit.>. Other definitions of speed that have been studied show micro-expressions to last less than 200 ms, this defined by Ekman and Friesen <cit.> as first to describe a micro-expression, 250 ms <cit.>, less than 330 ms <cit.> and less than half a second <cit.>. 

Micro-facial expression analysis is less established and harder to implement due to being less distinct than normal facial expressions. Feature representations, such as Local Binary Patterns (LBP) <cit.>, Histogram of Oriented Gradients (HOG) <cit.> and Histograms of Oriented Optical Flow (HOOF) <cit.>, are commonly used to describe micro-expressions. Although micro-facial expression analysis is very difficult, the popularity in recent years has grown due to the potential applications in security and interrogations <cit.>, healthcare <cit.> and automatic detection in real-world applications where the detection accuracy of humans peaks around 40% <cit.>.

Generally, the process of recognising normal facial expressions involves preprocessing, feature extraction and classification. Micro-expression recognition is not an exception, but the features extracted should be more descriptive due the small movement in micro-expressions compared with normal expressions. One of the biggest problems faced by research in this area is the lack of publicly available datasets, which the success in facial expression recognition <cit.> research largely relies on. Gradually, datasets of spontaneously induced micro-expression have been developed <cit.>, but earlier research was centred around posed datasets <cit.>.

Eliciting spontaneous micro-expression is a real challenge because it can be very difficult to induce the emotions in participants and also get them to conceal them effectively in a lab-controlled environment. Micro-expression datasets need decent ground truth labelling with Action Units (AUs) using the Facial Action Coding System (FACS) <cit.>. FACS objectively assigns AUs to the muscle movements of the face. If any classification of movements take place for micro-facial expressions, it should be done with AUs and not only emotions. Emotion classification requires the context of the situation for an interpreter to make a meaningful interpretation. Most spontaneous micro-expression datasets have FACS ground truth labels and estimated or predicted emotion. These have been annotated by an expert and self-reports written by participants. 

We contend that using AUs to classify micro-expressions gives more accurate results than using predicted emotion categories. By organising the AUs of the two most recent FACS coded state-of-the-art datasets, CASME II <cit.> and SAMM <cit.>, into objective classes, we ensure that the learning methods train on specific muscle movement patterns and therefore increase accuracy. Yan et al. <cit.> also state that it's inappropriate to categorise micro-expressions into emotion categories, and that using FACS AU research to inform the eventual emotional classification.

To date, experiments on micro-expression recognition using categories based purely on AU movements, has not been completed. Additionally, the SAMM dataset was designed for micro-movement analysis rather than recognition. We contribute by completing recognition experiments on the SAMM dataset for the first time with three features previously used for micro-expression analysis: LBP-TOP <cit.>, HOOF <cit.> and HOG 3D <cit.>. Further, the proposed objective classes could inform future research on the importance of objectifying movements of the face.

The remainder of this paper is divided into the following sections; Section 2 discusses the background of two of the FACS coded state-of-the-art datasets developed for micro-expression analysis and the related work in micro-expression recognition; Section 3 describes the methodology; Section 4 presents the results and discusses the effects of applying objective classification to a micro-expression recognition task; Section 6 concludes this paper and discusses future work.




§ BACKGROUND

This section will describe two datasets which are used in the experiments for this paper. A comparative summary of the datasets can be seen in Table <ref>. Previously developed micro-expression recognition systems are also discussed using established features to represent each micro-expression.





 §.§ CASME II

CASME II was developed by Yan et al. <cit.> and refers to Chinese Academy of Sciences Micro-expression Database II, which was preceded by CASME <cit.> with major improvements. All samples in CASME II are spontaneous and dynamic micro-expressions with a high frame rate (200 fps). There are a few frames kept before and after each micro-expression to make it suitable for detection experiments. The resolution of samples is 640×480 pixels for recording which saved as MJPEG and resolution about 280×340 pixels for cropped facial area. The participants' facial expressions were elicited in a well-controlled laboratory environment.








The dataset contains 255 micro-expressions (gathered from 35 participants) and were selected from nearly 3000 facial movements and have been labelled with AUs based on FACS. Only 247 movements were used in the original experiments on CASME II <cit.>. The inter-coder reliability of the FACS codes within the dataset is 0.846. Flickering light was avoided in the recordings and highlights to regions of the face was reduced. However, there were some limitations: firstly, the materials used for eliciting micro-expression are video episodes which can have different meanings to different people, for example eating worms may not always disgust someone. Secondly, micro-expressions are elicited under one specific lab situation. There was some difficulty in eliciting some types of facial expressions in laboratory situations, such as sadness.






When analysing the FACS codes of the CASME II dataset, it was found that there are many conflicts to the coded AUs and the estimated emotions. These inconsistencies do not help when attempting to train distinct machine learning classes, and adds further justification for the proposed introduction of new classes based on AUs only.

For example, Subject 11 with the micro-expression clip filename of ‘EP19_03f’, was coded as an AU4 in the ‘others’  estimated emotion category (shown in Fig. <ref>). However, Subject 26 with the micro-expression clip filename of ‘EP18_50’, was also coded with AU4 but in the ‘disgust’  estimated emotion category (shown in Fig. <ref>). As can be seen in the apex frame (centre image) of both Fig. <ref> and  <ref>, AU4, the lowering of the brow, is present. Having the same movement in different categories is likely to have an effect on any training stage of machine learning.



 §.§ SAMM

The Spontaneous Actions and Micro-Movements (SAMM) <cit.> dataset is the first high-resolution dataset of 159 micro-movements induced spontaneously with the largest variability in demographics. To obtain a wide variety of emotional responses, the dataset was created to be as diverse as possible. A total of 32 participants were recruited for the experiment with a mean age of 33.24 years (SD: 11.32, ages between 19 and 57), and an even gender split of 16 male and female participants. The inter-coder reliability of the FACS codes within the dataset is 0.82, and was calculated by using a slightly modified version of the inter-reliability formula, found in the FACS Investigator's Guide <cit.>, to account for three coders rather than two.

The inducement procedure was based on the 7 basic emotions <cit.> and recorded at 200 fps. As part of the experimental design, each video stimuli was tailored to each participant, rather than obtaining self reports during or after the experiment. This allowed for particular videos to be chosen and shown to participants for optimal inducement potential. The experiment comprised of 7 stimuli used to induce emotion in the participants who were told to suppress their emotions so that micro-facial movements might occur. To increase the chance of this happening, a prize of 50 was offered to the participant that could hide their emotion the best, therefore introducing a high-stakes situation <cit.>. Each participant completed a questionnaire prior to the experiment so that the stimuli could be tailored to each individual to increase the chances of emotional arousal.









The SAMM dataset was originally designed to investigate micro-facial movements by analysing muscle movements of the face rather than recognising distinct classes <cit.>. We are the first to categorise SAMM based on the FACS AUs and then use these categories for micro-facial expression recognition.


 §.§ Related Work


Currently, there are three features which many micro-expression recognition approaches rely on: Local Binary Patterns (LBP), Histogram of Oriented Gradients (HOG) and Histogram of Oriented Optical Flow (HOOF) based. We will discuss different methods that use these features in recent work on micro-expression recognition. Further, specific important micro-expression research is discussed.


As an extension to the original Local Binary Pattern (LBP) <cit.> operator, Local Binary Patterns on Three Orthogonal Planes (LBP-TOP) was proposed by Zhao et al. <cit.> demonstrated to be effective for dynamic texture and facial expression analysis in the spatial-temporal domain. Since video sequence of time length T, usually it can be thought as a stack of XY planes along the time axis T, but also it can be thought as three planes XY, XT and YT. These provide information about space and time transition. The basic idea of LBP-TOP is similar to LBP, the difference being that LBP-TOP extracts features from all three planes which will be combined in into a single feature vector.

Yan et al. <cit.> carried out the first micro-expression recognition experiment on the CASME II dataset. LBP-TOP <cit.> was used to extract the features and Support Vector Machine (SVM) <cit.> was employed as the classifier. The radii varied from 1 to 4 for X and Y, and from 2 to 4 for T (T=1 was not considered due to little change between two neighbouring frames at 200 fps), with classification occurring between five main categories of emotions provided in this experiment (happiness, disgust, surprise, repression and others).

Davison et al. <cit.> used the LBP-TOP feature to differentiate between movements and neutral sequences, attempting to avoid bias when classifying with an SVM.

The performance of <cit.> on recognising micro-expressions in 5-classes with LBP-TOP features extraction, achieved a best result of 63.41% accuracy, using leave-one-out cross-validation. This result is an average for recent micro-expression recognition research, and is likely due to the way micro-expressions are categorised. Of the 5-class in the CASME II dataset, 102 were classed as ‘others’, which denoted movements not suited for the other categories but related to emotion. The next highest category was ‘disgust’  with 60 movements, showing that the ‘others’  class made the categorisation imbalanced. Further, the categorisation was not based solely on AUs due to micro-expressions being short in duration and low in intensity, but also based on the participant's self-reporting. By classifying micro-expressions in this way, features are unlikely to exhibit a pattern, and therefore perform poorly during the recognition stage as can be seen in the other performance results. For example, in  <cit.>, the highest results is 63.41%, which is still relatively low.

More recently, LBP-TOP was used as a base feature for micro-expression recognition with integral projection <cit.>. These representations attempt to improve discrimination between micro-expression classes and therefore improve recognition rates.

Polikovsky et al. <cit.> used a 3D gradient histogram descriptor (HOG 3D) to recognise posed micro-facial expressions from high-speed videos. The paper used manually marked up areas that are relevant to FACS-based movement so that unnecessary parts of the face are left out. This does means that the method of classifying movement in these subjectively selected areas is time-consuming and would not suit a real-time application like interrogation. The spatio-temporal domain is explored highlighting the importance of the temporal plane in micro-expressions, however the bin selection for the XY plane is 8 and the XT, YT planes have been set to 12. The number of bins selected represents the different directions of movement in each plane.


For HOOF-based methods, a Main Direction Mean Optical Flow (MDMO) feature was proposed by Liu et al. <cit.> for micro-facial expression recognition using SVM as a classifier. The method of detection also uses 36 regions, partitioned using 66 facial points on the face, to isolate local areas for analysis, but keeping the feature vector small for computational efficiency. The best result on the CASME II dataset was 67.37% using leave-one-subject-out cross validation.

The basic HOOF descriptor was also used by Li et al. <cit.> as a comparative feature when spotting micro-expressions and then performing recognition. This is the first automatic micro-expression system which can spot and recognise micro-expressions from spontaneous video data, and be comparable to human performance.


Using Robust Principal Component Analysis (RPCA) <cit.>, Wang et al. <cit.> extract the sparse information from micro-expression data, and then use Local Spatiotemporal Directional Features, based on LBP-TOPs dynamic features, to extract the subtle motion on the face from 16 ROIs of local importance to facial expression motion.

A novel colour space model was created named Tensor Independent Color Space (TICS), that helps recognise micro-expressions <cit.>. By extracting the LBP-TOP features of independent colour components, micro-expression clips can be better recognised than the RGB space.

Huang et al. <cit.> proposed Spatio-Temporal Completed Local Quantization Patterns (STCLQP), which extracts the sign, magnitude and orientation of the micro-expression data, then an efficient vector quantization and codebook selection are developed in both the appearance
and temporal domains for generalising classical pattern types. Finally, using the developed codebooks, spatio-temporal features of sign, magnitude and orientation components are extracted and fused, with experiments being run on SMIC, CASME and CASME II.

By exploiting the sparsity in the spatial and temporal domains of micro-expressions, a Sparse Tensor Canonical Correlation Analysis was proposed for micro-expression characteristics <cit.>. This method reduces the dimensionality of micro-expression data and enhances LBP coding to find a subspace to maximise the correlation between micro-expression data and their corresponding LBP code.

Liong et al. <cit.> investigate the use of only two frames from a micro-expression clip: the onset and the apex frame. By only using a couple of frames, a good accuracy is achieved when using the proposed Bi-Weighted Oriented Optical Flow feature to encode the expressiveness of the apex frame.

As micro-movements on the face are heavily affected by the global movements of a person's head, Xu et al. <cit.> propose a Facial Dynamics Map to distinguish between what is a micro-expression and what would be classed as a non-micro-expression. The facial surface movement between adjacent frames is predicted using optical flow. The movements are then extracted in a coarse-to-fine manner, indicating different levels of facial dynamics. This step is used to differentiate micro-movements from anything else. Finally, an SVM is used for both identification and categorisation.

Wang et al. <cit.> recently proposed a Main Directional Maximal Difference (MDMD) method that uses the magnitude maximal difference in the main direction of optical flow features to find when facial movements occur. These movements can be used for both micro-expressions and macro-expressions to find the onset, apex and offset of a movement within the context of each examined clip.



§ METHODOLOGY


To overcome the conflicting classes in CASME II, we restructure the classes around the AUs that have been FACS coded. Using EMFACS <cit.>, a list of AUs and combinations are proposed for a fair categorisation of the SAMM <cit.> and CASME II <cit.> datasets. Categorising in this way removes the bias of human reporting and relies on the ground truth movement data, feature representation and recognition technique for each micro-expression clip. Table <ref> shows 7 classes and the corresponding AUs that have been assigned to that class. Classes I-VI are linked with happiness, surprise, anger, disgust, sadness and fear. Class VII relates to contempt and other AUs that have no emotional link in EMFACS <cit.>. It should be noted that the classes do not directly correlate to being these emotions, however the links used are informed from previous research <cit.>.



Each movement in both datasets were classified based on the AU categories of Table <ref>, with the resulting frequency of movements being shown in Table <ref>.



Micro-expression recognition experiments are run on two datasets: CASME II and SAMM. For this experiment, three types of feature representations are extracted from a sequence of grey images which represent the micro-expressions. These image sequences are divided into 5×5 blocks that are non-overlapping. The LBP-TOP features <cit.> radii parameters for X, Y and T are set to 1, 1 and 4 respectively and all neighbours in three planes set to 4. The HOG 3D <cit.> and HOOF <cit.> features are set to the parameters described in the original implementations.


Sequential Minimal Optimization (SMO) <cit.> is used in the classification phase with 10-fold cross validation and leave-one-subject-out (LOSO) to classify between I-V, I-VI and I-VII classes. SMO is a fast algorithm for training SVMs, and provide a solution to solving very large quadratic programming (QP) problems, which are required to train SVMs. SMO avoid time-consuming QP calculations by breaking them down into smaller pieces. Doing this allows for the classification task to be completed much faster than using traditional SVMs <cit.>.



§ RESULTS







Evidence to support the proposed AU-based categories can be seen in the confusion matrix in Fig. <ref>. A high proportion of micro-expressions have been classified as ‘others’, for example 28.95% of the ‘happiness’ and 28.57% of the ‘disgust’ categories are classified as ‘others’ respectively. The original chosen emotions, including many placed in the ‘others’  category, leads to a lot of conflict at the recognition stage. It should be noted that the CASME II dataset <cit.> included self-reporting, which adds another layer of complexity during classification.

The proposed classes I-V classification results using LBP-TOP can be seen in the confusion matrix in Fig. <ref>. In contrast, the classification rates are more stable and outperforming the original classes overall. The results are by no means perfect, however it shows that the most logical direction is to use objective classes based on AUs rather than estimated emotion categories. Further investigation using an objective selection of FACS-based regions <cit.> supports this with AUC results for detecting relevant movements to be 0.7512 and 0.7261 on SAMM and CASME II, respectively.







Table <ref> shows the experimental results on CASME II with each result metric being a weighted average calculation to account for imbalanced numbers within classes. Each experiment was completed for each feature and within the original classes defined in <cit.> and the proposed classes. Both the 10-fold cross-validation results and leave-one-subject-out (LOSO) are shown.

The top performing feature achieves a weighted accuracy score of 86.35% for the HOG 3D feature in the proposed class I-V. This shows a large improvement over the original classes which achieved 80.93% for the same feature. Using LOSO, the results were comparable with the original classes. The highest accuracy was 76.60% from the HOOF feature, in the proposed I-VII classes. For the CASME II dataset results, using LBP-TOP and 10-fold cross-validation, the original method outperformed the classes I-VI and I-VII. In addition, for HOG3D LOSO, the original method outperforms in class I-VII when using F-measure as a measurement.

The experiment based on the same conditions were then repeated for SAMM and can be seen in Table <ref>. Overall the recognition rates were good for SAMM, with the best result achieving an accuracy of 81.93% using LBP-TOP in I-VI classes for 10-fold cross validation. The best result using LOSO was from the HOG 3D feature, in the proposed I-VII classes and achieved 63.93%, however due to the lower amount of micro-expressions within the SAMM dataset compared with CASME II, the LOSO results were lower.

Some results show that using LOSO, HOOF outperforms in CASME II while HOG3D outperforms in SAMM and in CASME II using LOSO, the HOOF feature achieves a higher accuracy for classes I-VII over I-VI, but not for the F-measure metric. Explanations of this comes down to the data, and how large some variations of the settings, such as resolution and capture methods, are set. The imbalance of data, specifically the low amounts of micro-expression data, can skew LOSO results with low amounts of testing and training. This shows how using LOSO for micro-expression recognition is difficult to quantify with a fair amount of significance. Further data collection of spontaneous micro-expressions is required to rectify this.




§ CONCLUSION

We show that restructuring micro-expression classes objectively around the AUs, recognition results outperform the state-of-the-art, emotion-based classification approaches. As micro-expressions are so subtle, the best way to categorise is objectively as possible, so using AU codes is the most logical. Categorising using a combination of AUs and self-reports <cit.> can cause many conflicts when training a machine learning method. Further, dataset imbalances can be very detrimental to machine learning algorithms, and this is further emphasised with the relatively low amount of movements in both datasets. Future work will look into the effect of using more modern features, with AUs classification to improve on the recognition accuracy. This could include the MDMO feature <cit.>, local wrinkle feature <cit.> and the feature extraction methods described by Wang et al. <cit.>.

Further work can be done to improve micro-facial expression datasets. Firstly, more datasets or expanding previous sets would be a simple improvement that can help move the research forward faster. Secondly, a standard procedure on how to maximise the amount of micro-movements induced spontaneously in laboratory controlled experiments would be beneficial. If collaboration between established datasets and researchers from psychology occurred, dataset creation would be more consistent.

Deep learning has emerged as a new area of machine learning research <cit.>, and micro-expression analysis has yet to exploit this trend. Unfortunately, the amount of high-quality spontaneous micro-expression data is low and deep learning requires a large amount of data to work well <cit.>. Many video-based datasets previously used have over 10,000 video samples <cit.> and even over 1 million actions extracted from YouTube videos <cit.>. A real effort to gather spontaneous micro-expression data is required for deep learning approaches to be effective in the future.



1pc



Data available from: http://www2.docm.mmu.ac.uk/STAFF/m.yap/dataset.php

A.K. Davison carried out the design of the study, the re-classification of the Action Units grouping and drafted the manuscript (The tasks was completed when A.K. Davison was in Manchester Metropolitan University). W. Merghani conducted the experiments, analysed the data and drafted the manuscript. M.H.Yap designed the study, developed the theory, assisted development and testing and edited the manuscript. All the authors have read and approved this version of the manuscript.

We declare we have no competing interests.

This work was completed in Manchester Metropolitan University on a "Future Research Leaders Programme" awarded to M.H. Yap. M.H.Yap is a Royal Society Industry Fellow. 

























RS 
 

