Spotting Micro-Expressions on Long Videos Sequences
Jingting Li Catherine Soladie Renaud Sguier Sujing Wang Moi Hoon Yap

[]978-1-7281-0089-0/19/$31.00 2019 IEEE 
[] 

empty
empty

plain
Stages of Reionization as Revealed by the Minkowski Functionals
    Anonymous FG 2019 submission
 Paper ID 

    Received: date / Accepted: date
===============================================================








This paper presents two methods for the first Micro-Expression Spotting Challenge 2019 by evaluating local temporal pattern (LTP) and local binary pattern (LBP) on two most recent databases, i.e. SAMM and CAS(ME)^2. First we propose LTP-ML method as the baseline results for the challenge and then we compare the results with the LBP-χ ^2-distance method. The LTP patterns are extracted by applying PCA in a temporal window  on several facial local regions. The micro-expression sequences are then spotted by a local classification of LTP and a global fusion. The LBP-χ ^2-distance method is to compare the feature difference by calculating χ^2 distance of LBP in a time window, the facial movements are then detected with a threshold.
The performance is evaluated by Leave-One-Subject-Out cross validation.
The overlap frames are used to determine the True Positives and the metric F1-score is used to compare the spotting performance of the databases. The F1-score of LTP-ML result for SAMM and CAS(ME)^2 are 0.0316 and 0.0179, respectively. The results show our proposed LTP-ML method outperformed LBP-χ ^2-distance method in terms of F1-score on both databases. 







§ INTRODUCTION

Facial micro-expression (ME) is a local brief facial movement, which can be triggered under high emotional pressure. The duration is less than 500ms <cit.>. It is a very important non-verbal communication clue, the involuntary nature make it possible to analyze personal genuine emotional state. ME analysis has many potential applications in national security <cit.>, medical care <cit.>, educational psychology <cit.>, and political psychology <cit.>. Due to the growth and importance of MEs, researchers <cit.> have worked collaboratively to solicit the works in this area by conducting challenges in datasets and methods for MEs. This year, the theme of the Second Facial Micro-Expression Grand Challenge has extended to spotting challenges.
The main idea of most methods for ME spotting is to compare the feature differences between the first frame and the other frames in a time window. 
Meanwhile, the feature descriptors used in the state of the art are diverse, to name a few: LBP <cit.>, HOG <cit.>, optical flow <cit.>, integral projection <cit.>, Riesz pyramid <cit.>, and frequency domain <cit.>. Feature differences allow consistent comparisons between frames over a time window of the size of an ME. However, the movements spotted between frames might not be the ME movements, it could be noises, macro-movements and illumination changes. This is why the ability to distinguish MEs from other movements (such as blinking or subtle head movements) remains an open challenge.
Nowadays, methods utilizing machine learning are emerging <cit.>. Furthermore, <cit.> employed deep learning for the first time to perform the ME spotting.
The machine learning process enhances the ability of distinguishing micro-expression from others. However, the spatial patterns are still the primary feature for the classifier. The temporal variation pattern of facial movement in a ME duration has yet to attract sufficient attention. Meanwhile, few articles spotted micro-expression directly from local region. However, the characteristic of that the micro-expression is a local facial movement could help to reduce the false positives. 
In this paper, we spot the micro-expression clips in two recently published databases, and establish the baseline method for ME spotting challenge by using directly a temporal pattern extracted from local region <cit.>. Frames in a ME duration are taken into account to obtain a real temporal and local pattern (LTP), and then the LTPs are recognized by a classifier. Even though the spatial pattern is not studied, the spotted facial motions are differentiated by a fusion process from local to global. This method helps to improve the ability to distinguish ME from other movements. Furthermore, it allows finding the ME spatial local region and the temporal onset index of ME. We compare the results of our proposed LTP-ML method with a LBP approach - LBP-χ ^2-distance by Moilanen et al. <cit.>.
The rest of the paper is organized as follows:
Section <ref> presents the methodology and performance metrics. Section <ref> introduces the result and also shows the detailed experiment results. Section <ref> concludes the paper. 





§ METHODOLOGY


This section describes the benchmark databases, the proposed LTP-ML method, the state-of-the-art LBP method and the performance metrics.


 §.§ Databases

Two most recent long videos spontaneous micro-expression databases, SAMM <cit.> and CAS(ME)^2 <cit.>, are used for ME spotting challenge. Both databases contain long videos, which were recorded in the strictly controlled laboratory environment. Table <ref> compares the differences between these two databases. The notable differences are the resolution and frame rates used in the experimental settings. These are indeed a great challenge for computer vision and machine learning community to produce a robust method worked for both databases, The detailed information of these two databases is presented in the following two subsections. 







  §.§.§ SAMM Long Videos Database

SAMM database consists of a total of 32 subjects and each has 7 videos <cit.>. The average length of videos is 35.3s. The original release of SAMM consists of micro-movement clips labelled in Action Units. Recently, the authors <cit.> introduced objective classes and emotion classes for the database. The recognition challenge will be using the emotional classes from the database as ground truth. The spotting challenge focuses on 79 videos, each contains one/multiple micro-movements, with a total of 159 micro-movements. The index of onset, apex and offset frames of micro-movements were provided as the ground truth. The micro-movements interval is from onset frame to offset frame. In this database, all the micro-movements are labeled. Thus, the spotted frames can indicate not only ME but also other facial movements, such as eye blinks.



  §.§.§ CAS(ME)^2 Database

In the part A of CAS(ME)^2 database <cit.>, there are 22 subjects and 97 long videos. The average duration is 148s. The facial movements are classified as macro- and micro-expressions. The video samples may contain multiple macro or micro facial expressions. The onset, apex, offset index for these expressions are given in the excel file. In addition, the eye blinks are labeled with onset and offset time.



 §.§ LTP-ML: Our Proposed Baseline Method

The baseline method is developed based on the proposed LTP-ML (local temporal pattern-machine learning) method in  <cit.>. The method is extended for long videos by employing a sliding temporal window. The main idea and the modification of LTP-ML method is presented in the following paragraphs.    


  §.§.§ Pre-processing


As the ME is a local facial movement, we analyze ME only on a selection of regions of interest (ROIs). First of all, as shown in Figure <ref>, 84 facial landmarks are tracked in the video sequence by utilizing the Genfacetracker (Dynamixyz). Then the size of ROI square a is determined by the distance L between the left and right inner corners of eyes: a = (1/5) × L. 12 ROIs squares are chosen based on the regions where ME happens most frequently, i.e. the corner of the eyebrows and of the mouth. Two ROIs of nose region are chosen as references because the nose is the most rigid facial region. 


Since the average duration of ME is around 300ms, and the subjects barely moved in one second, the long videos in these two databases are processed by a temporal sliding window W_video whose length is 1s. The overlap is set to 300ms to avoid missing any possible ME movements. This, the video is separated into an ensemble of small sequences [I_1, I_2,...,I_M] by sliding temporal window as shown in Figure <ref>. The positions of 12 chosen ROIs for all frames in one sequence are determined by the detected landmarks of the first frame in the window.






  §.§.§ Feature Extraction

In this part, local temporal patterns (LTPs) <cit.> are analyzed in the local region to distinguish ME from other movements. They are extracted from 12 ROIs respectively in each small sequence. Supposing in sequence I_m (m ≤ M), as illustrated in the lower part of Figure <ref>, PCA is performed on the temporal axis of each ROI sequence to conserve the principal variation at this region. The first two components of each ROI frame are used to analyze the variation pattern of local movement. The PCA process for ROI sequence ROI^m_j (j ≤ 12) in I_m can be presented as in equation <ref>. 



    0.7![
    
       P_1^m,j(x)    ⋯    P_N^m,j(x)
    
       P_1^m,j(y)    ⋯    P_N^m,j(y)   
      ]= Φ× ( 
      [
      
       F_1^m,j(1)    ⋯    F_N^m,j(1)
       ⋱   
    
        F_1^m,j(a^2)    ⋯    F_N^m,j(a^2)   
      ]-I̅ )



where F_n^m,j represents the pixels in one ROI frame, P_n^m,j = [P_n^m,j(x),P_n^m,j(y)] are the first two components of PCA, n is the frame index in this ROI sequence (n ≤ N). Hence, each frame in ROI^m_j can be represented by a point P_n^m,j. 

Then, a sliding window W_ROI is set depending on the average duration of ME (300ms). The distances between the first frame and the other frames in this window are calculated. The window goes through each frame in the sequence  ROI^m_j, and the distance set can be got as [Δ^m_j(n,n+1),Δ^m_j(n,n+w),...,Δ^m_j(n,n+W_ROI-1) ], as shown in Figure <ref>.

The values of distance are then normalized for the entire ROI^m_j to avoid the influence of different movement magnitude in different videos. 
Hence, the feature of frame n for ROI^m_j can be represented as: [CN^m_j, d^m_j(n,n+1),⋯,d^m_j(n,n+W_ROI-1) ], where d^m_j(n,n+1) is the normalized distance value and the CN^m_j is the normalization coefficient. The more detailed deduction process can be found in <cit.>. The feature for one ROI sequence of the entire long video is the concatenation of features of all the separated sequences. 



  §.§.§ Local Classification

As presented in the above paragraph, one video contains 12 feature ensembles from 12 ROI. Li et al. <cit.> showed the LTP patterns are similar for all chosen ROIs for all kinds of ME. The patterns which can represent the ME local movements can be recognized by a local classification. A supervised classification SVM is employed with Leave-One-Subject-Out cross validation. The feature selection and label annotation are presented in <cit.>.  



  §.§.§ Global Fusion

After the LTPs which fit the local ME movement pattern are recognized, a global fusion is processed to eliminate the false positives concerning other movements and true negatives caused by our recognition process. As introduced in <cit.>, there are three steps: a local qualification, a spatial fusion and a merge process. 



 §.§ LBP-χ ^2-distance Method

This method is firstly proposed in  <cit.>. It is the most commonly used method for result comparison for ME spotting.
Based on <cit.> and <cit.>, the configuration of LBP-χ^2 is set as follows: the entire face region is divided into 36 blocks. The overlap rates between blocks on axis X and Y are are 0.2 and 0.3 respectively. LBP features are extracted from blocks with uniform mapping. The radius r is set to r = 3, and the number of neighboring points p is set to p = 8. The χ^2  distances of the each frame are computed in an 2 × L_interval+1 interval.
First of all, the value of LBP-χ^2-distance is compared in the whole long video. However, the method can barely spot any micro-expression intervals, while there are many false positives. This is due to this method spots the maximal movements in the video, and there are some larger movements than ME in both databases. Hence, the entire video is separated into a sub-video set by a sliding window, the setting is the same as the LTP-ML method. For each sub-video, the feature differences are calculated and sorted to find the maximal movement in this short interval. This gives the chance to spot more MEs which could be ignored in entire video comparison. 










 §.§ Performance Metrics

There are three evaluation methods used to compare the performance of the spotting tasks: 1. True positive in one video definition
Supposing there are m micro-expressions in the video, and n intervals are spotted. 
The result of this spotted interval W_spotted is considered as true positive (TP) if it fits the following condition:

    W_spotted∩ W_groundTruth/W_spotted∪ W_groundTruth≥ k

where k is set to 0.5, W_groundTruth represents the micro-expression interval (onset-offset).
Otherwise, the spotted interval is regarded as false positive (FP).
2. Result evaluation in one video
Supposing the number of TP in one video is a (a≤ m and a≤ n), then FP = n-a, false positive (FN) = m-a, the Recall, Precision and F1-score are defined:

    Recall = a/m,  Precision = a/n



    F-score = 2TP/2TP+FP+FN = 2a/m+n

In practical, these metrics might not be suitable for some videos, as there exist the following situations on a single video:

    
  * The test video does not have micro-expression sequences, thus, m=0, the denominator of recall will be zeros. 
    
  * The spotting method does not spot any intervals. The denominator of precision will be zeros since n=0.
    
  * If there are two spotting methods, Method_1 spots p intervals and Method_2 spots q intervals, and p≤ q. Supposing for both methods, the number of true positive is 0, thus the metrics (recall, precision or F1-score) values both equal to zeros. However, in fact, the Method_1 performs better than Method_2. 

Considering these situations, we propose for a single video, we record the result in terms of TP, FP and FN. For performance comparison, we produce a final calculation of other metrics for the entire database.
3. Evaluation for entire database
Supposing in the entire database, there are V videos and M micro-expression sequences, and the method spot N intervals in total. The database could be considered as one long video, thus, the metrics for entire database can be calculated by:

    Recall_D = ∑^V_i=1 a_i/∑^V_i=1 m_i = A/M


    Precision_D = ∑^V_i=1 a_i/∑^V_i=1 n_i = A/N


    F1-score_D = 2× (Recall_D× Precision_D)/Recall_D+Precision_D

The final results by different methods would be evaluated by F1-score since it considers the both recall and precision. 




§ RESULTS AND DISCUSSION


As introduced in Section <ref>, SAMM and CAS(ME)^2 have different frame rates and resolution. Hence, the lengths of sliding window W_video, the overlap size, the interval length of W_ROI and the ROIs size are different for these two databases. Table <ref> lists the experimental parameters.


For CAS(ME)^2 database, there are 97 videos, but only 32 videos contain micro-expressions. Thus, different results are given under two conditions: one is only considering 32 videos which have ME (CAS(ME)^2_ME), another one is to include the entire database (all 97 videos). Since the raw videos in SAMM database are too big to download (700GB), only 79 videos (full frame: 270GB and cropped face: 11GB) were provided for the challenge. In this work, we report the results based on these two versions of SAMM database: one is the cropped videos (SAMM_ME^c) provided by the authors using the method in <cit.>, and the other one is the videos with full frame (SAMM_ME^f). The spotting process is performed only on the downloaded databases.  


 §.§ Experiments Results of LTP-ML Method

After performing the LTP-ML method on these two databases, the spotting results for whole database are listed in Table <ref>.
The F1-score for (SAMM_ME^c) and CAS(ME)^2_ME are 0.0316 and 0.0179 respectively. LTP-ML performs better in SAMM_ME^c than SAMM_ME^f, since the cropped-face process has already aligned the face region in the video, and reduced the influence of irrelevant movements. Concerning the spotting result of CAS(ME)^2, there are more FPs because the video in this database which has no ME may contain macro-expressions. 



 


 §.§ Experiments Results of LBP-χ^2-distance (LBP-χ^2) Method

The result is compared with LBP-χ^2-distance (LBP-χ^2) method. The spotting result is listed in Table <ref>. For CAS(ME)^2_ME, when the threshold for peak selection is set to 0.15, we can get the best result for LBP-χ^2 method, the F1-score is 0.0111. Meanwhile, the highest F1-score of SAMM_ME^c is 0.0055 when the threshold is set to 0.05. 
Compared with LTP-ML method, LBP-χ^2 method is less accurate. LTP-ML method is capable of spotting the subtle movements based on the patterns which represented the temporal pattern variation of ME. Yet, the value of F1-score is low because of the large amounts of FP. Both databases contain noises and irrelevant facial movements, especially for CAS(ME)^2, it is not easy to separate macro-expressions from micro-expressions based on 30fps videos. The ability of distinguishing ME from other movements still need to be enhanced. 




§ CONCLUSIONS


This paper addresses the challenge in spotting ME on long videos sequences using two most recent databases, i.e. SAMM and CAS(ME)^2. We proposed LTP-ML for spotting MEs and provided a set of performance metrics as the guideline for result evaluation on ME spotting. The baseline results of these two databases are provided in this paper. We demonstrate that our proposed method is better than the LBP approach in spotting MEs. Whilst the method was able to produce a reasonable amount of TPs, there are still a huge challenge lays ahead due to the large amount of FPs. Further research will focus on enhancing the ability of distinguishing ME from other facial movements to reduce FPs, including the implementation of deep learning approaches when we have sufficient data.




§ ACKNOWLEDGMENTS


The authors gratefully acknowledge the contribution of the Organisers and Program Committee Members.



IEEEtran



