{'fraction_non_alphanumeric': 0.032533550223668155, 'fraction_numerical': 0.011928968415344991, 'mean_word_length': 4.9326497788500205, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Emotion Recognition With Knowledge Graph Based on Electrodermal Activity

Electrodermal activity (EDA) sensor is emerging non-invasive equipment in affect detection research, which is used to measure electrical activities of the skin. Knowledge graphs are an effective way to learn representation from data. However, few studies analyzed the effect of knowledge-related graph features with physiological signals when subjects are in non-similar mental states. In this paper, we propose a model using deep learning techniques to classify the emotional responses of individuals acquired from physiological datasets. We aim to improve the execution of emotion recognition based on EDA signals. The proposed framework is based on observed gender and age information as embedding feature vectors. We also extract time and frequency EDA features in line with cognitive studies. We then introduce a sophisticated weighted feature fusion method that combines knowledge embedding feature vectors and statistical feature (SF) vectors for emotional state classification. We finally utilize deep neural networks to optimize our approach. Results obtained indicated that the correct combination of Gender-Age Relation Graph (GARG) and SF vectors improve the performance of the valence-arousal emotion recognition system by 4 and 5% on PAFEW and 3 and 2% on DEAP datasets.


INTRODUCTION
Emotions are vital for humans because they influence affective and cognitive processes (Sreeshakthy and Preethi, 2016). Emotion recognition  over the years has received large attention from academic researchers and industrial organizations and has been applied in numerous sectors including transportation (De Nadai et al., 2016), mental health (Guo et al., 2013), robotics (Tsiourti et al., 2019), and person identification (Wilaiprasitporn et al., 2020). Emotions are accompanied by physical and psychological reactions when an external or internal input is introduced. Representative methods for emotion recognition can be categorized into two sectors, physical-based and physiological-based. Physical-based signals include facial expressions (Huang et al., 2019), body gestures (Reed et al., 2020), audio (Singh et al., 2019), etc. All these signals are easy to collect and show good emotion recognition performance. However, the dependability of the physical-based data can be equivocal as it can be easy for people to intentionally alter their reactions, which results in a false reflection of their real emotions. Physiological-based signals on the other hand include electroencephalogram (EEG), electrocardiogram (ECG), electromyogram (EMG), electrodermal activity (EDA)/galvanic skin response (GSR), blood volume pulse (BVP), temperature, photoplethysmography (PPG), respiration (RSP), and so on (Chen et al., 2021). While people may know the reasons why their signals are being collected, physiological signals relating to emotional states are hard to manipulate because they are controlled by the autonomic nervous system (ANS) (Shu et al., 2018). Compared to physical-based methods, emotion recognition based on physiological signal is less sensitive to societal and cultural differences amongst users and are acquired in natural emotional states (Betella et al., 2014). Physiological signals also perform better in detecting sounds than on beeps as they primarily reflect emotional states (Stuldreher et al., 2020).
Emotion should be well defined and approached in a quantifiable manner. Psychologists model emotions in two ways: discrete and multi-dimensional continuous (Liu et al., 2018;Mano et al., 2019;Yao et al., 2020). Discrete emotion theory claims that there is a small number of core emotions. They are usually limited and can barely differentiate heterogeneous emotions and composite mental states. Multi-dimensional emotion model on the other hand finds correlation among different discrete emotions which correspond to a higher level of a particular emotion. Valence-Arousal (V-A) space model has been widely used in affective computing research as a type of multi-dimensional continuous emotion model. Valence stands for a negative to a positive level of emotion that ranges from unpleasant to pleasant feelings. Arousal stands for low to high level of emotion that ranges from drowsiness to intense human excitement. The combination of valence and arousal space model enables the creation of continuous emotion models that captures both moderate and complex emotions needed for building accurate emotion recognition systems . This study adopts the V-A space model for final emotion state classification.
The electrodermal activity also known as galvanic skin response is a non-stationary signal. Figure 1 shows the EDA signal of participant #39 from the PAFEW dataset. Skin conductance response occurs when there is an induction of stimuli and may occur many times in short instances as shown in the figure. Portable low-cost EDA sensors (Milstein and Gordon, 2020) have been developed and applied in a physiological research context as signals become easier to collect. Prior works have connected electrodermal activities with biofeedback in diffusing brain activation as its cost effective and easy to apply treatment options are with no known risk (Gebauer et al., 2014). Researchers have begun combining and comparing the sufficiency of traditional machine learning and deep learning algorithms to predict users' mental states from EDA (Ayata et al., 2017).
Knowledge graph (KG) (Hogan et al., 2021) involves the acquisition and integration of information from data, creating relevant entities and relations, and predicting a meaningful output. As a new type of representation, KG has gained a lot of attention in the field of cybersecurity, natural language processing, recommendation systems, and human cognition (Jia et al., 2018;Guan et al., 2019;Chen et al., 2020;Qin et al., 2020;Yang et al., 2020). Wang  represented cognitive relations between different emotion types using a knowledge graph. Yu et al. (2020) studied a framework made up of non-contact intelligent systems, knowledge modeling, and reasoning to represent heart rate (HR) and facial features to predict emotional states. Farashi and Khosrowabadi (2020) used the knowledge of minimum spanning tree (MST) graph features derived from computational methods to classify emotional states. Wenbo et al. used knowledge embedding in a deep relational network to capture and learn relationships between cartoon, sketch, and caricature face recognition (Zheng et al., 2020). Gender and age are important factors that affect human emotion. To the best of our knowledge, there is no existing work on emotion recognition that attempts to combine gender and age with EDA/GSR statistical features (SF) to accurately model emotional states. In this paper, we propose an effective knowledge embedding graph model based on observed participants' gender and age to capture the relations between given entities. This has been lacking in EDA-Based Emotion Recognition Systems where researchers only focus on time, frequency, and time-frequency statistical features (SFs). We further propose a sophisticated feature fusion technique that exploits the knowledge embedding vectors as a weight to SFs. We then utilized a deep neural network to capture relevant complex information from participants' age and gender and predict emotional states.
The main contributions of this article is summarized as follows: 1) We propose an effective knowledge embedding graph model based on observed participants' gender and age to capture the relations between given entities.
2) We propose a sophisticated feature fusion technique that exploits the knowledge embedding vectors as a weight to SFs.
3) The proposed model shows better recognition accuracy when compared to other methods. The overall implementation framework of this paper is illustrated in Figure 2. FIGURE 2 | Overall implementation of the statistical features (SFs) and the observed knowledge graph features in accordance with the EDA signal. First, the network is used to extract EDA time and frequency SFs, S. We then construct a Gender-Age Relation graph (GARG) in a form of triples to learn relations and get embedding feature vectors. A transformation matrix P is introduced to exploit the graph embeddings. The fused feature F is gotten by combining the SFs and knowledge embeddingsĜ after dimension reduction for training, validation, and testing. Finally, a fully connected neural layer and SOFTMAX is used for emotion state classification within the valence-arousal scale.
The rest of the paper is organized as follows: Section 2 presents related background and approaches to EDA signals and knowledge graph and also presents our novel methodology and algorithms. Section 3 reports the experimental results. Section 4 discusses the novel approach and compares them with previous works. Finally, Section 5 concludes the study.

Related Background
Electrodermal activity-based emotion recognition has become popular in affective computing and cognitive developmental studies. Knowledge graphs have also been applied to embed entities in attempts to get more information from available data. Both ideas are frequently used nowadays in a variety of contexts and researchers have attempted to design frameworks to accurately solve today's problems.

EDA-Based Emotion Recognition
Electrodermal activity is a cheaper, easily collected physiological signal that reflects internal reactions to yield exhilaration. Deep learning algorithms (Yang et al., 2022) have become a catalyst in emotion recognition to capture time, frequency, and time-frequency information in EDA signals. Multilayer Perceptron (MLP), Convolutional neural network (CNN), Deep Belief Networks (DBN), Attention-Long Short Term Memory (A-LSTM) are often used. These algorithms with processors in the suggestive connection between neurons and layers can learn to extract relevant features for a reference task. In the study by Hassan et al. (2019), signals from EDA, Zygomaticus Electromyography (zEMG), and Photoplethysmogram (PPG) are fused using DBN for in-depth feature extraction to gear up a feature fusion vector. The vector is used to classify five discrete basic emotions, relaxed, happy, sad, disgust, and neutral. The study also used Fine Gaussian Support Vector Machine (FGSVM) together with radial basis kernel function to help classify the nonlinearity in the human emotion classification. Song et al. also proposed an attention-long short term memory (A-LSTM) to extract discriminative features from EDA and other physiological signals to strengthen sequence effectiveness . These studies did not use the gender and age information of participants as features to validate their results. Also, the performance of their approach did not quantify the contribution of the EDA signal but proves deep learning is effective in extracting emotional features.

Knowledge Graph
An effective way to learn representation from data is through embedding a KG into a vector space while maintaining its properties . It involves knowledge triples h, r, t compiled of two entities h and t and a relation r. Face recognition studies have tried to study representational gaps using deep learning techniques to derive useful facial information . KG embedding can also be used to find vector representation of known graph entities by regarding them as a translation of entities in space. Human knowledge allows for a formal understanding of the world (Ji et al., 2022). For cognition and human level intelligence, knowledge graphs that represent structural relations between entities have become an increasingly popular research direction. The performance of these approaches shows that when KG is integrated, model performance is enhanced. Graph convolutional network has been studied for drug prediction in computational medicine (Nguyen et al., 2022). KG has been used for image processing and facial recognition but has never been used with physiological signals for emotion recognition. We attempt to address this issue.

The PAFEW Dataset
For the PAFEW dataset , there are 57 healthy students that participated in the experiment and each view about 80-90 video clips in all 7 emotion categories. Participants are aged between 18 and 39 years and were subjectively rated to keep them focused. The dataset is summarized in Table 1. Participants watched videos with one label continuously. This encourages them to be fully dissolved into one emotion before moving on to watching another video with a different label. The initial dataset was collected in a multi-class fashion (shown in Figure 3). From the figure, we can conclude that the PAFEW dataset is highly overlapped resulting in class imbalance. To tackle the imbalance issue, we subset target labels into the valence and arousal dimensions (shown in Figure 4). This is to more effectively capture both moderate and complex emotions. For example, fear and anger are two separate emotions which may differ from person to person but fall under high arousal and negative valence. Also, to be able to assess the performance metrics, we defined the total number of samples per emotion used to train and test the performance of the proposed method (seen in Table 2). Overall, we have a total of 3,554 data points.

DEAP Dataset
The DEAP dataset (Koelstra et al., 2012) is made up of 32 participants that watched 40 video clips while their physiological signals and facial expressions are recorded. Participants are aged between 19 and 37 (mean age 26.9). Participants' rating was also collected. We used the GSR data/channel to design an emotion recognition system. A summary of the DEAP dataset is shown in Table 3.

Methodology
In this section, we first illustrate the Gender-Age Relation Graph (GARG) and SF structures that are used in conducting the experiment. We also describe the weighted fusion representation learning which is the keystone of our solution.

GARG Feature Learning Structure
Knowledge bases represent data by a directed graph with labeled edges (relations) between nodes (entities). We attempt to solve the problem of valence-arousal emotion recognition. Usually, the directed graph is represented by triples in the form (subject, predicate, and object). In our case, we use the gender and age information of a participant to construct the graph, so we have triples like (Participant, AgeIs, 28) and (Participant, GenderIs, F/M).
Formally speaking, we define our knowledge graph G = E × R × E: • E: a set of entities; • R: a set of relations.
Let E = (e 1 , e 2 , ..., e N e ) represent the set of entities in the knowledge graph and let R = (r 1 , r 2 , ..., r N r ) represent the set of all relation types. We represent each triple as x a,b,c = (e a , r c , e b ) and model its presence with a binary random variable y a,b,c ∈ {0, 1} indicating whether a triple exists or not. In this paper, we extract latent embedding features by maximizing the existential possibility of existing triples. Following (Trouillon et al., 2016), we use complex embedding features. The complex embeddings of the subject, object, and their relation are denoted by e s , e o , and w r ∈ C k , respectively. The scoring function of a triple is defined by: where Re(·) and Im(·), respectively, extract the real and imaginary parts of a complex vector, and denotes the parameters corresponding to the model.
It is expected to score the correct triples (e 1 , r, e 2 ) higher than incorrect triples (e ′ , r, e 2 ) and (e ′ , r, e 1 ) which is not equal to correct triples by one entity. Following Toutanova and Chen (2015), the conditional probability p(e 2 | e 1 , r) for the object entity given the relation and the subject entity is defined as: where (e 1 , r, ?) is a set of triples that do not match the object position in the relation triple. Since the entity number of such a set is large, the negative triples are randomly sampled from the full set. Similarly, given the relation and object, the conditional probability of the subject is defined as: Frontiers in Neuroscience | www.frontiersin.org FIGURE 4 | Valence-Arousal 2D Model: This is put into two separate groups. The first group is high arousal and low arousal. High arousal contains surprise, fear, anger, and happy. Low arousal contains disgust, neutral, and sad. Second group is positive valence and negative valence. Positive valence contains neutral, surprise, and happy. Negative valence contains fear, anger, sad, and disgust .
Given the definition of conditional probability, we can maximize the probability of existing triples. Our training loss function is defined as the sum of the negative log-probabilities of observed triples with L2 penalty. Suppose X denotes the set of observed triples and λ is the trade-off parameter, the training loss is given by: L(X, , λ) = − The algorithm of graph feature extraction is summarized in Algorithm 1.

SF Learning Structure 2.5.3. Preprocessing
In this paper, we used PAFEW and DEAP databases for analysis. E4 wristband is used to collect EDA signals at a rate of 4 Hz in the PAFEW dataset. Participants were asked to watch videos of the same emotional label consecutively. We treat each emotion category as continuous in time for normalization. Data obtained varies significantly from 0 to 6.68 among participants. Minmax normalization is applied to reduce in-between participant differences. The 11-point median filter is used to remove noise, smoothen signal sequence. For the DEAP dataset, we used the preprocessed data. We also normalized the data to reduce intra-and inter-individual differences associated with emotional responses. The data is downsampled to 128 Hz. After, it is segmented into 60 s trials and 3 s pre-trials, baseline is removed.  Define the score triple using Equation (1).

6:
Corrupt to generate Neg object triples G − = (E × R × E ′ ) at training for positive, or true triple.

7:
Evaluate by defining filters to ensure no negative statements generated by the corruption procedure are actually positives by concatenating train and text sets. Update entity and relation embedding w.r.t gradients of Equation (4). 10: end loop

Feature Extraction
We extract time and frequency signals which is listed in Table 4. Features were extracted for the entire duration as participants watch emotional videos. For the PAFEW data, extracted features were categorized into three groups: basic statistical variables, first-order differential variables, and second-order differential variables. For the DEAP dataset, we used the toolbox for emotional feature extraction from physiological signals (TEAP) (Soleymani et al., 2017) to extract SFs. These features are in line with the work of Shukla et al. (2019) that extensively studied and reviewed EDA features across statistical domains relevant for emotion recognition. Our choice of features is easy to compute online, thus, making them advantageous in the future for realtime recognition.

Graph Feature Weighted Fusion Representation Learning
We present our proposed weighted feature fusion mechanism for representation learning which effectively improves the performance of our model.
After training with the loss function (4), we extract the real part of the complex feature vector of the subjects as a graph embedding feature, and the feature matrix of all entities is denoted by G. High-dimensional space is required to embed the entities so that they are adequately distinguished, but the dimension is incompatible with the SF. Here, we use principal component analysis (PCA) to reduce the dimension to be the same as SF. The feature matrix after dimension reduction is denoted byĜ.
To fuse SF S and graph embedding featureĜ, one naive method is concatenation. Since they are extracted based on a totally different viewpoint, simple concatenation may not fuse them well. Here, we use a more sophisticated method by exploiting the graph embedding feature as the weight of the SF. Because the dimension order of the graph embedding feature does not naturally match the dimension order of SF, we introduce a learnable transformation matrix P. The fused feature is given by: where ⊗ denotes element-wise product. By taking F as the input of a neural network, we can obtain the transformation matrix P through training. Here, we use an MLP with the architecture of three hidden layers and an output layer. The dimension for the number of the hidden units is 64, 128, and 64. The output of the network is 2-dimension vector. We use the mean square error (MSE) criterion and Adam optimizer. Dropout is set at 0.2, batchnormalization and ReLU are applied after each hidden layer after which Softmax is employed for final classification. The whole procedure of our feature fusion learning method is summarized in Algorithm 2.

RESULTS
We reported experimental results separately for each experiment to have a clearer assessment of our knowledge graph EDA-based approach. Tables 5, 6 present the average results of classification recall, precision, accuracy, and F1-scores on PAFEW and DEAP. We also report the standard deviation of accuracy.
The results are based on leave-one-out training and testing, i.e., training all but one subject data, and using that one subjects data for testing. The results for SF features are consistent with previous studies. However, the GARG feature embeddings give a boost in model performance, hence improvement in classification results. This is because our graph embedding feature vectors can effectively learn the representation between participants' gender and age and their respective emotional labels. Also, because the GARG features are extracted differently and require a high dimensional space to effectively mine related information. Input: Graph embedding featureĜ and statistical feature S Procedure: 1: Extract embedding feature using Algorithm 1 in high dimension and denoted as G 2: Use PCA to reduce dimension of feature matrix then denote asĜ 3: Extract statistical feature and denote as S 4: Introduce a learnable feature transformation matrix P 5: Compute the fused feature F using Equation (5) 6: Define the neural network architecture 7: Take the fused feature F as input and train the network Output: P, W Furthermore, directly combining them with the SF features will be inappropriate. We, therefore, use PCA to match their dimensional space to the SFs. Hereby, we used them as a weight to the statistical features to enhance model performance. The tables also clearly show that, a weighted combination of SF-GARG features yield better performance than those of the SF features alone. MLP clearly shows the highest performance in all tables compared to SVM, KNN, RF, and NB.
The Nemenyi test (Demšar, 2006) and Friedman test (Friedman, 1937) are used to show statistical significance of our method. To detect the differences between multiple methods across multiple test results, the Friedman test, which is a nonparametric statistical test, is used. Its null hypothesis states that -all methods have the same performance. The Nemenyi test is used to distinguish whether the performances of the methods are significantly different if the null-hypothesis is rejected. In the PAFEW dataset, on the arousal scale, we calculated F (5,) = 3.57 for accuracy and F (5,) = 3.75 for F1 score. On the valence scale, we calculated F (5,) = 3.69 for accuracy and F (5,) = 3.89 for F1 score. This showed p < 0.05. Similarly, in the DEAP dataset on the arousal scale, we calculated F (5,) = 3.63 for accuracy and F (5,) = 3.76 for F1 score. On the valence scale, we calculated F (5,) = 3.68 for accuracy and F (5,) = 3.74 for F1 score which showed p < 0.05. The critical distance (CD) of the Nemenyi test is defined as follows: CD = q α k(k+1) 6N , where q α is a default critical value of 0.05, k denotes the number of methods (k = 5) in this work, and N the number of result groups. For PAFEW, N = 57 and for DEAP, N = 32. This resulted in no overlaps with methods suggesting that our proposed methods are statistically different in performance with compared methods. Figure 5 is the confusion matrix for the classification results on the PAFEW dataset and shows a clearer picture of our model performance. Figures 6, 7 show the training and validation loss on the PAFEW dataset for both valence and arousal classification. Our method reduces the complexity of the model with respect to the reduction of dimension of the complex embeddings and the number of layers in our neural network model. In the training phase, the time taken to train the model is relatively fast. The figures show that our model is able to reduce variance of bias and finds real relationships between The bold values are the highest accuracy obtained. The bold values are the highest accuracy obtained.

DISCUSSION
In this paper, we focused on two kinds of issues. 1) Knowledge graph generation using participant's gender and age information and 2) Weighted feature fusion issue for EDA emotion classification system. In generating a knowledge graph using gender and age, we created triple entities and drew relationships between EDA signals, participants' gender and age, and their respective emotion labels. The weighted feature fusion issue is an algorithmic idea that exhibits the performance of the proposed method in an effective way for EDA emotion applications. We also deliberated the advantages of MLP over other machine learning methods. Regarding knowledge graph generation, results indicated that our deep learning approach can capture relevant complex information from participants' gender and age and predict emotional states correctly. On the contrary, SVM, KNN, RF, and NB could not fully mine such complex information but averaged consistent results higher than only SF features which also proves GARG features as effective for learning representation in EDA. The deep learning approach can also capture invisible features that traditional methods cannot. The results also show that our model can effectively identify true positive results (recall) than previous EDA-based approach.
Pertaining to the weighted feature fusion issue, the proposed sophisticated algorithm outruns existing methods. Unlike previous methods where features from other signals are directly concatenated, we introduced a transformation matrix that exploits the gender-age graph embedding features as a weight to the SF and performed element-wise multiplication.  Koelstra et al. (2012) and Soleymani et al. (2017) that uses machine learning algorithms, the features extracted did not particularly reflect participants' affective states. Our deep learning approach is data driven and based on training data. Also, the GARG embedded features reflect participants' affective state in the valence-arousal scale. Hence, the robustness of our method compared to theirs. Additionally, Ganapathy et al. (2020), Zhang et al. (2020), and Liu et al. (2020) uses DL but our model was manifestly superior with regard to training speed with low SD in both valence and arousal dimension. Table 8 summarizes all compared methods with respect to some qualitative features. It is easily observed that our proposed approach possesses two kinds of qualities-integrating knowledge graph embedding vectors and EDA features and excellent complexity order those other methods do not.
The proposed knowledge graph and EDA-based method take advantage of a single EDA module and graph embedding tricks to improve fusion performance. Our approach does not consume more computation time compared with other methods that use multiple kernel combination techniques. The introduction of the learnable transformation matrix enables the feature space to form an implicit combination. This is less computationally expensive compared to other methods. The multilayer neural network employed exhibits strength and flexibility in representation learning hence superior performance regarding fusion capabilities.
Finally, none of the previous works utilized participants' GARG embeddings as weight to statistical features to optimize their approach. This is an important advantage of our proposed approach. Investigating GARG embeddings and accurately combining them with SFs lead to a 4% and 5% increase in performance on the PAFEW dataset and 3% and 2% on the DEAP dataset for valence and arousal, respectively.
In the future, we can try to explore more physiological signals like EEG, EMG, and BVP with knowledge graph embeddings and devise new fusion techniques to improve emotion classification results.

CONCLUSION
This paper investigates the feasibility of employing GARG embeddings features and EDA/GSR Statistical Features (SF) to build an emotional state classification system. We proposed an effective knowledge graph method that uses gender and age information that is mostly ignored in feature analysis to predict participants' emotional states. We then extract SFs from the EDA/GSR data consistent with cognitive research. Finally, we introduced a weighted fusion strategy that uses GARG embeddings as a weight to SFs to improve classification performance. We evaluated our study on PAFEW and DEAP datasets. The results obtained show the superiority of our approach when compared to previous works.

DATA AVAILABILITY STATEMENT
Publicly available datasets were analyzed in this study. The data can be found here: https://sites.google.com/view/emotiw2020 and https://www.eecs.qmul.ac.uk/mmv/datasets/deap/.