MDDD: Manifold-based Domain Adaptation with Dynamic Distribution for Non-Deep Transfer Learning in Cross-subject and Cross-session EEG-based Emotion Recognition
Ting Luo Jing Zhang Yingwei Qiu Li Zhang Yaohua Hu Zhuliang Yu Zhen Liang



























































Journal of  Class Files, Vol. 14, No. 8, August 2015
Shell et al.: Bare Demo of IEEEtran.cls for Computer Society Journals






































Emotion decoding using Electroencephalography (EEG)-based affective brain-computer interfaces (aBCIs) represents a significant area within the field of affective computing. However, the application and generalization of existing models are severely limited by the non-stationarity and individual variability of EEG emotional data, resulting in poor cross-subject model sharing and applicability. Additionally, the typically small sample sizes of EEG signals, the high costs and time requirements for acquiring large volumes of labeled data, and the substantial hardware demands of deep learning approaches pose further challenges. To address these key issues, we propose a novel non-deep transfer learning method, termed as Manifold-based Domain adaptation with Dynamic Distribution (MDDD). The proposed MDDD includes four main modules: manifold feature transformation, dynamic distribution alignment, classifier learning, and ensemble learning. The data undergoes a transformation onto an optimal Grassmann manifold space, enabling dynamic alignment of the source and target domains. This process prioritizes both marginal and conditional distributions according to their significance, ensuring enhanced adaptation efficiency across various types of data. In the classifier learning, the principle of structural risk minimization is integrated to develop robust classification models. This is complemented by dynamic distribution alignment, which refines the classifier iteratively. Additionally, the ensemble learning module aggregates the classifiers obtained at different stages of the optimization process, which leverages the diversity of the classifiers to enhance the overall prediction accuracy. The proposed MDDD model is validated with two well-known databases and two different validation methods (cross-subject single-session and cross-subject cross-session). The experimental results indicate that MDDD outperforms traditional non-deep learning methods, achieving an average improvement of 3.54%, and is comparable to deep learning methods. This suggests that MDDD could be a promising method for enhancing the utility and applicability of aBCIs in real-world scenarios.





Electroencephalography, Emotion Recognition, Dynamic Domain Adaptation, Non-Deep Transfer Learning, Manifold Transformation.



MDDD: Manifold-based Domain Adaptation with Dynamic Distribution for Non-Deep Transfer Learning in Cross-subject and Cross-session EEG-based Emotion Recognition
    Ting Luo1,
Jing Zhang1,
Yingwei Qiu,
Li Zhang,
Yaohua Hu,

Zhuliang Yu, and
Zhen Liang*


Ting Luo, Li Zhang, and Zhen Liang are with the School of Biomedical Engineering, Medical School, Shenzhen University, Shenzhen 518060, China, and also with the Guangdong Provincial Key Laboratory of Biomedical Measurements and Ultrasound Imaging, Shenzhen 518060, China. E-mail: 2310247018@email.szu.edu.cn, and {lzhang, janezliang}@szu.edu.cn.
Jing Zhang is with the School of Public Health, Shenzhen University Medical School, Shenzhen, China. E-mail: zhangjing1985zj@163.com.
Yingwei Qiu is Department of Radiology, Nanshan Hospital of Shenzhen University, Shenzhen, China. E-mail: qiuyw1201@gmail.com.
Yaohua Hu is with the School of Mathematical Sciences, Shenzhen University, Shenzhen, China. E-mail: mayhhu@szu.edu.cn.
Zhuliang Yu is with Shien-Ming Wu School of Intelligent Engineering, South China University of Technology, Guangzhou, Guangdong, China, and also with Institute for Super Robotics, Guangzhou, Guangdong, China. E-mail: zlyu@scut.edu.cn.
*Corresponding author: Zhen Liang.
    
==================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================



[1]Equal contributions.



























§ INTRODUCTION

Emotion, functioning as both a physiological and psychological state within an individual, serves as a rich repository of insights into mental and physical well-being <cit.><cit.><cit.>. Emotion recognition stands as a crucial method for objectively discerning human emotional states. Currently, the practice relies on the analysis of either non-physiological signals or physiological signals. In contrast to cues like facial expressions, voice intonation, and gestures, Electroencephalography (EEG) stands out as a non-invasive brain information collection technique. Its strong correlation with human emotional states, coupled with its resistance to deception, positions EEG as a highly accurate means to objectively reflect human emotions <cit.><cit.><cit.>.

EEG signals exhibit non-stationary characteristics and individual differences <cit.><cit.>, leading to significant variations in the distribution of EEG signals within the same subject across different time periods or between different individuals. Traditional machine learning methods assume independence and identical distribution of data, causing emotion recognition models to have poor generalization across different sessions and different individuals. This limitation poses a challenge in expanding the widespread adoption and application of these models to new sessions and individuals. On the other hand, EEG signals encounter the challenge of a small sample size, typically comprising only a few hundred to a few thousand samples. The labeling process is often expensive and time-consuming, making it challenging to amass a large volume of data. This difficulty in obtaining substantial data poses a hurdle in training a reliable model.

Early EEG-based emotion recognition methods mainly rely on conventional non-deep machine learning algorithms<cit.>, including Support Vector Machine (SVM), K-Nearest Neighbor (KNN), and Linear Discriminant Analysis (LDA). In these studies, there is an underlying assumption that EEG samples are independent and identically distributed. However, variations among individuals significantly undermine the effectiveness of traditional machine learning methods in cross-subject emotion recognition tasks. This results in suboptimal performance and poor generalization effects. On the other hand, transfer learning techniques can be beneficial in transferring informative data from the source (training data) to the target domain (test data), which could reduce the variations between different distributions in the modeling. More related transfer learning methods are detailed in Section <ref>.

Compared to deep learning-based transfer learning methods, non-deep learning-based transfer learning offers a more lightweight approach. Unlike deep learning methods, non-deep learning-based transfer requires less labeled data, which is better suited to the current conditions in EEG tasks. In this paper, we propose a novel non-deep learning-based transfer learning framework for cross-subject emotion recognition, named as Manifold-based Domain adaptation with Dynamic Distribution (MDDD). In the proposed MDDD framework, EEG data are mapped to a manifold space where both the marginal and conditional distributions are dynamically analyzed and aligned. Subsequently, classifier learning and ensemble learning are integrated to facilitate reliable cross-subject and cross-session emotion recognition. The results obtained are comparable to those achieved by existing deep-learning-based methods. The main contributions of the paper are summarized below.


  * A novel non-deep learning-based transfer learning model, MDDD, is proposed to address the variations in the distribution of EEG signals across different sessions and subjects.

  * Both marginal distribution and conditional distribution are considered during domain alignment, where  a dynamically adjusted weighting of the importance of the two distributions is incorporated.

  * An adaptive classifier learning together with an ensemble learning is introduced to  iteratively refine the classifier learning process and leverage the strength of multiple learning models to optimize the prediction of data labels.

  * Extensive experiments are conducted on two well-known databases under two different validation protocols (cross-subject single-session and cross-subject cross-session). The obtained from the proposed MDDD (non-deep learning) are comparable to those from existing deep learning approaches.









§ RELATED WORK


Domain adaptation stands out as the predominant method in transfer learning, frequently employed to address differences between the distributions of source and target data. It targets the reduction of distribution disparities between the two domains and align the feature distributions, achieving the assumption that the source and target data is independently and identically distributed. Existing domain adaptation-based transfer learning algorithms can be generally categorized into two broad categories: deep transfer learning and non-deep transfer learning.



 §.§ Deep Transfer Learning Methods

The current trend in the development of transfer learning methods has mainly focused  on deep learning-based approaches. For example, Li <cit.> pioneered the use of the Domain Adversarial Neural Network (DANN) in the study of EEG-based emotion recognition. The optimization objective was to establish a shared common feature representation that mitigates distribution differences between the source and target domains. Building upon the aligned feature representation achieved with DANN, Zhou <cit.> introduced a prototypical representation-based pairwise learning framework, aiming to further enhance cross-subject emotion recognition performance. However, the current DANN network is limited by its exclusive focus on addressing marginal distribution differences in EEG data among different individuals, neglecting joint distribution differences. Considering that individual differences in EEG arise from joint distribution differences between different EEG signals, Li <cit.> extended the DANN network by incorporating the Joint Domain Adaptation Network. This extension provides a more comprehensive approach that considers both marginal and joint distribution differences in deep transfer learning studies.

Despite the notable achievements of the deep transfer learning methods in emotion recognition tasks, there are certain limitations. Firstly, deep learning methods demand a substantial number of training samples. However, in EEG tasks, the available samples are typically limited to a few hundred or a few thousand. Insufficient samples would constrain performance, leading to overfitting and a decrease in generalization ability. Secondly, deep transfer learning algorithms heavily rely on a substantial amount of accurately labeled source data. If there is noticeable noise in the labels from the source domain, the recognition performance would significantly deteriorate <cit.>. However, acquiring a sufficient number of samples with precise label information is not only costly but also time-consuming. Thirdly, deep transfer learning methods exhibit high computational complexity, substantial computational requirements, and involve slow and complicated training processes. The reliance on high-end hardware facilities is a significant demand, posing challenges in practical application environments where such resources may not be readily available. In contrast, non-deep transfer learning models exhibit lower complexity, reduced hardware requirements, and lightweight properties, showing the potential for good performance in widespread applications in real-life scenarios.




 §.§ Non-Deep Transfer Learning Methods

Non-deep transfer learning methods align the feature distribution between source and target domains using conventional machine learning approaches. For example, Pan <cit.> introduced a Transfer Component Analysis (TCA) algorithm, leveraging the reproducing kernel Hilbert space to learn transfer information. Through maximizing the Mean Discrepancy (MMD), it successfully mitigated the marginal distribution differences. Zheng <cit.> introduced a Transductive Parameter Transfer (TPT) algorithm to build personalized classifiers for subjects from various source domains. This algorithm mapped the individualized classifier parameters to the target domain, minimizing the MMD between the source and target domains and adapting to marginal distribution. 
Wang <cit.> introduced a Stratified Transfer Learning (STL) algorithm, which constructed a weak classifier with source data and making predictions on the target domain. Based on this foundation, it further conducted a more thorough exploration of intra-class relationships, leveraging class-relatedness to accomplish adaptive spatial dimensionality reduction.

Although the transfer learning strategies mentioned above effectively mitigate differences between the distributions of source and target data by considering marginal or conditional distributions, it's crucial to note that the actual disparities in EEG between subjects primarily exist in their joint distribution <cit.>. Current domain adaptation algorithms either focus on marginal or conditional distributions, neglecting to assess the combined contributions in domain adaptation process. It would lead to inefficient data utilization and suboptimal results. To tackle this issue, He <cit.> proposed a Joint Distribution Adaptation (JDA), which involved matching these two distributions with equal weights to approximate the adjustment of the joint distribution <cit.>. Building upon the proposed JDA, other studies extended its capabilities by incorporating regularization <cit.>, sparse representation <cit.>, structural consistency <cit.>, and domain-invariant clustering <cit.>.

However, current methods commonly equate conditional probability distribution with marginal probability distribution. In real-world applications, the contributions of conditional and marginal distributions to the joint probability distribution could be different, which would further affect the alignment effect between the source and target domains. For example, when two domains exhibit high dissimilarity, aligning the marginal distribution becomes more crucial. Conversely, when two domains exhibit high similarity, greater emphasis should be placed on the conditional distribution. Therefore, for better alignment between the two domains, adjusting the importance of marginal and conditional distributions based on the data distribution in the two domains is a critical issue in non-deep transfer learning studies.



To tackle the existing critical issues and enhance the performance of non-deep transfer learning in EEG-based emotion recognition applications, we propose a novel Manifold-based Domain adaptation with Dynamic Distribution (MDDD) model in this paper. Based on the extracted differential entropy (DE) features, we first incorporate the TCA algorithm to reduce feature dimensionality and narrow the distribution gap between the features derived from the source and target domains. Subsequently, the features are embedded into the Grassmann manifold space through the computation of the Geodesic Flow Kernel (GFK)<cit.>. To enhance the adaptability of our framework, we introduce an adaptive factor that dynamically assesses the significance of both marginal and conditional distributions in the two domains. This factor aids in optimizing the embedding process and ensuring the effective representation of manifold features. Upon achieving a robust manifold feature representation, we proceed with classifier learning to optimize the classification performance under the complex conditions of cross-subject and cross-session scenarios. In this phase, conventional machine learning methods such as Support Vector Machine (SVM), K-Nearest Neighbors (KNN), AdaBoost, Gaussian Naive Bayesian (GNB), Decision Tree (DT), and Bagging classifiers are employed as the initial classifiers. Finally, the classification results derived from diverse parameter settings are harmonized through an ensemble learning approach. This ensemble strategy enhances the overall robustness and reliability of the proposed framework, ultimately contributing to more accurate and reliable classification outcomes.



§ METHODOLOGY
 

In this section, we will provide a detailed introduction to the proposed MDDD model. As shown in Fig. <ref>, MDDD includes four main modules: manifold feature transformation, dynamic distribution alignment, classifier learning, and ensemble learning. In the manifold feature transformation module, the input data initially undergoes a reduction to lower-dimensional features through TCA. Then, a manifold kernel G is learned to facilitate the transformation of the reduced features into an optimal manifold space. In the dynamic distribution alignment module, Structural Risk Minimization (SRM) <cit.> is incorporated to dynamically align the feature distribution within the obtained manifold space. This adaptive alignment process enhances the model's capacity to effectively capture and adapt to variations in the underlying data distribution. In the classifier learning module, a classifier is constructed using the aligned manifold features. Initially, conventional machine learning methods are employed, and the classifier is then further iteratively optimized by solving the given classification loss. In the ensemble learning module, all the optimized classification results from the iterative process in the classifier learning module are harmonized to generate a more robust and reliable performance. These are the final results for cross-subject and cross-session EEG-based emotion recognition.




 §.§ Manifold Feature Transformation


Traditional feature alignment methods typically rely on original data features, which can be problematic due to deformations within the original feature space. These deformations make it difficult to effectively reduce differences between the source and target domains, negatively impacting model performance. Alternatively, the manifold space offers a solution by capturing the core aspects of data and representing original information in a more compact form <cit.>. In the present study, our emphasis centers on leveraging the Grassmann manifold space to encapsulate the intrinsic essence of the data, which enables the creation of a more robust and dimensionality-reduced representation and offers a more effective way to align features across different domains.

In this paper, we introduce a manifold feature transformation process, as illustrated in Fig. <ref>. The source and target domains are represented as D_S={(x_1,y_1),...,(x_n,y_n)} and D_T={(x_n+1,y_n+1),...,(x_n+m,y_n+m)}, respectively, with each domain featuring D-dimensional data representations. Through Transfer Component Analysis (TCA) <cit.>, we map both domains into a lower d-dimensional feature space (d ≪ D), denoted as T_S and T_T. This process effectively reduces the dimensionality of the feature space, facilitating more efficient data analysis and interpretation. Subsequently, we embed the mapped d-dimensional feature subspace into the Grassmann manifold, denoted as 𝔾. In this manifold space, our goal is to enhance the similarity between the source and target domains' data as much as possible. Simultaneously, we aim to retain the unique attributes and inherent characteristics of each domain's data. This approach ensures that, despite the alignment of the domains in a shared feature space, the distinctiveness and originality of each domain are preserved, maintaining the integrity and context of the data from both domains.

To make the data distributions of the two domains more similar in the Grassmann manifold, we either bring the source domain closer to the target domain or vice versa. Here, we construct geodesics within 𝔾, treating the source and target domains as points in this manifold space. By navigating along these geodesics, we facilitate a systematic journey from the source domain towards the target domain, enabling a gradual harmonization of their data distributions within the Grassmann manifold and ensuring a smoother and more effective domain alignment. In the process of constructing geodesics, we apply the standard Euclidean metric to the Riemannian manifold. This allows the geodesic to be described by a parameterization function, Ψ, where t ranges from 0 to 1, mapping to Ψ(t) within the Grassmann manifold. Specifically, Ψ(0) and Ψ(1) correspond to the embedded representations of the source and target domains in 𝔾, respectively. Thus, the geodesic delineates the trajectory by which Ψ(0) evolves into Ψ(1), representing the most direct path within the manifold space that connects these two domain representations. To ensure a smooth and continuous transformation across the manifold, the intermediate values of t (t ∈ (0,1)), Ψ(t) satisfies the following conditions:

    Ψ(t)=T_SU_1Γ(t)-R_SU_2Σ(t),

where T_S∈ℝ^M × d denote the set of orthonormal base for the source domain. M is the sample size. R_S∈ℝ^M × (D-d) signifies the orthogonal complement of T_S, serving to encapsulate the dimensions outside the primary subspace spanned by T_S. U_1∈ℝ^d × d and U_2∈ℝ^(D-d) × d are both orthogonal matrices, which can be obtained through the following two sets of Singular Value Decomposition (SVD) as

    T^⊤_ST_T=U_1Γ V^⊤, R^⊤_ST_T=-U_2Σ V^⊤.

Here, T_T∈ℝ^M × d refer to the set of orthonormal base for the target domain. Γ and Σ are both d × d diagonal matrices, where their diagonal elements being cosθ_k and sinθ_k (k = 1,2,...,d), respectively. The variable θ_k denotes the angles between the orthonormal bases T_S and T_T, with the constraint 0 ≤θ_1≤θ_2≤ ... ≤θ_d≤π2. θ_k serve as a metric for quantifying the alignment between the domains within the 𝔾, indicating how closely the subspaces represented by T_S and T_T are oriented to one another.

The operation Ψ(t)^⊤x represents the projection of a feature vector x from the d-dimensional feature space onto the Grassmann manifold space 𝔾, where x could be any data sample originating from either the source or the target domain. The selection of an appropriate value or range of values for t becomes a significant challenge, as it directly influences the quality of the projection for x. The optimal choice of t ensures that the projected data maintains a balance between adapting to the new domain and preserving its original domain characteristics. This balance is essential for effective domain adaptation, as it allows for the integration of domain-specific features while minimizing the loss of critical information during the transition.

Here, we introduce an integration strategy to identify the optimal t or a set of t points between any two given data samples x_i and x_j, and utilize a geodesic kernel to facilitate the mapping process. This methodology effectively transforms the d-dimensional feature space into an infinite-dimensional feature space, which serves to mitigate domain drift (a common challenge in domain adaptation). For each pair of data samples, x_i and x_j, we iteratively calculate their projections onto Ψ(t), where t varies incrementally from 0 to 1. Through this process, each data sample is mapped across a continuum within the Grassmann manifold space, capturing the evolution of its feature representation as it transitions from being aligned with the source domain (t=0) to being aligned with the target domain (t=1). The culmination of this iterative projection process results in the creation of two infinite-dimensional feature vectors, denoted as z^∞_i and z^∞_j,

    z^∞_i=∫^1_0(Ψ(t)^⊤x_i)dt,


    z^∞_j=∫^1_0(Ψ(t)^⊤x_j)dt,

where z^∞_i and z^∞_j encapsulate the comprehensive trajectory of each data sample across the manifold, providing a rich, multidimensional perspective on the domain adaptation process. This strategy enhances the adaptability and accuracy of domain adaptation by leveraging the inherent structure of the data within the manifold, thereby offering a more effective approach to overcoming the challenges of domain drift.
From <cit.>, the geodesic kernel is then determined as the inner product of the two infinite-dimensional vectors, as

    <z^∞_i,z^∞_j>=∫^1_0(Ψ(t)^⊤x_i)^⊤(Ψ(t)^⊤x_j)dt=x^⊤_iGx_j,

where G ∈ℝ^D × D is a positive semi-definite matrix, computed as

    G=[ T_SU_1 R_SU_2 ][ Λ_1 Λ_2; Λ_2 Λ_3 ][ U^⊤_1 T^⊤_S; U^⊤_2 R^⊤_S ].

Here, Λ_1, Λ_2, Λ_3 are all diagonal matrices, and the corresponding diagonal elements are given as 

    λ_1,k=1+sin2θ_k2θ_k,
    λ_2,k=cos2θ_k-12θ_k,
    λ_3,k=1-sin2θ_k2θ_k,
where k is in the range of [1,d]. The computed geodesics kernel G enables the effective transformation of a data sample x, initially positioned within the d-dimensional feature space, into the Grassmann manifold space, as

    z=√(G)x.

This transformed representation, z, integrates the intrinsic features of x as it is projected through the manifold, leveraging the geometrical properties of 𝔾 to achieve a more domain-adaptive feature representation and address domain drift in transfer learning.




 §.§ Dynamic Distribution Alignment


After the manifold feature transformation, the feature distributions between the source and target domains would exhibit greater similarity than seen with the original feature representation. Still, differences remain in both the marginal probability distribution and conditional probability distribution. Current distribution adaptation methods commonly assume equal importance for both the marginal distribution P and conditional distribution Q <cit.>. Traditional distribution adaptation methods often treat the adaptation of marginal and conditional distributions as equally important, an assumption that may not always be applicable. In scenarios where significant differences exist between the source and target domains, the adaptation of the marginal distribution takes on increased importance. Conversely, in situations where the source and target domains are more closely aligned, the adaptation of the conditional distribution becomes more critical. Therefore, recognizing and adjusting the emphasis on marginal or conditional distribution adaptation based on the domains' specific characteristics is essential for achieving successful distribution adaptation.

In the present study, an adaptive factor μ∈ [0,1] is incorporated to dynamically adjust the emphasis placed on marginal and conditional distributions. This adaptive factor allows for the flexible allocation of importance between the two types of distributions based on the characteristics of the source and target domains, which is defined as

    D_f(T_S,T_T)=(1-μ)D_f(P_s,P_t) 
    
        +μ∑^C_c=1D_f^(c)(Q_s,Q_t),

where D_f(T_S, T_T) represents the dynamic distribution adaptation between the source and target domains within the manifold feature space. D_f(P_s, P_t) is the marginal distribution adaptation, and D_f^(c)(Q_s, Q_t) is the conditional distribution adaptation for the class c (c ∈{1,..., C}). The total number of classes is represented by C. When μ is close to 0, it indicates significant differences between the source and target domains, thus prioritizing the adaptation of marginal distributions. When μ approaches 1, it suggests a greater similarity between the domains, thereby emphasizing the need for conditional distribution adaptation. At the midpoint, μ = 0.5, the model treats both marginal and conditional distributions with equal importance.

In (<ref>), the marginal distribution adaptation D_f(P_s, P_t) is defined as

    D_f(P_s,P_t)=‖ E[f(z_s)]-E[f(z_t)] ‖^2_H_K,

where z_s and z_t are the source and target data represented in the manifold feature space, given in (<ref>). f refers to the classifier. The term H_K denotes the reproducing kernel Hilbert space (RKHS), which is a space of functions generated by the feature mapping Ψ(·). This mapping is critical for capturing the complex structures within the data by projecting it into a higher-dimensional space where linear separability is more feasible. For the computation of the conditional distribution adaptation D_f^(c)(Q_s, Q_t), it calculate in a similar way, as
   
    D_f^(c)(Q_s,Q_t)=‖ E[f(z_s^(c))]-E[f(z_t^(c))] ‖^2_H_K,

where z_s^(c) and z_t^(c) are the c-class source and target data represented in the manifold feature space (<ref>). Then, (<ref>) could be rewritten as

    D_f(T_S,T_T)=(1-μ)‖ E[f(z_s)]-E[f(z_t)] ‖^2_H_K
    
         +μ∑^C_c=1‖ E[f(z_s^(c))]-E[f(z_t^(c))] ‖^2_H_K.


Given the absence of label information for the target domain during the training phase, we cannot directly obtain f(z_t^(c)) in (<ref>). To tackle this issue, we approximate f(z_t^(c)) using the class-conditional probability distribution <cit.>. Specifically, the process begins by training an initial weak classifier, such as a Support Vector Machine (SVM) or K-Nearest Neighbors (KNN), on the source data. This initial classifier, f, is subsequently utilized to infer pseudo-labels for the target data, serving as a provisional substitute for the unavailable true labels of the target domain. To enhance the reliability and accuracy of the pseudo-labels, an iterative refinement strategy is employed to improve the decodability f. This involves using the outcomes of the initial classifier to iteratively train subsequent classifiers, thereby progressively refining the pseudo-labels. This iterative process aims to converge towards more accurate pseudo-labels, thus improving the model's ability to generalize from the source to the target domain effectively. Further details on the specific iterative refinement process and how it contributes to classifier learning will be discussed in Section <ref>.


In (<ref>), the adaptive factor μ is not a constant value predetermined by prior knowledge. Instead, it is dynamically learned based on the underlying data distribution. Specifically, we use the A-distance measurement <cit.> as a metric to estimate the marginal distribution difference, denoted as d_A, between the source and target domains, as

    d_A(T_S,T_T)=2(1-2ϵ(h)),

where ϵ(h) denotes the hinge loss, which is derived from a binary classifier trained specifically to distinguish between samples from the source and target domains. For the conditional distribution difference, based on the provided label information for source data and the estimated pseudo-label information of target data, we conduct the A-distance measurement within each c class (c ∈ [1,…, C]) as

    d_c=d_A(T_S^(c),T_T^(c)).

Here, T_S^(c) and T_T^(c) represent the subsets of data corresponding to class c from the source and target domains, respectively. Then, the adaptive factor μ can be estimated as

    μ=1-d_Ad_A+∑^C_c=1d_c.

In the process of dynamic distribution alignment, the adaptive factor μ is recalculated at each iteration, underscoring the iterative and responsive nature of this adaptation strategy. This re-calibration is essential to ensuring that the adaptation process remains attuned to the evolving similarities and differences between the source and target domain feature distributions as they are represented in the manifold feature space.






 §.§ Classifier Learning


Through the integration of manifold feature learning and dynamic distribution alignment, we can formulate an adaptive classifier f at the ιth iteration as

    f^ι=min_f∈ H_K∑^n_i=1(y_i-f^(ι-1)(z_i))^2+η‖ f^(ι-1)‖^2_H_K
    
        +λD_f^(ι-1)(T_S,T_T)
        +ρ R_f^(ι-1)(T_S,T_T).

Here, ι is the iteration number, y_i is the groundtruth, and f(z_i) is the predicted classification result. ‖·‖^2_H_K refers to the L2 norm in the reproducing kernel Hilbert space. The dynamic distribution adaptation term, D_f(T_S,T_T), is obtained in (<ref>). Further, a Laplacian regularization term, R_f(T_S,T_T), is incorporated to leverage the inherent geometric similarities among neighboring points in the manifold 𝔾 <cit.>. Here, the classifier f can be initialized using a commonly employed machine learning classifier, followed by adaptive optimization through an iterative learning process.

In the SRM framework with the representer theorem <cit.>, f can be expanded as
    
    f(z)=∑^n+m_i=1β_iK(z_i,z),

which includes both labeled and unlabeled samples from source and target domains, and  β = (β_1, β_2, …, β_n+m)^T ∈ℝ^(n+m)× 1 is the coefficient vector. K is the kernel function. For the SRM on the source domain, the first part in (<ref>) could be expressed as  
    
    ∑^n_i=1(y_i-f(z_i))^2+η‖ f‖^2_H_K
    
            =∑^n+m_i=1A_ii(y_i-f(z_i))^2+η‖ f‖^2_H_K,

where ‖·‖_F is the Frobenius norm. The matrix A ∈ℝ^(n+m) × (n+m) is a diagonal indicator matrix with A_ii = 1 if i ∈ T_S, and A_ii = 0 if i ∈ T_T. Substituting (<ref>) into (<ref>), then we can obtain

    ∑^n_i=1(y_i-f(z_i))^2+η‖ f‖^2_H_K
    
        =‖(Y-β^⊤K)A‖^2_F+η tr(β^⊤Kβ),

where K∈ℝ^(n+m) × (n+m) is the kernel matrix, with K_ij = K(z_i,z_j). The label information of the source and target data is denoted by Y = [y_1,...,y_n+m]. For the target data, the corresponding label information is the pseudo label generated by the learned classifier f(·). tr(·) is the trace operation.

For the second part in (<ref>), through applying (<ref>), we can represent the calculation of dynamic distribution adaptation (<ref>) as
    
    D_f(T_S,T_T)   =tr(β^⊤KMKβ),

where M = (1-μ)M_0 + μ∑^C_c=1M_c. c is the class category, and C refers to the total number of classes. M_0 and M_c are defined as

    (M_0)_ij=  
         1/n^2    z_i,z_j∈ T_S
    1/m^2    z_i,z_j∈ T_T
    
         -1/mn    otherwise
         ,

    
    (M_c)_ij=  
        1/n^2_c    z_i,z_j∈ T_S^(c)
    1/m^2_c    z_i,z_j∈ T_T^(c)
    
         -1/m_cn_c   
         z_i∈ T_S^(c),z_j∈ T_T^(c)
    
         z_i∈ T_T^(c),z_j∈ T_S^(c)
    
         0     otherwise 
         .

Here, n_c = |T_S^(c)| and m_c = |T_T^(c)|. The third part in (<ref>) (the Laplacian regularization term R_f(T_S,T_T)) could be computed as
    
    R_f(T_S,T_T)=∑^n+m_i,j=1W_ij(f(z_i)-f(z_j))^2 
    
            =∑^n+m_i,j=1f(z_i)L_ijf(z_j),

where W_ij is a pairwise similarity matrix, and L=D-W is the normalized graph Laplacian matrix. D is a diagonal matrix, with D_ii=∑^n+m_j=1W_ij. W could be represented as
    
    W_ij=
        
        sim(z_i,z_j)     z_i∈ N_p(z_j) or z_j∈ N_p(z_i),
    
        0    otherwise,

where sim(·,·) is a similarity function (e.g., cosine distance) used to assess the proximity between two points. N_p(z_i) represents a set of p nearest points to z_i. p is a hyperparameter.


Similarly, by applying (<ref>), the Laplacian regularization given in (<ref>) can be expressed as

    R_f(T_S,T_T)=tr(β^⊤KLKβ).


Then, by substituting (<ref>), (<ref>) and (<ref>)into the (<ref>), the classifier f can be calculated as:
    
    f=min_f∈ H_K‖(Y-β^⊤K)A‖^2_F+η tr(β^⊤Kβ) 
    
        +tr(β^⊤K(λ M+ρ L)Kβ).


Setting ∂ f/∂β = 0, (<ref>) could be solved as:
    
    β^*=((A+λ M+ρ L)K+η I )^-1AY^⊤.
  
After obtaining β^*, the classifier f can be determined using (<ref>). Then, the corresponding soft label of the target domain at ιth iteration could be denoted as ŷ_t^ι=f^ι(z_t).
 



[]
  The algorithm flow of the proposed MDDD model.
  
  




            


 §.§ Ensemble Learning


Ensemble learning with LinkCluE algorithm <cit.> is adopted here to strategically reduce classification bias inherent in individual models and improve the overall generalization performance of the model. Based on the obtained ŷ_t in a certain round of classifier learning, the ensemble learning leverages the strengths over a group of ŷ_t^ι (ι=1,…,l. Empirically, l is set to 10) and form together to generate a strong prediction results ŷ_t.

Fig. <ref> illustrates the two main steps involved in the LinkCluE algorithm: the link-based similarity matrix function and the consensus function. For the link-based similarity matrix function, it constructs a similarity matrix based on the links or relationships between data points in terms of Connected-Triple-Based Similarity (CTS), SimRank-Based Similarity (SRS), and Approximate SimRank-Based Similarity (ASRS). This matrix serves as a foundation for understanding the complex, often non-linear relationships that exist between data points, leveraging the structure of the data to enhance the learning process. For the consensus function, it integrates the outcomes of various individual models within the ensemble by leveraging three different hierarchical agglomerative clustering algorithms, Single Linkage (SL), Complete Linkage (CL), and Average Linkage (AL). This step is crucial for mitigating individual model biases and errors, leading to more accurate and generalizable results. 

The overall algorithm of the proposed MDDD is illustrated in Algorithm <ref>.

 





§ EXPERIMENTAL RESULTS





 §.§ Emotional Databases

The model performance is carefully validated on two well-known publicly available databases: SEED <cit.> and SEED-IV <cit.>. The SEED database includes EEG emotion data collected from 15 participants. Each participant was exposed to a total of 15 movie clips designed to evoke three types of emotions (positive, negative, and neutral). The SEED-IV database also includes EEG emotion data from 15 participants, exposed to a total of 24 movie clips aimed at eliciting four types of emotions (happy, sad, neutral, and fear). For both SEED and SEED-IV databases, participants underwent three sessions spread across separate days, with a one-week break between sessions. The EEG signals were recorded using the ESI Neuroscan system with 62 channels.

The EEG preprocessing followed established procedures in the literature. The process began with the downsampling of EEG signals to 200 Hz, followed by the removal of artifacts such as the Electrooculogram (EOG) and Electromyography (EMG). A bandpass filter with a 0.3-50 Hz range was then applied to improve the signal quality. Following this, each trial was segmented into multiple 1-second data samples. To extract emotional related information, differential entropy (DE) features were extracted across five frequency bands: Delta (1-3 Hz), Theta (4-7 Hz), Alpha (8-13 Hz), Beta (14-30 Hz), and Gamma (31-50 Hz). Thus, for each 1-second data sample, a total of 310 features (5 frequency bands × 62 channels) were characterized, which serves as the input for the model.





 §.§ Experimental Protocols

In order to thoroughly assess the robustness and stability of the proposed model and ensure a detailed comparison with existing literature, two types of experimental protocols are conducted.

    
  * Cross-subject single-session leave-one-subject-out cross-validation. As illustrated in Fig. <ref> (a), the data samples from 14 subjects in the first session are used as the source domain, and the data samples from the remaining 1 subject in the same session are used as the target domain. This procedure repeats 15 times, ensuring that each participant is treated as the target domain at least once.
    
  * Cross-subject cross-session leave-one-subject-out cross-validation. As shown in Fig. <ref> (b), the data samples from 14 subjects across all three sessions are used as the source domain, and the data samples from the remaining 1 subject across all three sessions are used as the target domain. This procedure also repeats 15 times, ensuring that each participant is treated as the target domain at least once.




In the present study, we adopt six widely recognized machine-learning classifiers to generate the initial classification results (ŷ_t^0) as mentioned in Section <ref>. The adopted machine-learning classifiers include K-Nearest Neighbors (KNN) <cit.>, Support Vector Machine (SVM) <cit.>, Decision Tree (DT) <cit.>, Adaboost classifier <cit.>, Gaussian Naive Bayes (GNB) classifier <cit.>, and Bagging classifier <cit.>. In the SVM classifier, the radial basis function (RBF) kernel is utilized. In ensemble learning, the application of three similarity matrix functions (CTS, SRS, and ASRS) combined with three hierarchical agglomerative clustering algorithms (SL, CL, and AL) yields a total of 9 possible combinations. Here, the combination that achieves the highest accuracy in predicting target domain labels will be chosen as the optimal outcome. Notably, all the best results are achieved using the CTS-SL approach. For performance evaluation, a comprehensive analysis of the model's efficacy is conducted using the accuracy and F1-score metrics.







 §.§ Experimental Results on Cross-Subject Single-Session

The experimental results on SEED database under the cross-subject single-session cross-validation are reported in Table <ref>. The best performance is obtained when DT is adopted as the initial classifier, where the corresponding accuracy and F1-score are 84.57% and 76.11%. We compare the obtained experimental results with the existing literature, as shown in Table <ref>. The comparison results show that the proposed non-deep transfer learning model (MDDD) demonstrates superior performance. In comparison to the best results obtained by the traditional machine learning methods, our model shows a performance enhancement of 8.26%. Furthermore, it performs comparably or even surpasses some transfer learning deep learning methods, such as DANN <cit.>, DAN <cit.>, BiDANN <cit.>, R2G-STNN <cit.>, and ADA <cit.>.






For the SEED-IV database, the experimental results with the cross-subject single-session leave-one-subject-out cross-validation protocol are presented in Table <ref>. The best accuracy is obtained when DT is used as the initial classifier, where the corresponding accuracy and F1-score are 60.94% and 50.49%, respectively. The performance comparison results with the existing methods are reported in Table <ref>. Compared to the best results reported in the traditional machine learning methods, the proposed MDDD demonstrates a slight performance advantage of 0.72%. Furthermore, it performs comparably or even surpasses some transfer learning deep learning methods, including DANN <cit.> and DAN <cit.>.







 §.§ Experimental Results on Cross-Subject Cross-Session

To evaluate the efficiency and stability of the proposed MDDD in managing variations in EEG data collected from the same subject at different times, we also conduct a model validation under the cross-subject cross-session leave-one-subject-out cross-validation protocol. The experimental results are reported in Table <ref>, when different initial classifiers are used. The best model performance is achieved when Adaboost is adopted as the initial classifier, where the accuracy and F1-score are 76.60% and 65.43%, respectively. The performance results against the existing methods are compared in Table <ref>. It shows, the proposed model demonstrates an accuracy improvement of 4.73%  in comparison to traditional machine learning methods. Further, it exhibits performance that is comparable, to some extent, with deep learning methods, showing the effectiveness of the proposed MDDD in managing variations in data collected from the same subject across different time periods.






For the SEED-IV database, we present the experimental results obtained with different initial classifiers using the cross-subject cross-session leave-one-subject-out cross-validation protocol in Table <ref>. The performance across various initial classifiers is relatively consistent, with the highest accuracy reaching 64.90% when DT is employed as the initial classifier. Table <ref> reports the performance comparison results against the existing machine learning and deep learning methods. Compared to the best results reported in various literature on machine learning methods, the proposed MDDD shows a performance improvement of 0.46%. Furthermore, it performs comparably or even outperforms some deep learning methods, including DANN <cit.>, DAN <cit.>, and DGCNN <cit.>.






















§ DISCUSSION




 §.§ Ablation Study

To comprehensively examine the contributions of the model components within the proposed MDDD framework, we conduct a thoroughly ablation study. Note here that all the presented experimental results in this section are based on the SEED database using cross-subject single-session leave-one-out cross-validation. The initial classifier is decision tree and the iterative loop is 10.

To evaluate the efficacy of Transfer Component Analysis (TCA) in the manifold feature transformation, we assess the model performance by comparing the use of TCA against substituting TCA with Principal Component Analysis (PCA). As shown in Table <ref>, the model exhibits significantly better performance in terms of accuracy and F1-Score when comparing TCA with PCA. This improvement can be attributed to TCA's ability to enhance model performance by simultaneously mapping and reducing dimensions, and by maximizing the similarity between datasets from two domains. It would facilitate more effective dynamic distribution alignment and classifier categorization.

To evaluate the effectiveness of dynamic distribution alignment, we opted for a fixed μ value approach, rather than using an adaptive μ. The μ values are set at 0, 0.5, and 1. At μ=0, the model exclusively focuses on the marginal distribution. With μ=0.5, both marginal and conditional distributions are considered equally. At μ=1, the model solely emphasizes the conditional distribution. As reported in Table <ref>, it shows that employing an adaptive μ can significantly enhance model performance, resulting in a 4.26% improvement in accuracy. The results demonstrate that dynamic distribution alignment can effectively quantify the disparity between source and target domains and accurately evaluate the significance of the differences between marginal and conditional distributions. This capability is particularly advantageous for the classification performance, which is tailored for addressing variations between distributions.

Further, we examine the effectiveness of classifier learning and ensemble learning in the proposed MDDD. We compare the initial classification results and the classification results after iterative learning and ensemble learning, and report the comparison results in Table <ref>. The results reveal a significant reduction in all evaluation metrics, indicating the substantial impact of the classifier learning and ensemble learning process on enhancing model performance.




 §.§ Impact of ensemble learning methods


To evaluate the impact of ensemble learning on model performance, we adjust our approach by incorporating various ensemble learning techniques, namely the averaging method, the voting method, and the LinkCLuE method. Note here that LinkCLuE is the method implemented in the proposed MDDD. Based on the obtained classification results in the classifier learning (ŷ_t^(ι) (ι=1,⋯,10)), the effectiveness of ensemble learning is analyzed. Furthermore, we drew comparisons to scenarios devoid of ensemble learning strategies by directly employing the final classification outcome from the last iteration loop in the classifier learning, denoted as ŷ_t^(ι) (with ι=10), which serves as our baseline. The experimental comparison results are reported in Table <ref>. It shows that ensemble learning with voting and LinkCluE could enhance the reliability of classification results and improve the overall performance. This confirms the effectiveness of ensemble algorithms. Furthermore, the LinkCLuE method outperforms the voting method, demonstrating a superior ability to synthesize classification outcomes across iterations and to discern the underlying patterns of the classification results.





 §.§ Parameter effect


To examine the classifier learning process, we also investigate how sensitive the model is to changes in the number of iterations (ι) in the learning process. Here, we vary ι value in a range of [2,5,10,20,30,50], and present the corresponding learning results in Fig.<ref>. It shows the model's classification performance experiences fluctuations during the initial few iterations but achieves stability beyond ι=10 iterations.

Further, we explore how the μ value is adjusted during the learning process. As shown in Table <ref>, the μ value dynamically adapts, taking into account the importance of both marginal and conditional distributions. It helps the learning process adapts to the internal structure of data and promotes a deeper understanding and representation of the data being processed.















§ CONCLUSION

In this paper, we present a novel non-deep transfer learning framework, termed as MDDD, for cross-subject and cross-session EEG-based emotion recognition. A manifold-based domain adaptation with dynamic distribution is proposed, where the importance of marginal and conditional distributions is adaptively adjusted during the feature alignment process based on Grassmann manifold space. Manifold feature transformation evidently offers a better description of the intrinsic properties of the data and is beneficial in addressing the disparity between the source and target domains. Optimized classifier learning in conjunction with ensemble learning helps generate more robust and reliable emotion recognition performance, especially for processing EEG data with various variations in the data and labels. Extensive experiments are conducted on two well-known databases, considering two experimental protocols: leave-one-subject-out cross-subject single-session and leave-one-subject-out cross-subject cross-session. Compared to the existing literature, the proposed MDDD model achieves promising results relative to non-deep methods and achieves comparable results to deep methods. An average improvement of 3.54% in classification accuracy is observed. The results demonstrate the potential of using non-deep transfer learning to alleviate individual and session differences in affective brain-computer interfaces, consequently extending the possibilities for adoption in real-world applications.






§ ACKNOWLEDGMENTS

This work was supported in part by the National Natural Science Foundation of China under Grant 62276169 and 62071310, in part by the Medical-Engineering Interdisciplinary Research Foundation of Shenzhen University, in part by the Shenzhen University-Lingnan University Joint Research Programme, in part by the Shenzhen-Hong Kong Institute of Brain Science-Shenzhen Fundamental Research Institutions under Grant 2022SHIBS0003.











































IEEEtran





























 














[
    < g r a p h i c s >
]Ting Luo

 is a master's student in the Department of Biomedical Engineering at Shenzhen University, China. Her research interests include non-deep transfer learning, visual attention and affective brain-computer interface.
 




[
    < g r a p h i c s >
]Jing Zhang

received her Ph.D. degree from the China Medical University in 2018. She was a visiting scholar at the University of North Carolina at Chapel Hill during 2019-2021. She was an assistant professor at the First Affiliated Hospital of China Medical University. She has been working in the clinical field of adolescent health for more than 10 years. She is currently the director of the Department of Maternal, Child, and Adolescent Health at Shenzhen University Medical School. Her current research interests include mental disorders and big data analysis.
 





[
    < g r a p h i c s >
]Yingwei Qiu

received his Ph.D. degree from the Southern Medical University, Guangzhou, in 2011. From 2011 to 2014, he was a radiologist at Guangdong No.2 provincial people hospital. From 2015 to 2016, he was a post-doctoral at Duke-NUS graduate medical school, Singapore. From 2017 to 2021, he was a specially-appointed professor of Guangzhou Medical University, Guangzhou. He is currently a chief physician at the affiliated Nanshan Hospital of Shenzhen University. His current research interests include neural basis of affective disorders and AI application in radiology.
 




[
    < g r a p h i c s >
]Li Zhang

received his Ph.D. degree from the Department of Electrical and Electronics, University of Hong Kong in 2017 and joined the School of Biomedical Engineering, Health Science Center, Shenzhen University in 2018 as an associate researcher. His current research interests mainly focus on biomedical signal processing, numerical optimization, and imaging genetics.
 





[
    < g r a p h i c s >
]Yaohua Hu

received the Ph.D. degree in Applied Mathematics from Hong Kong Polytechnic University in 2013. From 2013 to 2015, he was a Postdoctoral Fellow in Hong Kong Polytechnic University and Zhejiang University. From 2015, he joined School of Mathematical Sciences at Shenzhen University, and now is a distinguished professor there. His current research interests include theory and algorithms for large-scale optimization, and applications in machine learning and bioinformatics.
 





[
    < g r a p h i c s >
]Zhuliang Yu
received his BSEE in 1995 and MSEE in 1998, both in electronic engineering from the Nanjing University of Aeronautics and Astronautics, China. He received his Ph.D. in 2006 from Nanyang Technological University, Singapore. He joined Center for Signal Processing, Nanyang Technological University from 2000 as a research engineer, then as a Group Leader from 2001. In 2008, he joined the College of Automation Science and Engineering, South China University of Technology and was promoted to be a full professor in 2010. Now, he serves as the Executive Dean of  Shien-Ming Wu School of Intelligent Engineering,  the Executive Director of Institute for Super Robotics. His research interests include signal processing, machine learning and their applications in biomedical engineering, intelligent robotics etc.






[
    < g r a p h i c s >
]Zhen Liang
received her Ph.D. degree from The Hong Kong Polytechnic University, Hong Kong, in 2013. From 2012 to 2017, she was an algorithm development scientist at NeuroSky, Inc., Hong Kong. From 2018 to 2019, she was a specially‐appointed assistant professor of Graduate School of Informatics, Kyoto University, Japan. She is currently an associate professor in the School of Biomedical Engineering, Health Science Center, Shenzhen University, China. Her current research interests include brain encoding and decoding systems, affective computing, visual attention, and neural engineering.



















































