Semi-supervised Deep Generative Modelling of Incomplete Multi-Modality Emotional Data
Changde Du Changying Du Hao Wang Jinpeng Li Wei-Long Zheng Bao-Liang Lu Huiguang He










  Research Center for Brain-inspired Intelligence & National Laboratory of Pattern Recognition, CASIA
  University of CAS, Beijing, China

duchangde@gmail.com


1234-5678-9012

  360 Search Lab

  Beijing
  China

ducyatict@gmail.com
cashenry@126.com



  Research Center for Brain-inspired Intelligence & National Laboratory of Pattern Recognition, CASIA
  University of CAS, Beijing, China
  
lijinpeng2015@ia.ac.cn



  Department of Computer Science and Engineering, SJTU
  Shanghai
  China

weilong@sjtu.edu.cn










  Department of Computer Science and Engineering, SJTU
  Shanghai
  China

bllu@sjtu.edu.cn


Dr. Huiguang He is the corresponding author and he is also with the Center for Excellence in Brain Science and Intelligence
Technology, Chinese Academy of Sciences and the
University of Chinese Academy of Sciences.


  Research Center for Brain-inspired Intelligence & National Laboratory of Pattern Recognition, CASIA

huiguang.he@ia.ac.cn






There are threefold challenges in emotion recognition. First, it is difficult to recognize human's emotional states only considering a single modality.  Second, it is expensive to manually annotate the emotional data. Third, emotional data often suffers from missing modalities due to unforeseeable sensor malfunction or configuration issues.  In this paper, we address all these problems under a novel multi-view deep generative framework. Specifically, we propose to model the statistical relationships of multi-modality emotional data using multiple modality-specific generative networks with a shared latent space. By imposing a Gaussian mixture assumption on the posterior approximation of the shared latent variables, our framework can learn the joint deep representation from multiple modalities and evaluate the importance of each modality simultaneously. To solve the labeled-data-scarcity problem, we extend our multi-view model to semi-supervised learning scenario by casting the semi-supervised classification problem as a specialized missing data imputation task.  To address the missing-modality problem, we further extend our semi-supervised multi-view model to deal with incomplete data, where a missing view is treated as a latent variable and integrated out during inference. This way, the proposed overall framework can utilize all available (both labeled and unlabeled, as well as both complete and incomplete) data to improve its generalization ability. The experiments conducted on two real multi-modal emotion datasets demonstrated the superiority of our framework.






Semi-supervised Deep Generative Modelling of Incomplete Multi-Modality Emotional Data
    Huiguang He
    December 30, 2023
=====================================================================================





§ INTRODUCTION

With the development of human-computer interaction (HCI), emotion recognition has become increasingly important. Since human's emotion contains many nonverbal cues, various modalities ranging from facial expressions, body gesture, voice to physiological signals can be used as the indicators of emotional states  <cit.>.
In real-world applications, it is difficult to recognize human's emotional states only considering a single modality, because signals from different modalities represent different aspects of emotion and provide complementary information.  Recent studies show that integrating multiple modalities can significantly boost the emotion recognition accuracy  <cit.>.
The most successful approach to fuse the information from multiple modalities is based on deep multi-view representation learning  <cit.>. E.g.,  <cit.> proposed a joint density model for emotion analysis with a multi-modal deep Boltzmann machine (DBM)  <cit.>. This multi-modal DBM is exploited to model the joint distribution over visual, auditory, and textual features.
 <cit.> proposed a multi-modal emotion recognition method by using multi-modal autoencoders (MAE)  <cit.>, in which the joint representations of Electroencephalogram (EEG) and eye movement signals were extracted.
Nevertheless, there are still limitations with these deep multi-modal emotion recognition methods, e.g., their performances depend on the amount of labeled data and they could not handle incomplete data.

By using the modern sensing equipments, we can easily collect massive emotion-related data from multiple modalities. But, the data labeling procedure requires lots of manual efforts. Therefore, in most cases only a small set of labeled samples is available, while the majority of whole dataset is left unlabeled.  In addition to challenges with insufficient labeled data, one must often address the incomplete-data problem, i.e., not all modalities are available for every data point. Generally, we can identify various causes for incomplete data. E.g., unforeseeable sensor malfunction may fail to collect sensing information, thus providing us incomplete data with one or more missing modalities. Traditional multi-modal emotion recognition approaches  <cit.> only utilized the limited amount of labeled data, which may result in severe overfitting.  Also, most of them neglect the missing modality issue, which greatly limits their applications in real-world scenarios.  The most attractive way to deal with the aforementioned issues is semi-supervised learning (SSL) with incomplete data. SSL can improve model's generalization ability by exploiting both labeled and unlabeled data simultaneously  <cit.>, and learning from incomplete data can guarantee the robustness of the emotion recognition system  <cit.>.


In this paper, we show that the problems mentioned above can be resolved under a unified multi-view deep generative framework. For modeling the statistical relationships of multi-modality emotional data, a shared latent variable is transformed by different modality-specific generative networks to different data views (modalities).
Instead of treating each view equally, we impose a non-uniformly weighted Gaussian mixture assumption on the posterior approximation of the shared latent variables.  This is critical for inferring the joint latent representation and the weight factor of each view from multiple modalities.  During optimization, a second lower bound to the variational lower bound is derived to address the intractable entropy of a mixed Gaussians.
To leverage the contextual information in the unlabeled data to augment the limited labeled data, we then extend our multi-view framework to SSL scenario. It is achieved by casting the semi-supervised classification problem as a specialized missing data imputation task.  Specifically, we treat the unknown labels as latent variables and estimate them within a multi-view auto-encoding variational Bayes framework.
We further extend the proposed SSL algorithm to the incomplete-data case by introducing latent variables for the missing views. Besides the unknown labels, the missing views are also integrated out so that the marginal likelihood is maximized with respect to model parameters. In this way, our SSL algorithm can utilize all available data: both labeled and unlabeled, as well as both complete and incomplete. Since the category information and the uncertainty of missing view are taken into account in the training process, our SSL algorithm is more powerful than traditional missing view imputation methods  <cit.>. We finally demonstrate the superiority of our framework and provide insightful observations on two real multi-modal emotion datasets.




§ RELATED WORK


Multi-modal approaches have been widely implemented for emotion recognition  <cit.>. E.g.,  <cit.> used a multi-modal deep belief network (DBN) to extract features from face, body gesture, voice and physiological signals for emotion classification.  <cit.> classified the combination of EEG and eye movement signals into three affective states. But, very few of them explored SSL. To the best of our knowledge, only  <cit.> proposed an enhanced multi-modal co-training algorithm for semi-supervised emotion recognition, but its shallow structure is hard to capture the high-level correlation between different modalities. In addition, most prior work in this field assumes that all modalities are available at all times  <cit.>, which is not realistic in practical environments. In contrast to the above methods, our framework naturally allows us to perform multi-modal emotion recognition within SSL and incomplete-data situations.



The variational autoencoder (VAE)  <cit.> is one of the most popular deep generative models (DGMs). VAE has shown great advantages in semi-supervised classification  <cit.>. E.g., Kingma et al.  <cit.> proposed a semi-supervised VAE (M2) by modeling the joint distribution over data and labels.   proposed the auxiliary DGMs (ADGM and SDGM)  <cit.> by introducing auxiliary variables, which improve the variational approximation.

However, these models cannot effectively deal with multi-view data, especially in incomplete-view case.
Our proposed semi-supervised multi-view DGMs distinguish our method from all existing ones using VAE framework  <cit.>.



Incomplete-data problem is often circumvented via imputation methods  <cit.>. Common imputation schemes include matrix completion  <cit.> and autoencoder-based methods  <cit.>. Matrix completion methods, such as SoftImputeALS  <cit.>, focus on imputing the missing entries of a partially observed matrix based on assumption that the completed matrix has a low-rank structure. Matrix completion methods often assume data is missing at random (MAR), which might not be optimal for our problem where modalities are missing at continuous blocks. On the other hand, autoencoder-based methods, such as DCCAE  <cit.> and CorrNet  <cit.>, exploit the connections between views, enabling the incomplete view to be restored with the help of the complete view.
Besides low-rank structure of the data matrix and the connections between views, category information is also important for missing view imputation tasks, though category labels may be partially observed. So far, very few algorithms  <cit.> can estimate the missing view under the SSL scenario. Although CoNet  <cit.> utilized deep neural networks (DNNs) to predict the missing view based on existing views and partially observed labels, its feedforward structure could not integrate multiple views effectively in classification. Additionally, most previous works treat the missing data as fixed values and hence ignore the uncertainty of the missing data. Unlike them, our SiMVAE essentially performs infinite imputations by integrating out the missing data.




§ METHODOLOGY

In this section, we first develop a multi-view variational autoencoder (MVAE) model for fusing multi-modality emotional data. Based on MVAE, we further build a semi-supervised emotion recognition algorithm. Finally, we develop a more robust semi-supervised algorithm to address the incomplete multi-modality emotional data.
For simplicity we restrict further discussion to the case of two views, though all the proposed methods can be extended to
more than two views. Assume we are faced with multi-view data that appears as pairs (𝔛,  y) = ({𝐱^(v)}_v=1^2,  y), with  observation 𝐱^(v) from the v-th view and the corresponding class label y.




 §.§ Multi-view Variational Autoencoder



  §.§.§ DNN-parameterized Likelihoods

We assume that multiple data views (modalities) {𝐱^(v)}_v=1^2 are generated independently from a shared latent space with  multiple view-specific generative networks.
Specifically, we assume a shared latent variable 𝐳 generates 𝐱^(v) with the following generative model P_1 (cf. Figure <ref>a):

    p_θ^(v)(𝐱^(v)|𝐳)     = f(𝐱^(v); 𝐳, θ^(v)),    v∈{1, 2},

where f(𝐱^(v); 𝐳, θ^(v)) is a suitable likelihood function (e.g. a Gaussian for continuous observation or Bernoulli for binary observation), which is formed by a non-linear transformation of the latent variable 𝐳. This non-linear transformation is essential to allow for higher moments of the data to be captured by the density model, and we choose these non-linear functions to be DNNs, referred to as the generative networks, with parameters {θ^(v)}_v=1^2. Note that, the likelihoods for different views are assumed to be independent of each other, with  potentially different DNN types for different modalities.




  §.§.§ Gaussian Prior and Gaussian Mixture Posterior

In vanilla VAE  <cit.>, which can only handle single-view data, both the prior p(𝐳) and the approximate posterior  q_ϕ(𝐳|𝔛) are assumed to be Gaussian distributions in order to maintain mathematical and computational tractability. Although this assumption has leaded to favorable results on several tasks, it is clearly a restrictive and often unrealistic assumption. Specifically, the choice of a Gaussian distribution for p(𝐳) and q_ϕ(𝐳|𝔛) imposes a strong uni-modal structure assumption on the latent space. However, for data distributions that are strongly multi-modal, the uni-modal Gaussian assumption inhibits the model's ability to extract and represent important structure in the data.
To improve the flexibility of the model, one way is to impose a Mixture of Gaussians (MoG) assumption on p(𝐳). However, it has the risk of creating separate “islands" of discontinuous manifolds that may break the meaningfulness of the representation in the latent space.

To learn more powerful and expressive models (in particular, models with multi-modal latent variable structures for multi-modal emotion recognition
applications) we seek a MoG for q_ϕ(𝐳|𝔛), while preserving p(𝐳) as a standard Gaussian. Thus the prior distribution and the inference model Q_1 (cf. Figure <ref>b) are defined as: p(𝐳) = 𝒩(𝐳|0, 𝐈),

    q_ϕ(𝐳|𝔛)      = ∑_v=1^2λ^(v)𝒩(𝐳|μ_ϕ^(v)(𝐱^(v)), Σ_ϕ^(v)(𝐱^(v)) ),

where the mean μ_ϕ^(v) and the covariance Σ_ϕ^(v) are nonlinear functions of the observation 𝐱^(v), with variational parameter ϕ^(v). As in our generative model, we choose these nonlinear functions to be DNNs, referred to as the inference networks. λ^(v) is the non-negative normalized weight factor for the v-th view, i.e., λ^(v) > 0 and ∑_v=1^2λ^(v) =1. Note that, Gershman et al. <cit.> proposed a nonparametric variational inference method by simply assuming the variational distribution to be a uniformly weighted Gaussian mixture. However, treating each component equally will lose flexibility in fusing multiple data views. Instead of treating each view equally, our non-uniformly weighted Gaussian mixture assumption can weight each view automatically in subsequent emotion recognition tasks, which is useful to identify the importance of each view.




 §.§ Semi-supervised Multi-modal Emotion Recognition

Although many supervised emotion recognition algorithms exist (see  <cit.> for a thorough literature review), very few
semi-supervised algorithms have been proposed to improve the recognition performance by utilizing both labeled and unlabeled data.
Here we extend MVAE by introducing a conditional probabilistic distribution for the unknown labels to obtain a semi-supervised multi-view classification algorithm.



  §.§.§ Generative model P_2

Since the emotional data is continuous, we choose the Gaussian likelihoods.
Then our generative model P_2 (cf. Figure <ref>c) is defined as p(y)p(𝐳)∏_v=1^2p_θ^(v)(𝐱^(v)|y, 𝐳):

    p(y)    = Cat(y|π),      p(𝐳) = 𝒩(𝐳|0, 𝐈),
     p_θ^(v)(𝐱^(v)|y, 𝐳)     = 𝒩(μ_θ^(v)(y, 𝐳), diag(σ^2_θ^(v)(y, 𝐳)) ),

where Cat(·) denotes the categorical distribution, y is treated as a latent variable for the unlabeled data points,

and the mean μ_θ^(v) and variance σ^2_θ^(v) are nonlinear functions of y and 𝐳, with parameter θ^(v).




  §.§.§ Inference model Q_2

The inference model Q_2 (cf. Figure <ref>d) is defined as q_φ(y|𝔛)q_ϕ(𝐳|𝔛, y):

    q_φ(y|𝔛)    = Cat(y|π_φ(𝔛)),
     q_ϕ(𝐳|𝔛, y)     = ∑_v=1^2λ^(v)𝒩(𝐳|μ_ϕ^(v)(𝐱^(v), y), Σ_ϕ^(v)(𝐱^(v), y) ),

where q_φ(y|𝔛) is the introduced conditional distribution for y, and q_ϕ(𝐳|𝔛, y) is assumed to be a mixture of Gaussians to combine the information from multiple data views and the label. Intuitively,  q_ϕ(𝐳|𝔛, y), p_θ^(v)(𝐱^(v)|y, 𝐳) and q_φ(y|𝔛) correspond to the encoder, decoder and classifier, respectively.
For brevity, we omit the explicit dependencies on 𝐱^(v), y and 𝐳 for the moment variables mentioned above hereafter. In principle, μ_θ^(v), σ^2_θ^(v), π_φ, μ_ϕ^(v) and Σ_ϕ^(v)  can be implemented by various DNN models, e.g., multi-layer perceptrons and convolutional neural networks.




  §.§.§ Objective function


In the semi-supervised setting, there are two lower bounds for the labeled and unlabeled cases, respectively.
The variational lower bound on the marginal likelihood for a single labeled data point is

    log p_θ(𝔛, y)    ≥𝔼_q_ϕ(𝐳|𝔛, y)[log p_θ(𝔛, y, 𝐳)/q_ϕ(𝐳|𝔛, y)]  
       ≥𝔼_q_ϕ(𝐳|𝔛,y)[∑_v=1^2log p_θ^(v)(𝐱^(v)|y, 𝐳) + log p(y)
        + log p(𝐳) ] - ∑_v=1^2 λ^(v)·log(∑_l=1^2 λ^(l)·ω_v, l) 
       ≡ -ℒ(𝔛, y),

where ω_v, l = 𝒩( μ_ϕ^(v) | μ_ϕ^(l), Σ_ϕ^(v) + Σ_ϕ^(l)). It should be noted that, the Shannon entropy 𝔼_q_ϕ(𝐳|𝔛,y)[- log q_ϕ(𝐳|𝔛,y)] is hard to compute analytically, and we have used the Jensen's inequality to derive a lower bound of it (see Supplementary Material Section A for details).
For unlabeled data point, the variational lower bound on the
marginal likelihood can be given by:

    log p_θ(𝔛)    ≥𝔼_q_φ, ϕ(y, 𝐳|𝔛)[log p_θ(𝔛, y, 𝐳)/q_φ, ϕ(y, 𝐳|𝔛)]  
        = 𝔼_q_φ(y|𝔛)[-ℒ(𝔛,y)- log q_φ(y|𝔛)]
    ≡ -𝒰(𝔛),

with q_φ, ϕ(y, 𝐳|𝔛) = q_φ(y|𝔛) q_ϕ(𝐳|𝔛, y).

Therefore, the objective function for the entire dataset is:

    𝒥_SMVAE  = ∑_(𝔛,y) ∈ S_lℒ(𝔛,y)_labeled +  ∑_𝔛∈ S_u𝒰(𝔛)_unlabeled,

where S_l and S_u denote labeled and unlabeled dataset, respectively. The classification accuracy can be improved by introducing
an explicit classification loss for labeled data, and the extended objective function is  now:

    ℱ_SMVAE  = 𝒥_SMVAE + α·∑_(𝔛,y) ∈ S_l[-log q_φ(y|𝔛)]  ,

where α is a weight parameter between generative and discriminative learning. We set α = c ·(N_l + N_u)/N_l, where c is a scaling constant, and N_l and N_u are the numbers of labeled and unlabeled data points in one minibatch, respectively.
Note that, the classifier q_φ(y|𝔛) is also used at test phase for the prediction of unseen data.
Eq. (<ref>) provides a unified objective function for optimizing the parameters of encoder, decoder and classifier networks.



  §.§.§ Parameter optimization

Parameter optimization can be done jointly by using the stochastic backpropagation technique  <cit.>.
The reparameterization trick  <cit.> is a vital component, because it allows us to take derivative of  𝔼_q_ϕ(𝐳|𝔛,y)[log p_θ^(v)(𝐱^(v)|y, 𝐳)] w.r.t. the variational parameters ϕ.
However, the use of Gaussian mixture for variational posterior distribution q_ϕ(𝐳|𝔛,y) makes it infeasible to apply the reparameterization trick directly.
It can be shown that, for any v ∈{1, 2}, 𝔼_q_ϕ(𝐳|𝔛,y)[log p_θ^(v)(𝐱^(v)|y, 𝐳)]  can be rewritten, using the location-scale transformation for the Gaussian distribution, as:

    𝔼_q_ϕ(𝐳|𝔛,y)[log p_θ^(v)(𝐱^(v)|y, 𝐳)] 
    
         = ∑_l=1^2 λ^(l)𝔼_𝒩(ϵ^(l)|0,𝐈)[log p_θ^(v) (𝐱^(v)|y, μ_ϕ^(l) + 𝐑_ϕ^(l)ϵ^(l) ) ],

where 𝐑_ϕ^(l)𝐑_ϕ^(l)^⊤ = Σ_ϕ^(l) and  l∈{1, 2}.
While the expectations on the right hand side still cannot be solved analytically, their gradients w.r.t. θ^(v), ϕ^(l) and λ^(l) can be efficiently estimated using Monte-Carlo method (see Supplementary Material Section B for details).
The gradients of the objective function (Eq. (<ref>)) can then be computed by using the chain rule and the derived Monte-Carlo estimators.


 §.§ Handling Incomplete Data

In the above discussion it is assumed that all modalities are available for every data point. In practice, however, many samples generally have incomplete modalities (i.e., with one or more missing modalities)  <cit.>. In light of this, we further develop a semi-supervised incomplete multi-view classification algorithm (SiMVAE).

For simplicity, we assume only one view (either 𝐱^(1) or 𝐱^(2)) is incomplete, though our model can be easily extended to more sophisticated cases.  We partition each data point into an observed view 𝐱^o and a missing view 𝐱^m (i.e., 𝔛 = {𝐱^o, 𝐱^m}).



  §.§.§ Generative model P_3

In this setting, only a subset of the samples have complete views and corresponding labels. We regard both the unknown label y and the missing view 𝐱^m as latent variables.

Then our generative model P_3 (cf. Figure <ref>e) is defined as p(y)p(𝐳) p_θ^m(𝐱^m|y, 𝐳) p_θ^o(𝐱^o|y, 𝐳, 𝐱^m):

    p_θ^m(𝐱^m|y, 𝐳)     = 𝒩(μ_θ^m(y, 𝐳), diag(σ^2_θ^m(y, 𝐳)) ),
     p_θ^o(𝐱^o|y, 𝐳, 𝐱^m)     = 𝒩(μ_θ^o(y, 𝐳, 𝐱^m), diag(σ^2_θ^o(y, 𝐳, 𝐱^m)) ),

where p_θ^m(·) and p_θ^o(·) are DNNs with parameters θ^m and θ^o, respectively. p(y) and p(𝐳) are defined as in Eq. (<ref>).



  §.§.§ Inference model Q_3

As multi-modality emotional data are collected from the same subject, there must be some underlying relationships between modalities, though they focus on different information. Given the observed modality, the estimation of missing modality is feasible if we capture the relationships between modalities. Therefore, the inference model Q_3 (cf. Figure <ref>f) is defined as q_ψ(𝐱^m|𝐱^o)q_φ(y|𝔛)q_ϕ(𝐳|𝔛, y), with

    q_ψ(𝐱^m|𝐱^o) = 𝒩(μ_ψ(𝐱^o), diag(σ^2_ψ(𝐱^o)) ),

where q_ψ(·) is a DNN with parameter ψ. q_φ(y|𝔛) and q_ϕ(𝐳|𝔛, y) are defined as in Eq. (<ref>). Intuitively, we formulate the missing view imputation as a conditional distribution estimation task (conditioned on the observed view). Compared with existing single imputation methods  <cit.>, our model essentially performs infinite imputations and hence takes the uncertainty of the missing data into account.  To obtain a single imputation of 𝐱^m rather than the full conditional distribution one can evaluate 𝐱^m = 𝔼[ q_ψ(𝐱^m| 𝐱^o)].



  §.§.§ Objective function

In semi-supervised incomplete multi-view setting, there are four lower bounds for the labeled-complete, labeled-incomplete, unlabeled-complete and unlabeled-incomplete cases, respectively.

Similar to Eq. (<ref>), the variational lower bound on the marginal likelihood for a single labeled-complete data point is

    log p_θ(𝔛, y)
    
    
    
    
    
       ≥𝔼_q_ϕ(𝐳|𝔛, y)[log p_θ^o(𝐱^o| 𝐱^m, y, 𝐳)  + log p(y) 
        + log p_θ^m(𝐱^m|y, 𝐳) + log p(𝐳) ] - ∑_v=1^2 λ^(v)·log(∑_l=1^2 λ^(l)·ω_v, l) 
       ≡ -ℒ𝒞(𝔛, y),

where ω_v, l = 𝒩( μ_ϕ^(v) | μ_ϕ^(l), Σ_ϕ^(v) + Σ_ϕ^(l)).
In the labeled-incomplete context, the variational lower bound on the
marginal likelihood for a single data point can be given by:

    log p_θ(𝐱^o, y)
    
    ≥∫_𝐳∫_𝐱^mlog p_θ(𝔛, y, 𝐳) d 𝐳 d 𝐱^m
    
        = 𝔼_q_ψ(𝐱^m| 𝐱^o)[   -ℒ𝒞(𝔛, y)  - log q_ψ(𝐱^m| 𝐱^o) ]
     ≡ -ℒℐ(𝐱^o, y).

The solution to 𝔼_q_ψ(𝐱^m| 𝐱^o)[ - log q_ψ(𝐱^m| 𝐱^o)] is analytical since the conditional distribution q_ψ(𝐱^m| 𝐱^o) is assumed to be a Gaussian (cf. Eq. (<ref>)).
For unlabeled-complete data point, the variational lower bound on the
marginal likelihood can be obtained by

    log p_θ(𝔛)
    
    ≥∫_𝐳∫_ylog p_θ(𝔛, y, 𝐳) d 𝐳 d y
    
         = 𝔼_q_φ(y|𝔛)[   -ℒ𝒞(𝔛, y)  - log q_φ(y|𝔛) ]
     ≡ -𝒰𝒞(𝔛).


For unlabeled-incomplete case, the variational lower bound on the
marginal likelihood can be given by:

    log p_θ(𝐱^o)
    
    ≥∫_𝐳∫_y∫_𝐱^mlog p_θ(𝔛, y, 𝐳) d 𝐳 d y d 𝐱^m
    
        = 𝔼_q_ψ(𝐱^m| 𝐱^o){𝔼_q_φ(y|𝔛)[   -ℒ𝒞(𝔛, y) - log q_φ(y|𝔛) ] 
        - q_ψ(𝐱^m| 𝐱^o)}≡ -𝒰ℐ(𝐱^o).


Comparing to Eq. (<ref>) we see that aside from the explicit conditional distribution for unknown label y we have added a conditional distribution q_ψ(𝐱^m| 𝐱^o) for missing view 𝐱^m.


The objective function for all available data points is now:

    𝒥_SiMVAE =     ∑_(𝔛,y) ∈ S_lcℒ𝒞(𝔛,y)_labeled-complete + ∑_(𝐱^o,y) ∈ S_liℒℐ(𝐱^o,y)_labeled-incomplete
        + ∑_𝔛∈ S_uc𝒰𝒞(𝔛)_unlabeled-complete + ∑_𝐱^o∈ S_ui𝒰ℐ(𝐱^o)_unlabeled-incomplete.


Model performance can be improved by introducing explicit imputation loss and classification loss for complete data and labeled data, respectively. Therefore, the final objective function is

    ℱ_SiMVAE    = 𝒥_SiMVAE + α_1 ·∑_𝔛∈ S_c[-log q_ψ(𝐱^m| 𝐱^o)]
        + α_2 ·∑_(𝔛,y) ∈ S_l[-log q_φ(y|𝔛)],

where α_1 and α_2 are weight parameters, S_c = S_lc∪ S_uc and S_l = S_lc∪ S_li. We set α_1 = c_1 ·(N_c + N_i)/N_c and α_2 = c_2 ·(N_l + N_u)/N_l, where c_1 and c_2 are scaling constants, and N_c, N_i, N_l and N_u are the numbers of complete, incomplete, labeled and unlabeled data in one minibatch, respectively.  Noted that the explicit
classification loss (i.e., last term in Eq. (<ref>)) allows SiMVAE to use the partially observed category information to assist the generation of 𝐱^m given 𝐱^o, which is more effective than the unsupervised imputation algorithms  <cit.>.
Similarly, Eq. (<ref>) can be optimized by using the stochastic backpropagation technique  <cit.>.

In principle, our SiMVAE can also handle multiple missing views simultaneously.  The formulas are omitted here since they can be derived straightforwardly by using multiple distinct conditional density functions q_ψ(𝐱^m| 𝐱^o).



  §.§.§ Connections to auxiliary deep generative models

  <cit.> proposed auxiliary DGMs (ADGM and SDGM) by defining the inference model as q_ψ(𝐚|𝐱^o)q_φ(y|𝐚, 𝐱^o)q_ϕ(𝐳|𝐚, 𝐱^o, y), where 𝐚 is the  auxiliary variable introduced to make the variational distribution more expressive, and q_ψ(𝐚|𝐱^o)=𝒩(μ_ψ(𝐱^o),diag(σ^2_ψ(𝐱^o))).
If 𝐱^m is a totally unobservable variable in Figures <ref>e and <ref>f, similar to SDGM, SiMVAE becomes a two-layered stochastic model. Since the generative process is conditioned on the auxiliary variable, two-layered stochastic model is more flexible than ADGM  <cit.>. Standard ADGM and SDGM could not handle incomplete multi-view data. We endow them with this ability by forcing the inferred auxiliary variable 𝐚 close to 𝐱^m on the set of complete data. E.g., we can obtain the objective function of SDGM+ by introducing an additional imputation loss to SDGM:

    ℱ_SDGM+    = 𝒥_SDGM + α_3·∑_𝔛∈ S_c[-log q_ψ(𝐚| 𝐱^o)],

where α_3 is a regularization parameter, 𝔛={𝐱^m, 𝐱^o} and S_c denotes the set of complete data. 𝒥_SDGM can be found in  <cit.>. Intuitively,  SDGM+ not only enjoys the advantages of SDGM (in terms of flexibility, convergence and performance), but also captures the relationships between views via the auxiliary inference model q_ψ(𝐚| 𝐱^o).  However, SDGM+ sets a single Gaussian in the variational distribution q_ϕ(𝐳|𝐚, 𝐱^o, y), which may restrict its ability in multi-modality fusion.




§ EXPERIMENTS

We conduct experiments on two multi-modal emotion datasets to demonstrate the effectiveness of the proposed framework.


 §.§ Datasets

SEED:  The SEED dataset  <cit.> contains Electroencephalogram (EEG) and eye movement (Eye) signals from 9 subjects during watching 15 movie clips, where each movie clip lasts about 4 minutes long. The EEG signals were recorded from 62 channels and the Eye signals contained information about blink, saccade fixation and so on. We used the EEG and Eye data from 9 subjects across 3 sessions, totally 27 data files. For each data file, data from watching the 1-9 movie clips were used as training set, while data from watching the 10-12 movie clips were used as validation set and the rest (13-15) were used as testing set.

DEAP:The DEAP dataset   <cit.> contains EEG and peripheral physiological signals (PPS) from 32 subjects during watching 40 one-minute duration music videos. The EEG signals were recorded from 32 channels, whereas the PPS was recorded from 8 channels. The participants, using values from 1 to 9, rated each music video in terms of the levels of valence, arousal and so on. In our experiment, the valence-arousal space was divided into four quadrants according to the ratings. The threshold we used was 5, leading to four classes of data. Considering the variations of participants' ratings possibly associated with individual difference in rating scale, we discarded the samples whose ratings of arousal and valence are between 4 and 6. The dataset was randomly divided into 10-folds, where 8 folds for training, one fold for validation and the last fold for testing.

For SEED, we used the extracted differential entropy (DE) features and eye movement features (blink, saccade fixation and so on)  <cit.>. For DEAP, following  <cit.>, we split the time series data into many one-second non-overlapping segments, where each segment is treated as an instance. Then we extracted the DE features from EEG and PPS data instances. The DE features can be calculated in four frequency bands: theta (4-8Hz), alpha (8-14Hz), beta (14-31Hz), and gamma (31-45Hz), and we used all band's features. The details of the data used in our experiments were summarized in Table <ref>.



 §.§ Semi-supervised Classification with Multi-Modality Emotional Data



  §.§.§ Experimental setting

To simulate SSL scenario, on both datasets, we randomly labeled different proportions of samples in the training set, and remained the rest samples in the training set unlabeled.  For transductive SSL, we trained models on the dataset consisting of the testing data and labeled data belonging to training set. For inductive SSL, we trained models on the entire training set consisting of the labeled and unlabeled data. For supervised learning, we trained models on the labeled data belonging to training set, and test their performance on the testing set.
We compared our SMVAE with a broad range of solutions, including MAE  <cit.>, DCCA  <cit.>, DCCAE  <cit.>, AMMSS  <cit.>, AMGL  <cit.>, M2  <cit.> and SDGM  <cit.>. For SMVAE, we considered multi-layer perceptrons as the type of inference and generative networks. On both datasets, we set the hidden architectures of the inference and generative networks for each view as `100-50-30' and `30-50-100', respectively, where `30' is the dimension of the latent variables.

We used the Adam optimizer <cit.> with a learning rate η = 3×10^-4 in training. The scaling constant c was selected from {0.1, 0.5, 1}.

For MAE, DCCA and DCCAE, we considered the same setups (network structure, learning rate, etc.) as our SMVAE. Furthermore, we used support vector machines (SVM) and transductive SVM (TSVM) for supervised learning and transductive SSL, respectively.

For AMGL, M2 and SDGM we used their default settings, and we evaluated M2's performance on each modality and the concatenation of all modalities, respectively.




  §.§.§ Classification accuracy with very few labels

Table <ref> presents the classification accuracies of all methods on SEED and DEAP datasets. The proportions of labeled samples in the training set vary from  1% to 3%. Results (mean±std) were averaged over 20 independent runs. Several observations can be drawn as follows.
First, the average accuracy of SMVAE significantly surpasses the baselines in all cases.
Second, by examining SMVAE against supervised learning approaches trained on very limited labeled data, we can find that SMVAE always outperforms them. This encouraging result shows that SMVAE can effectively  leverage the useful information from unlabeled data.
Third,  multi-view semi-supervised algorithms AMMSS and AMGL perform worst in all cases. We attribute this to the fact that graph-based shallow models AMMSS and AMGL cannot extract the deep features from the original data.

Fourth, the performances of three TSVM-based semi-supervised methods are moderate.

Finally, compared with the single-view methods M2 and SDGM, our multi-view method is more effective in integrating multiple modalities.




  §.§.§ Flexibility and stability

The proportion of unlabeled samples in the training set will affect the performance of semi-supervised models. Figure <ref>a shows the changes of inductive SMVAE's average accuracy on SEED with different proportions of unlabeled samples in the training set. We can observe that the unlabeled samples can effectively boost the classification accuracy of SMVAE.
Instead of treating each modality equally, SMVAE can weight each modality and perform classification simultaneously. Figure <ref>b
shows the learned weight factors by inductive SMVAE on both datasets (1% labeled).  From it, we can observe that EEG modality has the highest weight on both datasets, which is consistent with single modality's performance of M2 shown in Table <ref> and the results in previous work <cit.>.
The scaling constant c controls the weight of discriminative learning in SMVAE. Figure <ref>c shows the performance of inductive SMVAE with different c values (1% labeled). From it, we can find that the scaling constant c can be chosen from {0.1, 0.5, 1}, where SMVAE achieves good results.



 §.§ Semi-supervised Learning with Incomplete Multi-Modality Data




  §.§.§ Experimental setting

To simulate the incomplete data setting, we randomly selected a fraction of instances (from both labeled and unlabeled training data) to be unpaired examples, i.e., they are described by only one modality, and the remaining ones appear in both modalities. We varied the fraction of missing data from 10% to 90% with an interval of 20%, while no missing data in validation and testing sets. In our experiment, we assumed the Eye modality of SEED and the PPS modality of DEAP are incomplete.

There are two main solutions for semi-supervised classification of incomplete multi-view data. One way is to complete the missing view firstly in an unsupervised way, and then conduct semi-supervised classification. Another way is to integrate missing view imputation and semi-supervised classification into an end-to-end learning framework. We compared our (inductive) SiMVAE algorithm with these two ways. Specifically, we compared SiMVAE with SoftImputeALS  <cit.>, DCCAE  <cit.>, CorrNet  <cit.>, CoNet  <cit.> and SDGM+ (a variant of SDGM  <cit.>, cf. Section <ref>).
For SoftImputeALS, DCCAE and CorrNet, we first estimated the missing modalities by using the authors' implementation, and then conducted semi-supervised classification by using our (inductive) SMVAE algorithm. For CoNet and SDGM+, we conducted missing modality imputation and semi-supervised classification simultaneously based on our own implementations.
Additionally, we also compared SiMVAE with the following two baselines:
1) SiMVAE with complete data (FullData, i.e., no missing modality for any training instances), which can be regarded as a upper bound of SiMVAE;
2) SiMVAE with only paired data (PartialData, i.e., we simply discard those incomplete samples in training process), which can be regarded as a lower bound of SiMVAE. These two bounds define the potential range of SiMVAE's performance.

 
For SiMVAE, both c_1 and c_2 were selected from {0.1, 0.5, 1}. For SDGM+, we selected the regularization parameter α_3 from {1e-3, 1e-2, ⋯, 1e3}.



  §.§.§ Semi-supervised classification

The performance of our SiMVAE and the compared methods was shown in Figure <ref>, where each point on every curve is an average over 20 independent trials. 
From Figure <ref>, it is seen that SiMVAE consistently outperforms the compared methods. Compared with the two-stage methods (SoftImputeALS, DCCAE and CorrNet), the advantage of SiMVAE is significant, especially when there are sufficient labeled data (3%). This is because SiMVAE can make good use of the available category information to generate more informative modalities, which in turn will improve classification performance. Whereas the two-stage methods couldn't obtain the global optimal results. Also, SiMVAE shows obvious advantage over the semi-supervised methods CoNet and SDGM+. This may be because CoNet and SDGM+ are not designed to integrate multiple modalities.
Moreover, SiMVAE has been successful even when a high percentage of samples are incomplete.
Specifically, SiMVAE with even about 50% incomplete samples achieves comparable results to the fully complete case (FullData). With fractions lower than that, we observe that SiMVAE roughly reached FullData's performance, especially when the labeled data are sufficient.
Finally, SiMVAE's performance is more closer to FullData than to PartialData, which indicates the effectiveness of SiMVAE in learning from incomplete data.



  §.§.§ Missing modality imputation

Since the quality of recovered missing modalities directly affects the classification results, we also evaluated the performance of missing modality imputation for all methods. For SiMVAE and SDGM+, we obtained the single imputation of 𝐱^m by evaluating the conditional mean (𝐱^m = 𝔼[ q_ψ(𝐱^m| 𝐱^o)]). We used the Normalized Mean Squared Error (NMSE) to measure the relative distance between the original and the recovered modalities. NMSE = 𝐗 - 𝐗̂_F/𝐗_F, where 𝐗 and 𝐗̂ are the original and the recovered data matrices, respectively. ·_F demotes the Frobenious norm. Figure <ref> shows the experimental results.



From Figure <ref>, it can be seen that as the fraction of missing data increases, the relative distance between the original modalities and the recovered modalities increases.  Further, the semi-supervised imputation methods (SiMVAE, CoNet and SDGM+) consistently outperforms the unsupervised imputation methods (SoftImputeALS, DCCAE and CorrNet), and increasing the number of labeled training data improves the imputation performance of semi-supervised methods. This demonstrates that the category information plays an important role in missing modality imputation. SoftImputeALS shows the worst performance, which verifies that matrix completion method is not suitable for missing modality imputation. CoNet and SDGM+ obtain comparable imputation errors to SiMVAE. This indicates that their moderate classification performance in Figure <ref> may be caused by their inability in modality fusion. Except for SiMVAE and SDGM+, other methods ignore the uncertainty of the missing view, which also limits their imputation performance.
To compare the imputation performance more intuitively, we visualize the original and recovered data matrices in Figure <ref> (on SEED, 3% labeled and 10% missing Eye). From it, we see that SiMVAE recovered more individual characteristics of the original data matrix than other methods.





  §.§.§ Sensitivity analysis

Figure <ref> shows the classification accuracies of inductive SiMVAE with different scaling constants c_1 and c_2 on both datasets (1% labeled and 10% missing data). From it, we can find that SiMVAE is not very sensitive to the values of c_1 and c_2. We choose the best c_1 and c_2 from {0.1, 0.5, 1} in the experiments.





§ CONCLUSION

We have proposed a novel semi-supervised multi-view deep generative framework for multi-modal emotion recognition with incomplete data. Under our framework, each modality of the emotional data is treated as one view, and the importance of each modality is inferred automatically by learning a non-uniformly weighted Gaussian mixture posterior approximation for the shared latent variable. The labeled-data-scarcity problem is naturally addressed within our framework through casting the semi-supervised classification problem as a specialized missing data imputation task. The incomplete-data problem is elegantly circumvented by treating the missing views as latent variables and integrating them out.
Compared with previous emotion recognition methods, our method is more robust and flexible.
Experimental results confirmed the superiorities of our framework over many state-of-the-art competitors.



This work was supported by National Natural Science Foundation of China (No. 91520202, 61602449), Beijing Municipal Science&Technology Commission (Z181100008918010), Youth Innovation Promotion Association CAS and Strategic Priority Research Program of CAS.




ACM-Reference-Format






Supplementary Material


In this document, we provide additional materials to supplement our main submission. In Section A, we provide further details on how we derive a second lower bound to variational lower bound. In Section B, we show Monte-Carlo estimators used to compute the gradients of the objective function.


§ SECTION A

The Shannon entropy 𝔼_q_ϕ(𝐳|𝔛,y)[- log q_ϕ(𝐳|𝔛,y)] is hard to compute analytically. In general, there is no closed-form expression for the entropy of a Mixture of Gaussians (MoG). Here we lower bound the entropy of MoG using Jensen's inequality:

    𝔼_q_ϕ(𝐳|𝔛,y)[- log q_ϕ(𝐳|𝔛,y)]  
        = - ∫ q_ϕ(𝐳|𝔛,y) log q_ϕ(𝐳|𝔛,y)  d 𝐳
        = - ∑_v=1^2 λ^(v)·∫𝒩(𝐳 | μ_ϕ^(v), Σ_ϕ^(v)) log∑_l=1^2 λ^(l)
       ·𝒩(𝐳 | μ_ϕ^(l), Σ_ϕ^(l))  d 𝐳
       ≥ - ∑_v=1^2 λ^(v)·log∑_l=1^2 λ^(l)·∫𝒩(𝐳 | μ_ϕ^(v), Σ_ϕ^(v))  
       ·𝒩(𝐳 | μ_ϕ^(l), Σ_ϕ^(l))  d 𝐳
        = - ∑_v=1^2 λ^(v)·log∑_l=1^2 λ^(l)·𝒩( μ_ϕ^(v) | μ_ϕ^(l), Σ_ϕ^(v) + Σ_ϕ^(l))
        = - ∑_v=1^2 λ^(v)·log(∑_l=1^2 λ^(l)·ω_v, l),

where we have used the fact that the convolution of two Gaussians is another Gaussian, and ω_v, l = 𝒩( μ_ϕ^(v) | μ_ϕ^(l), Σ_ϕ^(v) + Σ_ϕ^(l)).


§ SECTION B

𝔼_q_ϕ(𝐳|𝔛,y)[log p_θ^(v)(𝐱^(v)|y, 𝐳)]  can be rewritten, using the location-scale transformation for the Gaussian distribution, as:

    𝔼_q_ϕ(𝐳|𝔛,y)[log p_θ^(v)(𝐱^(v)|y, 𝐳)]
    
       = ∑_l=1^2 λ^(l)𝔼_𝒩(ϵ^(l)|0,𝐈)[log p_θ^(v) (𝐱^(v)|y, μ_ϕ^(l) + 𝐑_ϕ^(l)ϵ^(l) ) ],

where 𝐑_ϕ^(l)𝐑_ϕ^(l)^⊤ = Σ_ϕ^(l) and  l∈{1, 2}.
While the expectations on the right hand side of the above equation still cannot be solved analytically, their gradients w.r.t. θ^(v), ϕ^(l) and λ^(l) can be efficiently estimated using the following Monte-Carlo estimators

    ∂/∂θ^(v)𝔼_q_ϕ(𝐳|𝔛,y)[log p_θ^(v)(𝐱^(v)|y, 𝐳)] 
            = ∑_l=1^2  λ^(l)𝔼_𝒩(ϵ^(l) |0,𝐈)[∂/∂θ^(v)log p_θ^(v) (𝐱^(v)|y, 𝐳^(l) )] 
           ≈λ^(l)/T∑_t=1^T ∑_l=1^2 ∂/∂θ^(v)log p_θ^(v)(𝐱^(v)|y, 𝐳^(l,t)),

-0.1in

    ∂/∂ϕ^(l)𝔼_q_ϕ(𝐳|𝔛,y)[log p_θ^(v)(𝐱^(v)|y, 𝐳)] 
    
             = λ^(l)∂/∂ϕ^(l)𝔼_𝒩(ϵ^(l)|0,𝐈)[∂/∂𝐳^(l)log p_θ^(v)(𝐱^(v)|y, 𝐳^(l)) 
       ·( ∂μ_ϕ^(l)/∂ϕ^(l) + ∂𝐑_ϕ^(l)/∂ϕ^(l)ϵ^(l)) ] 
           ≈ λ^(l)/T∑_t=1^T  ∂/∂𝐳^(l,t)log p_θ^(v)(𝐱^(v)|y, 𝐳^(l,t))  
       ·( ∂μ_ϕ^(l)/∂ϕ^(l) + ∂𝐑_ϕ^(l)/∂ϕ^(l)ϵ^(l,t)),

-0.1in

    ∂/∂λ^(l)𝔼_q_ϕ(𝐳|𝔛,y)[log p_θ^(v)(𝐱^(v)|y, 𝐳)] 
             =  𝔼_𝒩(ϵ^(l)|0,𝐈)[log p_θ^(v) (𝐱^(v)|y, 𝐳^(l) ) ]
           ≈1/T∑_t=1^T  log p_θ^(v)(𝐱^(v)|y, 𝐳^(l,t)),

where 𝐳^(l) is evaluated at 𝐳^(l) = μ_ϕ^(l) + 𝐑_ϕ^(l)ϵ^(l) and 𝐳^(l,t) = μ_ϕ^(l) + 𝐑_ϕ^(l)ϵ^(l,t) with ϵ^(l,t)∼𝒩(0,𝐈). In practice, it suffices to use a small T (e.g. T = 1) and then estimate the gradient using minibatches of data points. Though the above Monte-Carlo estimators could have large variances if a small T is used, the experimental results show that it suffices to obtain good performance.

The same observation can be found in previous works <cit.>. Furthermore, we use the same random numbers ϵ^(l,t) for all estimators to have lower variances. The gradient w.r.t. φ is omitted here, since it can be derived straightforwardly by using traditional reparameterization trick <cit.>.

