Improving Bi-directional Generation between Different Modalities with Variational Autoencoders
Masahiro Suzuki Kotaro Nakayama Yutaka Matsuo
stat.ML cs.LG







Improving Bi-directional Generation 
 between Different Modalities with Variational Autoencoders
    Masahiro Suzukimasa@weblab.t.u-tokyo.ac.jp, Kotaro Nakayama, Yutaka Matsuo

The University of Tokyo, Japan

    
===============================================================================================================


We investigate deep generative models that can exchange multiple modalities bi-directionally, e.g., generating images from corresponding texts and vice versa. 
A major approach to achieve this objective is to train a model that integrates all the information of different modalities into a joint representation and then to generate one modality from the corresponding other modality via this joint representation. We simply applied this approach to variational autoencoders (VAEs), which we call a joint multimodal variational autoencoder (JMVAE).
However, we found that when this model attempts to generate a large dimensional modality missing at the input, the joint representation collapses and this modality cannot be generated successfully. Furthermore, we confirmed that this difficulty cannot be resolved even using a known solution.
Therefore, in this study, we propose two models to prevent this difficulty: JMVAE-kl and JMVAE-h. 
Results of our experiments demonstrate that these methods can prevent the difficulty above and that they generate modalities bi-directionally with equal or higher likelihood than conventional VAE methods, which generate in only one direction. Moreover, we confirm that these methods can obtain the joint representation appropriately, so that they can generate various variations of modality by moving over the joint representation or changing the value of another modality. 




§ INTRODUCTION



Information in our world is represented through various modalities. Although images are represented by pixel information, these can also be described with text or tag information. People often exchange such information bi-directionally. For instance, we can not only imagine what a “young female with a smile who does not wear glasses” looks like, but we can also add this caption to a corresponding photograph. Our objective is to design a model that can exchange different modalities bi-directionally like people. We call this ability bi-directional generation.



Each modality typically has a different kind of dimension and structure, e.g., images (real-valued and dense) and text (discrete and sparse). Therefore, the relations among modalities might have high nonlinearity. To discover such relations, deep neural network architectures have been used widely <cit.>.

A major approach to realizing bi-directional generation between modalities is to train the network architecture that shares the top of hidden layers in modality specific networks <cit.>. The salient advantages of this approach are that the model with multiple modalities can be trained end-to-end, and that the trained model can extract a joint representation, which is a more compact representation integrating all modalities. The model can easily generate one modality from another modality via this representation if it can obtain the joint representation properly. 
Another simple approach might be to create networks for each direction and to train them independently. Several models have been proposed to generate modalities in one direction <cit.>. However, when generating modalities bi-directionally, the number of required networks is expected to increase exponentially as the modality increases. Moreover, the hidden layers of each network would not be synchronized during training. Therefore, this simple approach is not efficient to realize our objective.

To generate different modalities, it is important to model the joint representation as probabilistic latent variables. This is because, as described above, different modalities have different structures and dimensions, so their relation should not be deterministic. The best known approach by this probabilistic manner is to use deep Boltzmann machines (DBMs) <cit.>. However, it is computationally difficult for DBMs to train especially high-dimensional data because of MCMC training. 

Variational autoencoders (VAEs) <cit.>, a deep generation model, have an advantage that they can handle higher-dimensional datasets than DBMs because back-propagation can be used to train them. Therefore, we extend VAEs to a model that be able to generate modalities bi-directionally. This extension method is extremely simple: as with previous neural networks and DBMs approaches, latent variables of generative models corresponding to each modality are shared. We call this model a joint multimodal variational autoencoder (JMVAE). However, results show that if we miss the input of the high-dimensional modality we want to generate, the latent variable, i.e. the joint representation, of JMVAE collapses and this modality cannot be generated successfully. Although a method for addressing this difficulty has been proposed <cit.>, we demonstrate that this method cannot resolve this difficulty when the missing modality has higher dimensions than another one.

Therefore, we propose two new models to address difficulty presented above: JMVAE-kl and JMVAE-h. JMVAE-kl takes an approach of preparing new encoders with one input for each modality apart from the encoder of JMVAE and reducing the divergence between them. By contrast, JMVAE-h makes its latent variable a stochastic hierarchical structure to prevent its collapse. Figure <ref> shows that JMVAE-kl can generate modalities bi-directionally with different dimensions such as images and attributes. 

The main contributions of this paper are described below.


  * We present a simple extension of VAEs to generate modality bi-directionally, which we call JMVAE. However, JMVAE cannot generate a high-dimensional modality well if the input of this modality is missing, and a known method of solution cannot resolve this issue.

  * We propose two models, JMVAE-kl and JMVAE-h, which prevent a latent variable from collapse when a high-dimensional modality is missing. We confirm experimentally that this method resolves this issue.

  * We demonstrate that these methods can generate modalities similarly or more properly than conventional VAEs that generate in only one direction. 

  * We demonstrate that they can appropriately obtain the joint representation containing different modality information, which shows that they can generate various variations of modality by moving over these latent variables or changing the value of another modality.




§ RELATED WORK
 

A common approach to dealing with multiple modalities in deep neural networks is to share the top of hidden layers in modality-specific networks. <cit.> proposes this approach with deep autoencoders, which revealed that it can extract better representations than single-modality settings can. Actually, <cit.> adapts this idea to work with deep Boltzmann machines (DBMs) <cit.>, which are generative models with undirected connections based on maximum joint likelihood learning of all modalities. Therefore, this model can generate modalities bi-directionally. Another study <cit.> improves this model to exchange multiple modalities effectively, which are based on minimizing the variation of information. 

Recently, VAEs <cit.> have been used to train such high-dimensional modalities. Several studies <cit.> have examined the use of conditional VAEs (CVAEs), which maximize a conditional log-likelihood by variational methods. In fact, many studies are based on efforts with CVAEs to train various multiple modalities such as handwriting digits and labels <cit.>, object images and degrees of rotation <cit.>, facial images and attributes <cit.>, and natural images and captions <cit.>. The main features of CVAEs are that the relation between modalities is one-way and a latent variable does not include information of a conditioned modality[According to <cit.>, this independence might not be satisfied strictly because the encoder in CVAEs still has dependence.], which is an unsuitable characteristic for our objective. <cit.> proposes the use of a conditional multimodal autoencoder (CMMA), which also maximizes the conditional log-likelihood but which makes this latent variable connected directly from a conditional variable, i.e., these variables are not independent. However, CMMA still considers that these modalities are generated in a single fixed direction. 

Another well-known approach of deep generation models is generative adversarial nets (GANs) <cit.>. Even in the case of GANs, when handling multiple modalities, it is often modeled by generation in one direction such as conditional GANs <cit.>.

Recently, GANs that can generate images bi-directionally from images have been proposed <cit.>. These models train perfect pixel-by-pixel correspondence between modalities of the same dimension and intentionally ignore probabilistic factors. However, no complete correspondence exists between modalities of different kinds that we examine specifically in this study. Our methods can train the probabilistic joint representation integrating all modality information, so they can obtain a probabilistic relation between modalities.





§ VAES WITH MULTIPLE MODALITIES

This section first briefly presents the formulation of VAEs. Subsequently, a simple extension of VAEs to multiple modalities is introduced.



 §.§ Variational autoencoders

Given observation variables 𝐱 and corresponding latent variables 𝐳, their generating processes are definable as 𝐳∼ p(𝐳)=𝒩(0,𝐈) and 𝐱∼ p_θ(𝐱|𝐳), where θ is a model parameter of p. The objective of VAEs is maximization of the marginal distribution p(𝐱)=∫ p_θ(𝐱|𝐳)p(𝐳)d𝐱. Because this distribution is intractable, we instead train the model to maximize the following lower bound.

    log p(𝐱)    ≥    -D_KL(q_ϕ(𝐳|𝐱)||p(𝐳)) + E_q_ϕ(𝐳|𝐱)[log p_θ(𝐱|𝐳)] 
       ≡   ℒ_VAE(𝐱),

where q_ϕ(𝐳|𝐱) is an approximate distribution of posterior and ϕ is a model parameter of q. We designate q_ϕ(𝐳|𝐱) as encoder and p_θ(𝐱|𝐳) as decoder. Moreover, in Eq. <ref>, the first term represents a regularization. The second one represents a negative reconstruction error.

To optimize the lower bound ℒ(𝐱) with respect to parameters θ,ϕ, we estimate the gradients of Eq. <ref> using stochastic gradient variational Bayes (SGVB). If we consider q_ϕ(𝐳|𝐱) as a Gaussian distribution 𝒩(𝐳;μ, diag (σ^2)), where ϕ={μ,σ^2}, then we can reparameterize 𝐳∼ q_ϕ(𝐳|𝐱) to 𝐳=μ+σ⊙ϵ, where ϵ∼𝒩(0,𝐈). Therefore, we can estimate the gradients of the negative reconstruction term in Eq. <ref> with respect to θ and ϕ as ∇_θ, ϕ E_q_ϕ(𝐳|𝐱)[log p_θ(𝐱|𝐳)] = E_𝒩(ϵ;0,𝐈)[∇_θ, ϕlog p_θ(𝐳|μ+σ⊙ϵ)]. The gradients of the regularization term are solvable analytically. Therefore, we can optimize Eq. <ref> using standard stochastic optimization methods.



 §.§ Joint Multimodal Variational Autoencoders


Next, we examine the i.i.d. dataset (𝐗, 𝐖)={(𝐱_1,𝐰_1)...,(𝐱_N,𝐰_N)}, where two modalities 𝐱 and 𝐰 have dimensions and structures of different kinds[In our experiment, they depend on the dataset, see the section of setting.]. We assume that these generative models are conditionally independent of the same latent variable 𝐳, i.e. joint representation. It means that the latent variables of generative models corresponding to each modality are shared. Therefore, their generative process becomes 𝐳∼ p(𝐳) and 𝐱,𝐰∼ p(𝐱,𝐰|𝐳)=p_θ_𝐱(𝐱|𝐳)p_θ_𝐰(𝐰|𝐳), where θ_𝐱 and θ_𝐰 respectively represent the model parameters of each independent p. 



Considering an approximate posterior distribution as q_ϕ(𝐳|𝐱,𝐰), we can estimate a lower bound of the log-likelihood log p(𝐱,𝐰), as shown below.


    ℒ_JM(𝐱,𝐰) 
    
        = -D_KL(q_ϕ(𝐳|𝐱,𝐰)||p(𝐳)) 
            + E_q_ϕ(𝐳|𝐱,𝐰)[log p_θ_𝐱(𝐱|𝐳)] + E_q_ϕ(𝐳|𝐱,𝐰)[log p_θ_𝐰(𝐰|𝐳)].



Eq. <ref> has two negative reconstruction terms that are correspondent to each modality. As with VAEs, we designate q_ϕ(𝐳|𝐱,𝐰) as the encoder and both p_θ_𝐱(𝐱|𝐳) and p_θ_𝐰(𝐰|𝐳) as decoders. This model is a simple extension of VAEs to multiple modalities, and we call it a joint multimodal variational autoencoder (JMVAE). Because each modality has different feature representation, we set different networks for each decoder. 



§ COMPLEMENT OF A MISSING MODALITY



After training JMVAE, we can extract a joint latent representation by sampling from the encoder q_ϕ(𝐳|𝐱,𝐰) at testing time. Our objective is to exchange modalities bi-directionally, e.g., images to text and vice versa. In this setting, modalities that we want to sample are expected to be missing, so that inputs of such modalities are set to zero or random noise (Figure <ref>(a)). In discriminative multimodal settings, this is a common means of estimating modalities from other modalities <cit.>, but it is difficult to handle this missing modality properly.

In VAE, <cit.> proposes a sophisticated approach of iterative sampling by Markov chain with a transition kernel to complement the missing value of input. In the case of JMVAE, the transition kernel when 𝐱 is missing is T(𝐱̃|𝐱,𝐰) = ∫ p(𝐱̃|𝐳)q(𝐳|𝐱,𝐰) d𝐳.

We can now estimate a missing modality 𝐱 by first setting the initial value of 𝐱 to random noise such as 𝐱∼ p(𝐱). Then we conduct iterative sampling following to the above kernel. As the number of iterations increases, we can estimate complemented values better. As described in this paper, we call it the iterative sampling method.

However, if missing modalities are high-dimensional and complicated such as natural images compared to other modalities, then the inferred latent variable might collapse and generated samples might become incomplete. In the experiment, we will show that this difficulty cannot be prevented even using the iterative sampling method.

To resolve this issue, we propose two new models: JMVAE-h and JMVAE-kl.



 §.§ JMVAE-h

Recently, some studies have extended the latent variables of VAEs to a stochastic hierarchical structure to improve the expressiveness and likelihood of models <cit.>. Such a hierarchical structure of latent variables becomes robust against a missing input, so it might contribute to preventing them from collapse. 

We can extend JMVAE to a hierarchical structure of latent variables. Let the latent variable be the stochastic hierarchy of the L layers 𝐳_1,...,𝐳_L[Note that the stochastic layers differ from the deterministic layers of neural networks.], the joint distribution of JMVAE becomes p(𝐱,𝐰) = ∫ ... ∫ p_θ(𝐳_L)p_θ(𝐳_L-1|𝐳_L) ... p_θ(𝐱|𝐳_1)p_θ(𝐰|𝐳_1) d𝐳_1 ... d𝐳_L, where all conditional distributions p_θ(𝐳_l-1|𝐳_l) are Gaussian and are parameterized by deep neural networks.

Various means are available to decompose the approximate distribution q(𝐳|𝐱,𝐰). We follow <cit.>, which is q(𝐳_1,...,𝐳_L|𝐱,𝐰) = q_ϕ(𝐳_1|𝐱,𝐰)… q_ϕ(𝐳_L|𝐱,𝐰), where all conditional distributions q_ϕ(𝐳_l|𝐱,𝐰) are Gaussian. In this study, as with <cit.>, the structure from the input to the final stochastic layer consists of deterministic mappings  (each mapping is parameterized in a deep neural network), and the probabilistic output 𝐳_l of each stochastic layer is obtained from a deterministic output 𝐡_l (see Figure <ref>(b)).

Therefore, the lower bound of JMVAE with a stochastic hierarchical structure becomes

    ℒ_JM_h(𝐱,𝐰) 
       = -∑_l=1^L E_q_ϕ(𝐳_l+1|𝐱,𝐰)[D_KL(q_ϕ(𝐳_l|𝐱,𝐰)||p_θ(𝐳_l|𝐳_l+1))] 
            + E_q_ϕ(𝐳_1|𝐱,𝐰)[log p_θ(𝐱|𝐳_1)] + E_q_ϕ(𝐳_1|𝐱,𝐰)[log p_θ(𝐰|𝐳_1)].


We call this model JMVAE-h, and demonstrate that it can prevent the issue of missing modality through experiments. In our experiments, we set L=2.
Figure <ref>(b) shows the flow of generating 𝐰 from 𝐱 with JMVAE-h.



 §.§ JMVAE-kl


In the stochastic hierarchical approach described above, the iterative sampling method is still important for generating missing modalities. However, it takes time to generate high-dimensional samples. Therefore, we propose JMVAE-kl as a model to generate appropriate missing samples without using the iterative sampling method.

Assume that we have encoders with a single input, q_λ(𝐳|𝐱) and q_λ(𝐳|𝐰), where λ is a parameter, then we would like to train them by reducing the divergence between their encoders and an encoder q_ϕ(𝐳|𝐱,𝐰) (see Figure <ref>(c)). Therefore, the object function of JMVAE-kl becomes

    ℒ_JM_kl(𝐱,𝐰) 
        = ℒ_JM(𝐱,𝐰) 
            - [D_KL(q_ϕ(𝐳|𝐱,𝐰)||q_λ(𝐳|𝐱)) + D_KL(q_ϕ(𝐳|𝐱,𝐰)||q_λ(𝐳|𝐰))].



Actually, JMVAE-h and JMVAE-kl have both benefits and shortcomings. JMVAE-kl must prepare encoders for each modality apart from the original encoder, but JMVAE-h requires no preparation of any additional encoder. Conversely, although JMVAE-h must apply the iterative sampling method to generate a missing modality, JMVAE-kl does not need to do so. Therefore, JMVAE-h is effective when there are three or more modalities and the dimension of the missing modality is not so large; JMVAE-kl is effective when there are modalities of only two kinds and the missing modality is high-dimensional.

 
 
 


 
 



§ EXPERIMENTS

In this section, we confirm the following three points through experimentation: (1) The missing modality difficulty certainly occurs on JMVAE and our proposed methods can prevent this issue. (2) Our proposed models can generate modalities bi-directionally with the likelihood equivalent to (or higher than) models that generate modality in only one direction. (3) Our models can appropriately obtain the joint representation, which integrates information of different modalities.



 §.§ Datasets

As described herein, we used two datasets: MNIST and CelebA <cit.>.

Originally, MNIST was not a dataset for multimodal setting. For this work, we used this dataset as a toy problem for various tests to verify of our model. We regard 784-dimensional handwriting images and corresponding 10 digit labels as two modalities. We used 50,000 as training set and the remaining 10,000 as a test set.

CelebA consists of 202,599 color facial images and 40 corresponding binary attributes such as male, eyeglasses, and mustache. For this work, we regard them as two modalities. This dataset is challenging because they have dimensions and structures of completely different kinds. Beforehand, we cropped the images to squares and resized them to 64 × 64 and normalized them. From the dataset, we chose 191,899 images that are identifiable facial images using OpenCV and used them for our experiment. We used 90% out of all the dataset contents as a training set and the remaining 10% of them as a test set.


 §.§ Settings



For MNIST, we considered images as 𝐱∈ℝ^784 and corresponding labels as 𝐰∈{0,1}^10. We set p(𝐱|𝐳) as Bernoulli and p(𝐰|𝐳) as categorical distribution.

We used warm-up <cit.>, which first forces training only of the term of the negative reconstruction error and which then gradually increases the effect of the regularization term to prevent local minima during early training. We increased this term linearly during the first N_t epochs as with <cit.>. Then we set N_t=200 and trained for 2000 epochs on MNIST. Moreover, as described by <cit.>, we resampled the binarized training values randomly from MNIST images for each epoch to prevent over-fitting.

For CelebA, we considered facial images as 𝐱∈ℝ^32 × 32 × 3 and corresponding attributes as 𝐰∈{-1,1}^40.  

We set a Gaussian distribution for the decoder of both modalities, where the variance of the Gaussian was fixed to 1. We set N_t=20 and trained for 50 epochs.

We used the Adam optimization algorithm <cit.> with a learning rate of 10^-3 on MNIST and 10^-4 on CelebA. The models were implemented using Theano <cit.>, Lasagne <cit.>, and Tars[https://github.com/masa-su/Tars]. See appendix for details of network architectures.



 §.§ Evaluation method

For this experiment, we estimated the test conditional log-likelihood log p(𝐱̃|𝐰) (or log p(𝐰̃|𝐱)) to evaluate the model performance. This estimate indicates how well a model can generate a modality from the corresponding another modality. Therefore, higher is better. We can estimate log p(𝐱̃|𝐰) as

    log p(𝐱̃|𝐰) ≃log∫ q(𝐳|𝐰)p(𝐱̃|𝐳) d𝐳
                        ≥∫ q(𝐳|𝐰)log p(𝐱̃|𝐳) d𝐳≃1/N∑_i=1^N log p(𝐱̃|𝐳^(l)),

where 𝐳^(l)∼ q(𝐳|𝐰). In Eq. <ref>, we apply the sample approximation to the lower bound of the log-likelihood rather than to the log-likelihood directly. This is because the log-likelihood is biased[This lower bound is unbiased estimator. Moreover, <cit.> shows that sample approximation of log-likelihood approaches the true log-likelihood as the number of samples increases.]. We set N=10 for all experiments.

To estimate the above, we should approximate q(𝐳|𝐰) (or q(𝐳|𝐱)) and draw samples from them. The way of it depends on how to complement missing modalities. In the cases of JMVAE and JMVAE-h, we first set the input of modality which we want to generate as missing, and then apply iterate sampling in multiple times to complement the missing modality. This means that the estimation of log-likelihood on these models depends on the number of iterate sampling. Conversely, JMVAE-kl can estimate the approximate distribution q(𝐳|𝐰) (or q(𝐳|𝐱)) directly at the training stage. For the approximation of conditional log-likelihood in JMVAE-h, see appendix.



 §.§ MNIST results


 


  §.§.§ Confirmation of the missing modality difficulty


Figure <ref>(a) presents results of generating 𝐱 from 𝐰. The top row, i.e. the image generated by application of iterative sampling only once, is blurred and is not properly generated. As the number of iterative sampling increases, the generated images become somewhat clearer,  but it is readily apparent that these images do not correspond to their labels. This result demonstrates that a missing modality cannot be generated even using the iterative sampling method. Figure <ref>(b) presents the result obtained in the case of JMVAE-h. Unlike the results presented in Figure <ref>(a), it is apparent that images corresponding to their labels are generated appropriately as the sampling number increases. Figure <ref>(c) is the result for JMVAE-kl. JMVAE-kl can generate digit images conditioned on numbers appropriately without iterating sampling.

Figure <ref> presents the conditional likelihood of each model under the number of iterative sampling changes. As the number of sampling increases, the log-likelihood increases for both JMVAE and JMVAE-h. However, in the case of normal JMVAE, its likelihood is much lower than that of JMVAE-kl no matter how much the number of sampling is increased. For JMVAE-h, the likelihood increases with a fewer sampling times than JMVAE, and the final likelihood becomes higher than normal JMVAE. On the other hand, in the case of JMVAE-kl, it can obtain high likelihood without iterative sampling.



  §.§.§ Evaluation of conditional log-likelihood

This section describes evaluation of our models with conditional log-likelihood to evaluate the bi-directional generation of modalities quantitatively. Additionally, we compare the log-likelihood evaluation with conventional conditional VAEs, which are CVAE <cit.> and CMMA <cit.>. Note that these models cannot generate modalities bi-directionally, so it is necessary to train them separately in each direction.

The upper of Table <ref> presents the test conditional log-likelihoods of JMVAE, our two improved models, and conventional conditional VAEs. The number of iterative sampling of JMVAE and JMVAE-h was set to 10. First, when generating an image from labels, JMVAE-kl and JMVAE-h improve their likelihood compared to JMVAE as expected. In addition, it turns out that these models have higher likelihood than existing VAEs modeling generation in only one direction. Next, in the case of generating labels from images, we find that there is not much difference in the likelihood for each model compared with generation in the opposite direction. This means that the missing modality difficulty does not occur when we generate a small dimensional modality from another large one. Note that the evaluation value of the conditional likelihood of JMVAE-h might be underestimated compared to the evaluation value of other models. This is because the approximation of conditional likelihood holds a lower bound than other models (see appendix for details). In light of this fact, we can see that both JMVAE-kl and JMVAE-h can be generated in both directions with the same or higher likelihood as that of conventional conditional VAEs.



  §.§.§ Visualization of the joint representation


Figure <ref> presents a visualization of the joint representation in each model. For the case of JMVAE-h, we sampled at the top stochastic layer. Here, we sampled from the trained encoders with the dimensions of the latent variable set to 2.

Specifically examining the left of each model results in Figure <ref>, samples from all models in the latent space are distributed and labeled for each label, which indicates that the joint representation including two modalities is obtained. 

Next, we specifically examine the right in Figure <ref>. From JMVAE results, we can see that all the samples are distributed in a considerably small area irrespective of their labels. This result demonstrates that, if the image information is missing, the latent representation actually collapses. By contrast, the result of JMVAE-kl shows that it can obtain the latent representation that is almost unchanged from sampling from all modalities. Because samples are not gathered in a small area as JMVAE, JMVAE-kl prevents the latent representation from collapsing. Finally, regarding the result of JMVAE-h, they are distributed considerably separately for each label just like the result on the left figure. From this, it can be said that the collapse of latent representation is also prevented in JMVAE-h too.
 
 
 

 
  
https://en.wikipedia.org/wiki/Mona_Lisa
https://en.wikipedia.org/wiki/Wolfgang_Amadeus_Mozart
 


 §.§ CelebA results




  §.§.§ Confirmation of the missing modality difficulty

 
Figure <ref> presents the results obtained from generating facial images from attributes. From Figure <ref>(b), we can see that an ordinary JMVAE cannot generate an appropriate facial images at all from its attributes. In addition, unlike MNIST, it can be seen that the facial image largely collapses if the number of iterative sampling is increased. This suggests that the greater the difference in the amount of information between modalities becomes, the more severely the missing modality difficulty may become. On the other hand, it is confirmed that JMVAE-h improves the facial image quality, and that it can generate a more beautiful images when the number of iterative sampling is increased (Figure <ref>(c)). However, it can also be confirmed that these generated images do not correspond closely to the attributes. Specifically, no generated image shows a subject wearing eyeglasses, probably because the stochastic hierarchy deepened, the gap separating the modes corresponding to the attributes becomes smaller, which makes mixing in the latent space easier. Finally, for the case of JMVAE-kl, we can observe that facial images corresponding to attributes are generated by one sampling (Figure <ref>(d)).



  §.§.§ Evaluation of the conditional log-likelihood

The lower of the Table <ref> shows the test conditional log-likelihoods of all models. The number of iterative sampling of JMVAE and JMVAE-h was set to 40. First, in the case of generating facial images from attributes, we can see that the likelihoods of JMVAE-kl and JMVAE-h are improved considerably compared to that of JMVAE. From this, we can see that even if the dimensions between the modalities are differ greatly, our improved models contribute to preventing the missing modality difficulty. Furthermore, as with the result of MNIST, we can see that our improved models can be generated in both directions with the same or higher likelihood as that of conventional VAEs.
 







 



  §.§.§ Generating faces from attributes and the joint representation on CelebA


Next, we confirm that our model can obtain the joint representation appropriately.
Based on the results presented above, we used JMVAE-kl in the remaining experiments. Moreover, we combined JMVAE-kl with GANs to generate clearer images. We considered the network of p(𝐱|𝐳) as generator in GANs. Then we optimized the GAN's loss with the lower bound of JMVAE-kl, which is the same approach to a VAE-GAN model <cit.>. See appendix for the network structure of the discriminator of this GAN.

Figure <ref>(a) portrays generated faces conditioned on various attributes. Results show that we can generate an average face for each attribute and various random faces conditioned on a certain attributes. Figure <ref>(b) shows that these samples are gathered for each attribute. In addition, in the group for each attribute, the average face is positioned almost at the center of each group, and random facial images are arranged around the average face. Furthermore, it can be confirmed that these arrangements are almost the same in both Base and Not Male. These results indicate that manifold learning of the joint representation works well.



  §.§.§ Bi-directional generation between faces and attributes on CelebA

Finally, we demonstrate that our model can do bi-directionally generation between faces and attributes. Figure <ref> shows that JMVAE-kl can generate both attributes and changed images conditioned on various attributes from images that had no attribute information. These are possible because, as the result above shows, JMVAE-kl can properly obtain the joint representation integrating different modalities.



§ CONCLUSION

As described herein, we introduced the extension of VAEs to generate modalities bi-directionally, which we call JMVAE. However, results show that we cannot generate a high-dimensional modality properly if the input of this modality is missing, and that the known method, the iterative sampling method, cannot resolve this issue. We proposed two models, JMVAE-kl and JMVAE-h, to prevent the above missing modality difficulty. Results demonstrated that these proposed models prevent a latent variable from collapse and that they can generate modalities bi-directionally with equal or higher quality compared to models which can only generate them in one direction. Furthermore, because these methods appropriately obtain the joint representation, we found that samples of various variations can be generated as corresponding to a certain modality or a changed modality.









§ THE NETWORK ARCHITECTURES




 §.§ Parameterization of distributions with deep neural networks

The Gaussian distribution can be parameterized with deep neural networks as follows.

    𝒩(𝐳;μ, diag (σ^2)),
       μ = f_μ(f_ MLP(𝐱)), 
       σ^2 =  Softplus(f_σ^2(f_ MLP(𝐱))),
 
where f_μ and f_σ^2 are linear single layer neural networks and f_ MLP means a deep neural network with arbitrary number of layers. Moreover, applying softplus function for each element of a vector is denoted as Softplus.

The Bernoulli distribution is parameterized as

    p_θ(𝐱|𝐳)=ℬ(𝐱;μ), μ =  Sigmoid(f_μ(f_ MLP(𝐳))),

where Sigmoid is sigmoid function.

In the case of categorical distribution, we can parameterize it as

    p_θ(𝐱|𝐳)=𝒞(𝐱;μ), μ =  Softmax(f_μ(f_ MLP(𝐳))),

where Softmax means softmax function.



 §.§ MNIST


For the notation of model structures, we denote a linear fully-connected layer with k units and ReLU as DkR, and DkR without ReLU as Dk. In addition, the process of applying J after I is denoted as I-J, and the process of concatenating the last layers of the two networks I, J into one layer is denoted as (I,J).

Therefore, the network structures of encoders and decoders on MNIST are as follows.


 
  * q(𝐳|𝐱, 𝐰) (Gaussian)
    
      
  * f_μ and f_σ^2: D64
      
  * f_ MLP: (D512R-D512R, D512R-D512R)
     
 
  * q(𝐳|𝐱), q(𝐳|𝐰), and p(𝐳_1|𝐳_2) (Gaussian)
    
      
  * f_μ and f_σ^2: D64
      
  * f_ MLP: D512R-D512R
    
 
  * p(𝐱|𝐳) (Bernoulli)
    
      
  * f_μ: D784
      
  * f_ MLP: D512R-D512R
        
 
  * p(𝐰|𝐳) (Categorical)
    
      
  * f_μ: D10
      
  * f_ MLP: D512R-D512R
                  
 
  * q(𝐳_2|𝐳_1) (Gaussian)
    
      
  * f_μ and f_σ^2: D64
      
  * f_ MLP: D512R-D512R-D64-D512R-D512R
       




 §.§ CelebA

We denote a convolutional layer (filter size : 4×4, number of channels : k, and stride : 2) with batch normalization and ReLU as CkBR, and CkBR without ReLU as CkB. In addition, we denote a deconvolutional layer (filter size : 4×4, number of channels : k, and crop : 2) with batch normalization and ReLU as DCkBR, DCkBR without ReLU as DCkB, and DCkB without batch normalization as DCk. Further, a linear fully-connected layer with k units, ReLU, and batch normalization is denoted as DkBR and a flatten layer is denoted as F.

Using the above notations, the model structures on CelebA are as follows.


 
  * q(𝐳|𝐱, 𝐰) (Gaussian)
    
      
  * f_μ and f_σ^2: D128
      
  * f_ MLP: (C64R-C128BR-C256BR-C256BR-F, D512R-D512BR)-D1024R
     
 
  * q(𝐳|𝐱) (Gaussian)
    
      
  * f_μ and f_σ^2: D128
      
  * f_ MLP: C64R-C128BR-C256BR-C256BR-F-D1024R
    
 
  * q(𝐳|𝐰) (Gaussian)
    
      
  * f_μ and f_σ^2: D128
      
  * f_ MLP: D512R-D512BR-D1024R
     
 
  * p(𝐳_1|𝐳_2) (Gaussian)
    
      
  * f_μ and f_σ^2: D128
      
  * f_ MLP: D512R-D512R
    
 
  * p(𝐱|𝐳) (Gaussian with fixed variance)
    
      
  * f_μ: DC3
      
  * f_ MLP: D4096R-DC256BR-DC128BR-DC64BR
        
 
  * p(𝐰|𝐳) (Gaussian with fixed variance)
    
      
  * f_μ: D10
      
  * f_ MLP: D512R-D4096R
                  
 
  * q(𝐳_2|𝐳_1) (Gaussian)
    
      
  * f_μ and f_σ^2: D128
      
  * f_ MLP: (C64R-C128BR-C256BR-C256BR-F, D512R-D512BR)-D1024R-D64-D512R-D512R
     


Moreover, we set C64R-C128BR-C256BR-C256BR
-F-D1024R-D1S (where DkS means Dk with sigmoid function) as a network structure of the discriminator used in CelebA experiment.



§ CONDITIONAL LOG-LIKELIHOOD OF JMVAE-H

We can estimate conditional log-likelihood of JMVAE-h as follows.

    log p(𝐱̃|𝐰)    ≃   log∫ q(𝐳_2|𝐰) ∫ p(𝐳_1|𝐳_2)p(𝐱̃|𝐳_1) d𝐳_1 d𝐳_2 
       ≥   ∫ q(𝐳_2|𝐰) log∫ p(𝐳_1|𝐳_2) p(𝐱̃|𝐳_1) d𝐳_1 d𝐳_2 
       ≥   ∫ q(𝐳_2|𝐰) ∫ p(𝐳_1|𝐳_2) log p(𝐱̃|𝐳_1) d𝐳_1 d𝐳_2 
       ≃   1/N∑_i=1^N ∫ p(𝐳_1^(l)|𝐳_2) log p(𝐱̃|𝐳_1) d𝐳_1,

where 𝐳_2^(l)∼ q(𝐳_2|𝐰) and 

    ∫ p(𝐳_1^(l)|𝐳_2) log p(𝐱̃|𝐳_1) d𝐳_1 ≃1/N∑_j=1^N log p(𝐱̃|𝐳_1^(k)),

where 𝐳_1^(k)∼ p(𝐳_1|𝐳_2^(l))．We set N=10 for all experiments.

We find that the approximation of Eq. <ref> holds a lower bound than that of JMVAE. Therefore, the evaluation value of the conditional likelihood of JMVAE-h might be underestimated compared to the evaluation value of JMVAE.

