Reconstructing Perceived Images from Brain Activity by Visually-guided Cognitive Representation and Adversarial Learning
Ziqi Ren Jie Li Xuetong Xue Xin Li Fan Yang Zhicheng Jiao Xinbo Gao















Reconstructing Seen Image from Brain Activity by Visually-guided Cognitive Representation and Adversarial Learning 
    Ziqi Ren

Xidian University







Jie Li

 Xidian University




Xuetong Xue

Xidian University

Xin Li

IIAI

Fan Yang

IIAI

Zhicheng Jiao

UPenn

Xinbo Gao

Xidian University


    Month Day, Year
========================================================================================================================================================================================






Reconstructing visual stimulus (image) only from human brain activity measured with functional Magnetic Resonance Imaging (fMRI) is a significant and meaningful task in Human-AI collaboration. However, the inconsistent distribution and representation between fMRI signals and visual images cause the heterogeneity gap. Moreover, the fMRI data is often extremely high-dimensional and contains a lot of visually-irrelevant information. Existing methods generally suffer from these issues so that a satisfactory reconstruction is still challenging. In this paper, we show that it is possible to overcome these challenges by learning visually-guided cognitive latent representations from the fMRI signals, and inversely decoding them to the image stimuli. The resulting framework is called Dual-Variational Autoencoder/ Generative Adversarial Network (D-Vae/Gan), which combines the advantages of adversarial representation learning with knowledge distillation. In addition, we introduce a novel three-stage learning approach which enables the (cognitive) encoder to gradually distill useful knowledge from the paired (visual) encoder during the learning process. Extensive experimental results on both artificial and natural images have demonstrated that our method could achieve surprisingly good results and outperform all other alternatives. 














§ INTRODUCTION

Reading mind, the act of “seeing” someone’s thoughts and feelings, has long been an ambitious capability in works of fiction. In recent years, breakthroughs in neuroscience and AI have brought such fictional technologies into the realm of science, i.e., brain decoding. Neuroscience studies <cit.> have suggested that there exist a mapping from visual stimuli to brain activity patterns, which takes a visual stimulus as input and produces the corresponding brain activity pattern. Recent studies <cit.> find that such mapping is invertible, and the perception is feasible to be reconstructed from brain activity patterns if the inverse mapping is precisely estimated.

Brain decoding can be distinguished into three categories: identification, classification and reconstruction. The first two have been recorded promising results <cit.> while the last one remains unsolved. This is because accurate reconstruction of perceived images requires both high-level semantic knowledge and low-level visual details. It is hard to extract all this information from noisy high-dimensional brain activity signals (e.g., the fMRI data). In addition, the modality gap commonly exists between cognitive signals and visual stimuli, making the mapping relationship even harder to estimate. Although there have been several attempts at reconstructing the visual stimuli from brain responses <cit.>, they generally yield blurry, cluttered, and low-quality results.



Driven by the success of latent models, adversarial learning and knowledge distillation, we design a novel framework, called Dual-Variational Autoencoder/ Generative Adversarial Network (D-Vae/Gan), to learn the mapping from fMRI signals to their corresponding visual stimuli (images). Our goal is to learn an encoder which can map the brain signal to a low-dimensional representation that preserves important visually-relevant information, as well as a powerful decoder that is able to recover this latent representation back to the corresponding visual stimulus. To achieve this goal, our D-Vae/Gan first generates low-dimensional latent features for both fMRI signals and perceived images (visual stimuli) with a Dual VAE-Based Encoding Network, then learns to capture visually important information and overcome modality gap between them in this space through a novel GAN-Based inter-modality knowledge distillation method, and finally decodes the learned cognitive latent features back to corresponding visual images using our Adversarial Decoding Network. Intuitively, our solution provides two major advantages. First, by learning lower-dimensional latent representation for the high-dimensional brain signal, the approach is able to filter out the majority of noise and clutters, and thus produces a more compressed representation. Second, by adapting cognitive features to visual features in the lower-dimensional latent space, the approach is able to learn visually important features and overcome the modality gap more effectively. 




To sum up, our contributions are as follows: (i) We introduce a new framework, namely D-VAE/GAN, established by combining our proposed Dual VAE-Based Encoding Network with the Adversarial Decoding Network, to learn a powerful encoder-decoder model for mapping fMRI signals to the corresponding visual stimuli. (ii) We propose to leverage free-of-cost guidance knowledge from the visual domain for guiding the latent representation learning, without additional annotations. (iii) We present a novel three-stage training approach, in which a GAN-Based inter-modality knowledge distillation method is introduced to make the learned cognitive latent features to capture more visually-relevant knowledge by forcing the cognitive encoder to mimic its paired visual encoder. (iv) Our method has achieved quite effective reconstruction results on four public fMRI datasets and made an encouraging breakthrough in visual stimuli reconstruction.














§ RELATED WORKS



Traditionally, machine learning methods play significant roles in the fMRI-Based brain decoding tasks. Miyawaki et al., for the first time, proposed the spares multinomial logistic regression (SMLR) by using multi-voxel patterns of fMRI signals and multi-scale visual representation to reconstruct the lower-order information <cit.>. Schoenmakers et al. reconstructed handwritten characters using a straightforward linear Gaussian approach <cit.>. Fujiwara et al. proposed to build a reconstruction model to automatically estimate image bases by Bayesian canonical correlation analysis (BCCA) <cit.>. However, because the linear hypothesis in the proposed models does not conform to the actual visual encoding-decoding process in human brain, their performance is far from satisfactory.

The recent integration of deep learning into neural decoding has been a very successful endeavor <cit.>. Gerven et al. reconstructed handwritten digits using deep belief networks <cit.>. Several proposed deep multi-view representation learning models, such as deep canonically correlated autoencoders (DCCAE) <cit.> and correlational neural networks (CorrNet) <cit.>, had the ability to learn deep correlational representations, and thus were able to reconstruct each view respectively. However, directly applying the nonlinear maps of DCCAE and CorrNet to limited noisy brain activities is prone to overfitting. A recent neural decoding method was based on multivariate linear regression and deconvolutional neural network (De-CNN) <cit.>. After that, Du et al. introduced Bayesian deep learning to study visual image reconstruction <cit.>, which can be viewed as a nonlinear extension of the linear method BCCA. In addition, Güçlütürk et al. reconstructed perceived faces with a deep adversarial neural decoding (DAND) model by combining probabilistic inference with deep learning <cit.>. Seeliger et al. trained a deep convolutional generative adversarial network to generate gray scale photos <cit.>. More recently, Shen et al. took advantage of deep neural networks (DNNs) to reconstruct color photos <cit.>. However, the generated visual stimuli by all existing deep learning-based methods are still rather far from the real ones. 






§ METHODOLOGY

In this section, we first formally define the brain-driven visual stimulus reconstruction problem, give an overview of our framework (D-Vae/Gan), and then introduce each component of our model in details. Finally, we show how to train our D-Vae/Gan with a novel three-stage training method. 









 §.§ Problem Definition and Notations

In this work, we study the visual stimulus (image) reconstruction problem. Assume 𝒮 = {(𝐱_i, 𝐲_i) | 𝐱_i ∈ℳ,  𝐲_i ∈ℐ_ℳ}_i=1^n are given as training data, where ℳ is the set of fMRI data and ℐ_ℳ is the corresponding visual stimulus images. The goal of this task is to learn a mapping function f: ℳ→ℐ_ℳ, so that for any novel input from test set 𝐱_t ∈𝒯, its corresponding visual stimulus can be generated through 𝐲_t = f(𝐱_t). However, directly learning the mapping function f is rather challenging due to the significant heterogeneity gap. To alleviate this problem, we formulate the mapping f as a two-stage process:  1) latent feature learning f_1: ℳ→ z_ℳ which transforms real brain signals to latent representations; and 2) visual image reconstruction f_2: z_ℳ→ℐ_ℳ that reconstructs stimulus images based on their latent features. Importantly, in the training phase, we also learn the latent representations for real visual stimulus, i.e., {𝐳^*_i|𝐳^*_i= f^*_1(𝐲_i)∈ z_ℳ_ℐ}_i=1^n where f^*_1: ℐ_ℳ→ z_ℐ_ℳ denotes the mapping function, and treat them as exemplars for guiding the learning of latent representations of brain signals (fMRI data) {𝐳_i|𝐳_i= f_1(𝐲_i)∈ z_ℳ}_i=1^n. The visual latent representations z_ℐ_ℳ and cognitive latent representations z_ℐ share the same reconstruction function so as to enforce them to be close in latent space , i.e., f_2: z_ℳ∪ z_ℐ_ℳ→ℐ_ℳ. 



 §.§ Overview of Framework

The overall framework of our D-Vae/Gan is illustrated in Figure <ref>. There are three major components in our framework: Cognitive Enconder  (E_Cog) , Visual Encoder (E_Vis), Generator (G) and Discriminator(D). The core component of D-Vae/Gan is the Cognitive Encoder network that can transform the high-dimensional and noisy brain signals (fMRI) to low-dimensional latent representations. Along with E_Cog, we have a Visual Encoder network that maps the visual stimuli into the latent representations. Intuitively, E_Vis is more capable of capturing all visually-related information during the encoding process, since it directly takes visual stimuli as inputs. Therefore the generated latent features by E_Vis can be considered as the free-of-cost guidance knowledge for E_Cog to learn the latent representations. From the standpoint of knowledge distillation, we can treat E_Vis as a teacher network, and train E_Cog (student network) by transferring knowledge from it. The Generator network in our framework is used to rebuild the visual stimuli based on the latent representations. Note that, during the training phase, the E_Cog and E_Vis share the same G in our framework, which forces E_Cog (student) to mimic E_Vis (teacher). Finally, we have a Discriminator network that acts as a classifier to distinguish real visual stimuli from fake stimuli produced by the generator. Note that the visual stimuli are used to guide cognitive latent feature learning and supervise the generation of reconstructed images only in the training stage. In the test stage, the only inputs are the cognitive signals (fMRI).






 §.§ Dual VAE-Based Encoding Network

In the training phase, we include two encoders in our framework, Cognitive Enconder  (E_Cog) and Visual Encoder (E_Vis), which form a dual encoding network architecture. Generally, both E_Cog and E_Vis are feed-forward neural networks and aim to encode a data sample 𝐱_i (or 𝐲_i) to the corresponding latent representation 𝐳∈ z_ℳ (or 𝐳^*∈ z_ℳ_ℐ). Ideally, the learned latent representation 𝐳 (or 𝐳^*) should capture important visually-related information (including the texture, color, location, stylistic properties, etc) such that the generator (G) is able to reconstruct the visual stimuli from these latent features.






  §.§.§ Cognitive Encoder (E_Cog).

To learn a robust cognitive latent representation 𝐳 for each input fMRI 𝐱, we resort to the Variational Autoencoder (VAE)-based encoder, i.e., probabilistic encoder, to model the mapping f_1: ℳ→ z_ℳ. Specifically, E_Cog is a lightweight neural network which is composed of a fully connected layer and an output layer. Given a cognitive pattern 𝐱, E_Cog produces a distribution over the possible values of the cognitive latent representation 𝐳 from which the corresponding visual stimulus 𝐲∈ℐ_ℳ could be generated. Following  <cit.>, the cognitive latent feature 𝐳 can be computed as follows:

    𝐳∼  E_Cog(𝐱; θ) = q_θ( 𝐳| 𝐱.),

where q_θ( 𝐳| 𝐱.) is a Gaussian distribution whose mean μ and diagonal covariance Σ are the output of E_Cog parameterized by θ. 

Here, we regularize the distribution q_θ( 𝐳| 𝐱.) to be a simple Gaussian distribution based on the Kullback-Leibler (KL) divergence between q_θ( 𝐳| 𝐱.) and the prior distribution p(𝐳) = 𝒩(0,𝐈),

    ℒ_prior-c (𝐱, 𝐳; θ)    =D_KL (q_θ( 𝐳| 𝐱.) || p(𝐳))
       = D_KL [𝒩(μ,Σ) || 𝒩(0,𝐈)],

where D_KL(·) means the KL divergence, and 𝐈 is the identity matrix.





  §.§.§ Visual Encoder (E_Vis).
 Our visual encoder is also a VAE-based encoder, which is used to model the mapping f^*_1: ℐ_ℳ→ z_ℐ_ℳ. But differently, E_Vis takes an image 𝐲∈ℐ_ℳ rather than an fMRI sequence as input, which makes it different from E_Cog in network architecture. Specifically, compared with E_Cog, E_Vis has three additional convolutional layers and one reshape layer on top of the fully connected layer so as to transform the imput image to the latent representation. Mathematically, the visual latent feature 𝐳^* is given as:

    𝐳^* ∼  E_Vis(𝐲; ϕ) = q_ϕ( 𝐳^*| 𝐲.),

where 𝐳^* is drawn from the posterior q_ϕ( 𝐳^*| 𝐱.) which is of Gaussian learned by E_Vis with learnable parameters ϕ. 

Similarly, the mean μ^* and diagonal covariance Σ^* of the distribution q_ϕ( 𝐳^*| 𝐲.) is regularized by

    ℒ_prior-v (𝐲, 𝐳^*; ϕ)    =D_KL (q_ϕ( 𝐳^*| 𝐲.) || p̂(𝐳^*))
       = D_KL [𝒩(μ^*,Σ^*) || 𝒩(0,𝐈)],

where p̂(𝐳^*) denotes the prior distribution, and p̂(𝐳^*) = 𝒩(0,𝐈).



 §.§ Adversarial Decoding Network

Given the learned latent representations, including cognitive latent feature 𝐳 and visual latent feature 𝐳^*, the generator tries to reconstruct the corresponding visual stimulus 𝐲. Here, the VAE decoder and the GAN generator are integrated by letting them share parameters. Through the above combination, we can take advantages of both VAE and GAN. On one hand, the adversarial loss computed from the discriminator makes generated visual stimulus more realistic. On the other hand, the optimization for VAE-Based decoder become quite stable, and thus avoids the “collapse” problem. We name such combined decoder network as Adversarial Decoding Network which includes a VAE-Based Decoder/ Generator (G) and a GAN discriminator (D), and use it to model the mapping f_2: z_ℳ∪ z_ℐ_ℳ→ℐ_ℳ. 



  §.§.§ VAE-Based Decoder/ Generator (G).
 
Our VAE-Based decoder G is symmetric to the visual encoder E_Vis, and it has a fully connected layer, a reshape layer, and three de-convolutional layers. For clarity, we denote the generator p_η(𝐲̃ | 𝐳') with learnable parameters η. For any latent feature 𝐳' ∈ z_ℳ∪ z_ℳ_ℐ, G estimates the  corresponding visual stimulus 𝐲̃, and thus,

    𝐲̃ =  G(𝐳'; η) =p_η(𝐲̃ | 𝐳'),


where η is the learnable network parameters of G. Our goal is to make 𝐲̃ similar to 𝐲 as much as possible. We update the network parameters η of the VAE-Based decoder G jointly with the given encoder E_Cog or E_Vis. Take the combination of E_Vis and G for example, the parameters η and ϕ are jointly updated by minimizing the loss:

    ℒ_𝒱    (𝐲, 𝐳^*, 𝐲̃; η, ϕ)= - 𝔼_q_ϕ(𝐳^*|𝐲) [logp_η(𝐲̃ | 𝐳^*) p(𝐳^*)q_ϕ(𝐳^* | 𝐲)]
       =- 𝔼_q_ϕ(𝐳^*|𝐲)[log p_η(𝐲̃ | 𝐳^*)] + D_KL (q_ϕ( 𝐳^*| 𝐲.) || p̂(𝐳^*))
       = ℒ_rec + ℒ_prior-v,

where ℒ_rec = - 𝔼_q_ϕ(𝐳^*|𝐲)[log p_η(ỹ | 𝐳^*)] denotes the reconstruction loss, and ℒ_prior-v  means the KL divergence <cit.> between the encoder distribution and a known prior. 






  §.§.§ GAN Discriminator (D).

Another important component in our adversarial decoding network is the GAN discriminator (D), which assigns 𝐜∈ [0, 1] for the input 𝐲 to measure the probability 𝐪 that 𝐲 is a real visual stimulus. Specifically, a standard discriminator D with adversarial training on its parameter τ is used to further improve the quality of 𝐲̃. The adversarial training loss for discriminator is formulated as,

    ℒ_gan (𝐲, 𝐲̃, 𝐜; τ) = log(D(𝐲, τ)) + log((1-𝐲̃), τ)),

where 𝐲 means the real visual stimulus. 

In addition, we can further replace ℒ_rec with a reconstruction loss expressed by the GAN discriminator. Here, we still take the combination of E_Vis and G for example. Following <cit.>, we employ a Gaussian observation model for D_l(𝐲, τ) with D_l(𝐲̃, τ) and identity covariance:

    p_η(D_l(𝐲, τ) | 𝐳^*) = 𝒩(D_l(𝐲, τ) | D_l(𝐲̃, τ), 𝐈),

where D_l(·) denotes the hidden representation of l_th layer of the discriminator. Therefore, ℒ_rec in Eq. <ref> can be replaced by

    ℒ^gan_rec-v (𝐲, 𝐳^*, 𝐲̃; η, ϕ, τ) = - 𝔼_q_ϕ(𝐳^*|𝐲)[log p_η(D_l(𝐲, τ) | 𝐳^*)].




 §.§ Three-Stage Training Method

As illustrated in Figure <ref>, three-stage training method is used to train our model in a gradual fashion. 






  §.§.§ Stage-I: Visual Latent Fearure Learning.


In the first stage, the visual stimuli ℐ_ℳ are used as input to jointly train the visual encoder E_Vis, VAE-Based generator G and GAN discriminator D. In this stage, we perform an intra-modality reconstruction task (within the same modality) for learning the visual latent representations. Specifically, E_Vis is optimized to encode the visual stimulus 𝐲∈ℐ_ℳ to the visual latent representation 𝐳^*, and then G and D are trained to reconstruct 𝐲 given 𝐳^*. In this stage, we train network  parameters ϕ of E_Vis, η of G and τ of D with a triple criterion:

    ℒ_𝒱 (𝐲, 𝐳^*, 𝐲̃, 𝐜; ϕ, η, τ) =
    ℒ^gan_rec-v (𝐲, 𝐳^*, 𝐲̃; η, ϕ, τ)
       +ℒ_prior-v (𝐲, 𝐳^*; ϕ) + ℒ_gan (𝐲, 𝐲̃, 𝐜; τ).






  §.§.§ Stage-II: Cognitive Latent Fearure Learning.
 

In the Stage-II, we perform an inter-modality knowledge distillation task that forces the cognitive encoder to mimic the visual encoder, so as to capture more visually important features in the cognitive latent space. Intuitively, it makes sense that E_Vis is more capable of capturing visually important features for the visual stimuli reconstruction since it learns latent representation within the same modality. Therefore, we treat E_Vis as a teacher network and E_Cog as a student network, and train E_Cog to mimic E_Vis so as to better capture visually-related information from noise brain signals.

Aiming to achieve this goal, we fix the generator G and use the images/stimuli {𝐲̃_̃ĩ}_i=1^n produced by the teacher network E_Vis as the real to the discriminator D for guiding the cognitive latent feature learning of E_Cog. Given the fMRI data {𝐱_i}_i=1^n, we train the network parameters θ for the cognitive encoder and update τ of D to encourage the generated visual stimulus 𝐲' from the cognitive latent feature 𝐳 to be similar with 𝐲̃ which generated from the visual latent feature 𝐳^* through the following loss function:

    ℒ_𝒞 (𝐱, 𝐳, 𝐲̃, 𝐲', 𝐜; θ, τ) = ℒ^gan_rec-c(𝐱, 𝐳, 𝐲'; θ, τ)
       +ℒ_prior-c (𝐱, 𝐳; θ) + ℒ_gan (𝐲̃, 𝐲', 𝐜; τ).








As illustrated in Figure <ref>, the cognitive encoder E_Cog is forced to mimic the visual encoder E_Vis by minimizing Eq. <ref>, and generates the latent feature 𝐳 which is similar to z^*. Our GAN-Based inter-modality knowledge distillation method is a generalization of  the distillation idea from <cit.>, and enables distillation of knowledge from the other modality. 



  §.§.§ Stage-III: Visual Stimuli Recovering.
 

Finally, in the third stage, we perform a fine-tuning task to slightly update the network parameters η of the generator G such that it can become more capable of recovering the visual stimuli when given cognitive latent feature 𝐳. In this stage, we fix parameters of the cognitive encoder E_Cog and use the visual stimulus 𝐲∈ℐ_ℳ as the real to D for fine-tuning η. The loss function is given as:

    ℒ'_𝒞 (𝐳, 𝐲, 𝐜; η, τ) = ℒ_gan (𝐳, 𝐲, 𝐜; η, τ).




Our three-stage training method is carefully designed to mitigate the gap between source and target domains in the training process. Our experiments in the next section demonstrate that the proposed three-stage training method is effective at learning good cognitive latent features from fMRI signals, and these learned latent features can be successfully recovered back to visual stimuli. 






§ EXPERIMENTS AND RESULTS




In this section, we evaluate the proposed D-Vae/Gan on four benchmark datasets. Following the convention of existing work <cit.>, both objective and subjective assessment methods are used for evaluation. 



 §.§ Dataset.
 

 

We evaluated the reconstruction quality on four public datasets. The information about the four datasets is summarized in Table <ref>.









	
	
  * Dataset 1: Geometric Shapes and Alphabetical Letters. This dataset consists of the contrast-defined patches and contains two independent sessions <cit.>. One is the “random image session”, in which spatially random patterns are sequentially presented. The other one is the “figure image session”, where alphabetical letters and simple geometric shapes are sequentially presented. The images from the “random image session” are used for training and images from the “figure image session” are used to test the performance. The fMRI data are taken from primary visual area V1.

	
	
  * Dataset 2: Handwritten Digits. This dataset contains a hundred of handwritten gray-scale digits with the size of 28×28 <cit.>. The images are taken from the training set of the MNIST and the fMRI data are taken from V1–V3. 
	
	
  * Dataset 3: Handwritten Characters. This dataset contains 360 gray-scale handwritten characters <cit.> . The images have the size of 56×56 which are taken from <cit.> and the fMRI data of V1–V2 are taken from three subjects.
	
  
  * Dataset 4: Real Color Photos. This dataset <cit.> is referred to as  dataset. It includes images with high resolutions (500 × 500), and have 150 object categories from the ImageNet. Following <cit.>, 1,200 images from 150 object categories (8 images from each category) are used for training, and 50 images from 50 object categories (no overlap with the training set) are used for testing. The fMRI data are taken from V1–V4, LOC, FFA and PPA. 
  
    
  








 §.§ Implementation Details and Evaluation Protocol.

We implement our D-Vae/Gan on Tensorflow, with a 1080Ti GPU×11G RAM card on a single server. During the training process, we use Adam optimizer with β=0.9. The base learning rate is set to 3×10^-4, and the decay_rate is set to 0.98. During the test, our D-Vae/Gan takes an fMRI instance as the input and generates a visual image. Note that all images (visual stimuli) are resized to 100× 100 for training and evaluation. 



We use four widely-used metrics to evaluate the performance of our D-Vae/Gan. First, we use two standard image similarity metrics including Pearson's Correlation Coefficient (PCC) and Structural Similarity Index (SSIM) <cit.>. Then, following <cit.>, we further evaluate the quality of results using both objective and subjective assessment methods. For the objective assessment, we compare the pixel-wise correlation coefficients of the reconstructed image with two candidates, in which one is its corresponding original stimulus and the other is a stimulus image randomly selected from the rest test images of the same image type. For the subjective assessment, we conduct a behavioral experiment with a group of 13 raters (5 females and 8 males, aged between 19 and 41 years). The subjects are presented a reconstructed image by using our D-Vae/Gan with two candidate images (designed in the same way as the objective assessment), and then are asked to select the option which appears more similar to the reconstructed image. For clarity, we name the objective assessment as Pix-Com, and the subjective assessment as Hum-Com. 





















 §.§ Comparison with State-of-the-Arts.
 
In this section, we compare our method with other state-of-the-art algorithms. The experiments are conducted on four datasets: Dataset 1 – Dataset 4. First, we evaluate the performance of our model on some relatively simple visual stimuli (Dataset 1 – Dataset 3), and then we show results on complex stimuli including images of real scenes (Dataset 4). 

As shown in Table <ref>, our D-Vae/Gan consistently outperforms the state-of-the-art algorithms on all benchmarks according to PCC and SSIM. We observe that traditional brain decoding methods like SMLR and BCCA only achieve limited performance, which is caused by its linear architecture and spherical covariance assumption. Recent deep learning models achieve significant breakthroughs in the brain signal reconstruct. Compared with recent deep learning-based methods <cit.>, our D-Vae/Gan can generate visual stimuli which are more consistent with the perceived images (see in Figure 3, 4, and 5 of the supplemental material). This is because our framework and the novel three-stage training method largely bridge the gap between multi-modal signals, and thus the learned latent representation can capture important visually relevant knowledge which is significant in the recovering process. Some sample results of different methods on Dataset 1 – Dataset 3 are given in Figure  <ref>,  <ref>, and  <ref>. 


We also evaluate our method on complex stimuli (real color photos). Note that only very limited models can work on Dataset4. We compare our D-Vae/Gan with two recently proposed methods: <cit.> and  <cit.>. Since the evaluation is too complex for the natural images, following <cit.>, we conduct the comparisons by using Pix-Com and Hum-Com. As can be seen in Figure <ref>, our D-Vae/Gan achieves 87.8% according to Pix-Com and 97.3% according to Hum-Com, which significantly outperforms  <cit.> by 9.7% in Pix-Com and 1.6% in Hum-Com, and surpasses <cit.> by 11.7% in Pix-Com and 0.3% in Hum-Com. Some exemplars are visualized in Figure <ref>.










 §.§ Ablation Studies.


In order to validate the superiority of our D-Vae/Gan framework, we performed a series of ablation experiments in this section. 

Effectiveness of our D-Vae/Gan. 
To quantify the contribution of our D-Vae/Gan, we derive a baseline, i.e., a standard DCGAN. On Dataset 1 and Dataset 3, DCGAN is very unstable in convergence, and thus unable to generate meaningful results for comparisons. On Dataset 2, as can be seen in Table <ref>, the reconstruction accuracy is quite dissatisfactory. We find our D-Vae/Gan brings significant performance improvements. 



Dual Vae-Based Encoder. 
Dual VAE-Based encoder is one of the major designs in this work, which forces the cognitive encoder to mimic the visual encoder. To evaluate this design, we offer a standard Vae/Gan model w/o the paired visual encoder. It can be observed a great performance degradation after excluding dual Vae-Based encoding model (see Table <ref>).

Standard Encoder vs. Vae-Based Encoder.  
To further investigate the superiority of using Vae for feature learning in our framework, we replace Vae in our model with standard Convolutional Neural Networks (CNNs). From Table <ref>, we also observe a significant performance drop. 



Impact of Visual-Feature Guidance.
To investigate the effect of visual-feature guidance operated in this work, we also provide another solution that can also capture visual knowledge for comparisons. Specifically, we first trained a standard Gan based on a lot of natural images (no overlap with the test image set) such that the model learned visual information. Then with the generator fixed, this Gan model was combined with an encoder (same structure as our cognitive encoder) and trained to reconstruct visual stimulus from fMRI data w/o knowledge distillation. As shown in Figure <ref>, our model achieves better performance than the model w/o visual-feature guidance (i.e., 87.8% vs. 78.9% according to Pix-Com and 97.3% vs. 80.3% according to Hum-Com). 







 §.§ Reconstruction Performance in Different ROIs.


To assess how the structures and colors of the stimulus images are reconstructed, we investigate the reconstruction quality for structure and color of different visual areas on Dataset 4 including V1 – V4 and Higher Visual Cortex (HVC; covering regions around LOC, FFA, and PPA). 

The Pix-Com and Hum-Com evaluations both suggest that structures are reconstructed better from early visual areas (Figure <ref> (B)), whereas colors are reconstructed better from the midlevel visual area V4 (Figure <ref> (C)). It reveals that most structure-relevant information is preserved in the low-level visual cortices (V1 – V3), while the midlevel (V4) and high-level cortices (LOC, FFA, and PPA) are responsible for capturing more color-relevant information. Above observations are entirely consistent with the coupling hierarchical relationship between perceived information and human visual areas <cit.>. Moreover, by applying the fMRI signals recorded from the entire visual cortex (VC), the structural and color characteristics are both effectively recovered so that these contrasting patterns further support the success of structure and color reconstructions. Besides, it indicates that our method can be used as a tool to characterize the information content encoded in the activity patterns of individual brain areas for visualization.





§ CONCLUSIONS AND FUTURE WORK


In this paper, we achieve an encouraging improvement for the brain-driven visual stimulus reconstruction. We propose a novel framework D-Vae/Gan and a new three-stage training method to better target visual stimulus reconstruction task. Extensive experiments confirm the effectiveness of the proposed method. We also conduct experiments to analyse how visual information is captured by the ROIs in brain, which is meaningful for the high-level AI research including neural architecture design and brain emulation. 



ieee



