Expert2Coder: Capturing Divergent Brain Regions Using Mixture of Regression Experts
Subba Reddy Oota Naresh Manwani Raju S. Bapi


𝐱
𝐰
𝐲
𝐛
ℝ








































































Expert2Coder: Capturing Divergent Brain Regions Using Mixture of Regression Experts
    Subba Reddy Oota, Member, IEEE,
        Naresh Manwani, Member, IEEE,
        Raju S. Bapi, Senior Member, IEEE
Subba Reddy Oota is with the Cognitive Science Lab (CSL) and Machine Learning Lab (MLL),
Kohli Center on Intelligent Systems (KCIS), International Institute of Information Technology (IIIT), Hyderabad,
India, 500032 e-mail: oota.subba@students.iiit.ac.in.
Naresh Manwani and Raju S. Bapi are, respectively, with the Machine Learning Lab (MLL) and Cognitive Science Lab (CSL),
Kohli Center on Intelligent Systems (KCIS), International Institut  of Information Technology (IIIT), Hyderabad, India, 500032 e-mail: naresh.manwani@iiit.ac.in, raju.bapi@iiit.ac.in.


    December 30, 2023
=====================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================






fMRI semantic category understanding using linguistic encoding models attempts to learn a forward mapping that relates stimuli to the corresponding brain activation. 
State-of-the-art encoding models use a single global model (linear or non-linear) to predict brain activation (all the voxels) given the stimulus. 
However, the critical assumption in these methods is that a priori different brain regions respond the same way to all the stimuli, that is, there is no modularity or specialization assumed for any region. This goes against the modularity theory, supported by many cognitive neuroscience investigations suggesting that there are functionally specialized regions in the brain.
In this paper we achieve this by clustering similar regions together and for every cluster we learn a different linear regression model using a mixture of linear experts model. The key idea here is that each linear expert captures the behaviour of similar brain regions.

Given a new stimulus, the utility of the proposed model is twofold (i) predicts the brain activation as a weighted linear combination of the activations of multiple linear experts and (ii) to learn multiple experts corresponding to different brain regions. We argue that each expert captures activity patterns related to a particular region of interest (ROI) in the human brain. 




This study helps in understanding the brain regions that are activated together given different kinds of stimuli. Importantly, we suggest that the mixture of regression experts (MoRE) framework successfully combines the two principles of organization of function in the brain, namely that of specialization and integration.
Experiments on fMRI data from paradigm 1 <cit.> where participants view linguistic stimuli show that the proposed MoRE model has better prediction accuracy  compared to that of conventional models. 
Our model achieves a mean absolute error (MAE) of 3.94, with an R^2-score of 0.45  on this data set. This is an improvement over performance of traditional methods including,
ridge regression (5.58 MAE, 0.15 R^2-score), MLP (4.63 MAE, 0.35 R^2-score). We also elaborate on the specializations captured by various experts in our mixture model and their implications.




fMRI, brain encoding, mixture of experts, cognitive neuroscience.

















§ INTRODUCTION

Functional magnetic resonance imaging (fMRI) measures brain activity by identifying the changes in the blood-oxygen-level-dependent (BOLD) imaging signals in different functional areas in response to particular stimuli.
In recent years, the use of both linear and non-linear multivariate encoding or decoding approaches for analyzing fMRI brain activity has become increasingly popular <cit.>.
An encoding model that predicts brain activity in response to stimuli is essential for the neuroscience community as the model predictions are useful to investigate and test hypotheses about the transformation from stimulus to brain responses both in the healthy brain and their breakdown in clinical conditions <cit.>.
Typically, experimental conditions involve sensory (visual or auditory), or motor stimuli, so an encoding model maps the input stimuli to their encoding representation in the respective brain region <cit.>.

Linguistic stimuli such as words/sentences are extensively used in fMRI experiments. Understanding the association between the semantics of words/sentences and evoked brain activation may throw light on how the brain organizes and represents linguistic information in neural circuits. One of the pioneering works of Mitchell et al. <cit.> proposed distributional semantic models that encode patterns found in fMRI brain activation based on hand-designed features.
Subsequently, models trained using word embedding have successfully overcome the limitations of manually-designed features to build encoding systems <cit.>. Psycholinguistic and behavioral characteristics are also useful for the encoding task <cit.> and for visually-grounded representations <cit.>.
These studies establish a higher correlation between the semantic features and brain activation patterns. However, they do not have a principled way to predict regions that specialize in a particular category of stimuli. 
Instead, they predict the voxel intensity values for the whole brain or a pre-selected set of voxels. 


Classical encoding models focus on univariate fMRI analysis, i.e., toward an understanding of different cognitive processes at individual brain voxels <cit.>. 
Researchers have also explored multi-voxel pattern analysis (MVPA) <cit.> to represent information across an ensemble of voxels.
The critical limitation of MVPA is that it may detect areas where brain activation differs across subjects, even if those differences are unrelated to neural coding.
Moreover, current encoding methods attempt to learn either weights in case of linear models <cit.> or complex representations in non-linear models <cit.>.
Recent studies show that deep learning models (e.g., convolution neural networks and LSTMs) are successful in encoding brain responses for various sensory inputs (auditory, visual, and linguistic) <cit.>. 
However, it remains unclear to what extent the deep learning models can exhibit principles of integration and differentiation that are the hallmark of how the brain responds to sensory stimulation. Moreover, all these models vary in their complexity.  In particular, interpretation of these non-linear models can be difficult due to unexpected and enigmatic representations <cit.>. 

It is often incongruous to get a single global model achieving the best results on the complete problem domain <cit.>. Two fundamental principles of organization of function in the brain seem to be functional differentiation (specialization) and functional integration (Friston, 2002). Extant linear and non-linear models tend to conform to the latter principle by modeling the integrative aspect in a single global model. However, we hypothesize that such integration is mediated in turn by regional functional specialization. Such a framework posits that information organization in the brain is achieved holistically by combining the principles of differentiation and integration. A machine learning framework that elegantly combines these principles is the mixture of experts model <cit.>.


Mixture of Experts (MoE) models <cit.> offer an exciting choice for the problem of learning distinct models for different regions of the input space. In mixture of experts, the feature space is probabilistically divided into several partitions. Every expert specializes over a distinct partition. MoE has been used successfully to investigate the intricate patterns of brain changes associated with non-pathological and pathological processes, such as the effects of growth, aging, injury, or a  disease <cit.>. 
Models involving MoEs have great potential for use in medical diagnostics to diagnose a variety of clinical conditions such as depression, Alzheimers' dementia. 
Yao et al. <cit.> proposed a hidden conditional random field (HCRF) framework in combination with a mixture of experts model to make predictions in all ROIs that are interconnected.


In this paper, we use the MoE model assuming that each expert specializes over a particular brain region (set of voxels that are significantly activated together) based on the category of words that are represented by the model. 
Encoding models have proven successful in using pre-trained word embedding methods such as Word2Vec and GloVe to predict brain responses <cit.>.
Here, we use bidirectional encoder representations from transformers (BERT) embeddings <cit.> to generate a feature vector for input stimuli.
The stimuli used in task-specific fMRI datasets arise from multiple categories of data and yield activation in different brain regions. The main objective is to demonstrate the feasibility of extracting brain activity patterns related to particular regions of specialization using a mixture of regression experts-based model (we call, Expert2Coder) while maintaining comparable accuracy of that of integrative global models.

We conducted simulation experiments demonstrating that functional differentiation into divergent brain regions is indeed achieved with the mixture of regression experts model rather than using a simple linear/non-linear model alone. In summary, we make the following contributions in this paper.

    
  * We propose a mixture of experts based model in which brain activity patterns related to each region of interest (ROI) is represented using a group of experts.
    
  * In particular, we focus on categorizing different brain regions associated with different experts, given input stimuli.
    
  * Showcase and highlight the importance of discrimination across different experts.
    
  * Better the accuracy of the proposed model to that achieved by existing linear and non-linear models, thereby demonstrating the functional integration capability of such mixture models.

 
The rest of this paper is organized as follows. We discuss the proposed mixture of regression experts (MoRE) approach in  Section <ref>, and our enhancements include a detailed analysis of the dataset, insights from analysis of results and discussion in Sections <ref> and <ref>. Finally, we conclude with a summary in Section <ref>.



§ PROPOSED APPROACH: EXPERT2CODER


We use a mixture of experts-based encoder model whose architecture is inspired from <cit.>. The mixture of experts architecture is composed of a gating network and several expert networks, each of which solves a function approximation problem over a local region of the input space. 
Figure <ref> shows an overview of our model where the input is the text vector extracted from the popular pre-trained neural word embedding model, namely BERT <cit.>. 
The input feature representations are given to both the expert networks and the gating network. The gating network uses a probabilistic model to choose the best expert for a given input vector. The corresponding brain activation (for all the voxels) is used as a target vector during training.

As a result of training, the model learns to select appropriate expert via gating parameters in order to predict the whole brain activation for a particular stimulus. Also, the model highlights the specific activated brain regions for a particular stimulus.
A similar architecture is used to build distinct models for different subjects.
In the experiments and results section, we provide an in-depth analysis of the model hyper-parameters and training.




 §.§ Architecture: 




Let S={(_1,_1),…,(_N,_N)} denote the training set where N is the number of examples. _i ∈^n, ∀ i∈ [N][[N] represent the set {1,2,…,N}.] are the input vectors (word embeddings). _i ∈^m, ∀ i∈ [N] are the target vectors (whole brain activation all the voxels in the fMRI images). Let K be the number of experts. 
The mixtures of experts model formulates the conditional density of  given  as a mixture of K different densities as follows.

    p(|) = ∑_j=1^KP(j|,θ_0)p(|,θ_j) = ∑_j=1^Kg_j(,θ_0)p(|,θ_j)

Here, P(j|,θ_0)=g_j(,θ_0) is the probability of choosing j^th expert for a given . Note that ∑_j=1^K g_j(,θ_0) =1 and g_j(,θ_0)≥ 0, ∀ j∈ [K]. g_j(,θ_0) is also called the gating function and is parameterized by θ_0. p(|,θ_j) denotes the density function for the output vector associated with the j^th expert
and θ_j denotes the parameters associated with the j^th expert. 

In this paper, we choose p(|,θ_j) as multivariate Gaussian probability density for each of the experts, denoted by:

    .91!p(|,W_j,Σ_j) = 1/(2π)^m/2|Σ_j|^1/2exp(-1/2(-W_j)^TΣ_j^-1(-W_j))


where W_j∈ℝ^m × n is the weight matrix and Σ_j∈ℝ^m× m is the variance-covariance matrix associated with the j^th expert. Thus, θ_j={W_j,Σ_j}. 
In this formulation we assume that the covariance matrix Σ_j is diagonal. 
Thus, Σ_j = diag(σ_j,1^2,σ_j,2^2,…,σ_j,m^2), ∀ j∈ [K].
Thus, we assume that the components of the output vector y∈ℝ^m are statistically independent of each another. 
We use this assumption to make the model simple by reducing the number of overall parameters. This assumption also makes the algorithm computationally less expensive.
Assuming Σ_j = diag(σ_j,1^2,σ_j,2^2,…,σ_j,m^2), we rewrite the conditional probability density model for j^th expert as follows:

    P( y| x,W_j,Σ_j) = 1/(2π)^m/2σ_j,1σ_j,2..σ_j,mexp(-∑_i=1^m (y_i- w_j,i^T)^2/2σ_j,i^2)

where w_j,i is the i^th row of W_j. We use softmax function for the gating variable g_j(,θ_0).

    g_j(,θ_0) = exp( v_j^T)/∑_i=1^Kexp( v_i^T)

where v_j ∈ℝ^n, ∀ j ∈ [K]. Thus, θ_0={ v_1,…, v_K}.
Let Θ be the set of all the parameters involved for the K-experts. Thus, Θ={θ_0,(W_1,Σ_1),…,(W_K,Σ_K)}.







 §.§ Training Mixture of Experts Using Expectation Maximization (EM) Algorithm

The EM algorithm is an iterative method for finding the maximum likelihood estimate (MLE) of the parameters of a probability model. 




 §.§ E-Step
  
In the E-step, we find the expectation of the complete log-likelihood.


    Q(Θ|Θ^(p)) = ∑_n=1^N∑_j=1^Kh_j^(p)(_n) [log(g_j(_n,θ_0))+ 
    log(P(_n|_n,W_j,Σ_j))]




    Q(Θ|Θ^(p))
      =
      [t]∑_n=1^N∑_j=1^Kh_j^(p)(_n)   [log(g_j(_n,θ_0))
        +log(P(_n|_n,W_j,Σ_j))]


where p is the iteration index and h_j^(p)(_n) is given by

    h_j^(p)(_n) 	= g_j(_n,θ_0^(p))P(_n|_n,W_j^(p),Σ_j^(p))/∑_i=1^Kg_i(_n,θ_0^(p))P(_n|_n,W_i^(p),Σ_i^(p))




 §.§ M-Step

The M step chooses a parameter Θ that maximizes Q function (given in eq.(<ref>)). Thus,

    Θ^(p+1)=Θargmax Q(Θ|Θ^(p))




  * Updating θ_0: We use gradient ascent to maximize Q function with respect to parameters θ_0 as there does not exist any closed-form solution for the maximizer.

    v_j^(p+1)   =  v_j^(p)+η∇_ v_jQ(Θ|Θ^(p))
       =  v_j^(p)+η∑_n=1^N[h_j^(p)(_n)-g_j(_n,θ_0^(p))]_n

where η is the step size.

  * Updating W_j: W_j  comprises m rows _j,i. We derived the closed-form solution for _j,i^(p+1) as follows:


    _j,i^(p+1) 
    = 
    [t][∑_n=1^N h^(p)_j(_n) _n _n^T]^-1   [∑_n=1^N h^(p)_j(_n) y_n,i_n]; 
        j∈[K]; i∈ [m]


where y_n,i is the i^th  element of y_n.

  * Updating Σ_j: 
Σ_j comprises  σ_j,1,…,σ_j,m. We derived the closed-form update equation for each of them as follows:

    σ_j,i^(p+1) 
    = 
    [t]1/∑_n=1^Nh_j^(p)(_n)∑_n=1^Nh_j^(p)(_n)   (y_n,i-_j,i^(p)._n)^2;
        j∈[K]; i∈ [m]



 An iteration of EM increases the original log-likelihood L(Θ|_1,…,_N). That is, L(Θ^(p+1)|_1,…,_N)> L(Θ^(p)|_1,…,_N).
The likelihood L increases monotonically along the sequence of parameter estimates generated by the EM algorithm <cit.>.



 §.§ Selection of Number of Experts


To find the number of experts, we used two methods.

Bayesian Information Criterion BIC  is one of the successful measures to approximate the Bayes factor <cit.>, i.e., to find a model that has maximum posterior probability or maximum marginal likelihood as well as a minimum number of model parameters. 
BIC can be formulated as follows.

    BIC = dlog(N) - 2log( L(Θ|_1,…,_N))

Where d is the number of parameters, N is the number of data points. In our proposed model, where we assume diagonal covariance matrices for each expert, d=K(mn+m+n), where K is the number of experts, n is the dimension of input feature 𝐱, m is the dimension of the output vector 𝐲. 
The objective is to find a model configuration that minimizes BIC. The model complexity increases with the increase in the number of parameters. However, the likelihood will also increase by increasing complexity. Thus, BIC makes a trade-off between the negative likelihood and the number of parameters. There exists an optimal choice of complexity (number of experts here) at which BIC takes minimum value.

Cross-Validation Since cross-validation technique was successful in predictive modeling framework <cit.>, we tested our MoRE model empirically with different number of experts – models with 2-, 3-, 4-, and 5-experts were evaluated. 
MoRE model exhibits better separation of word categories with 5 experts compared to the other configurations. While we report the results of the model with 5-experts in the main text, those of the others are included in  supplementary Section <ref>.




§ DATASET DESCRIPTION & ANALYSIS





In this section, we describe the dataset used for training and testing. In the next subsection, we present the process of selecting an appropriate number of experts in a model.


We used data from paradigm 1 of fMRI experiment 1 <cit.>, where authors conducted experiments on multiple subjects by asking them to passively view different category of words as stimuli (images adapted from[<https://osf.io/crwz7>]) as shown in Figure <ref>. 
Each category might correspond to activation of distinct brain regions. 
In paradigm 1, a target word is presented along with a picture that depicted some aspect(s) of the relevant meaning. This fMRI dataset was collected from a total of 16 participants. 

For each participant in the experiment, a whole set of 180 words (128 nouns, 22 verbs, 29 adjectives and adverbs, and one function word) were presented as stimuli, both the word along with a semantically related picture.
While participants viewed the stimuli, fMRI scans were collected. Each brain volume (scan) consists of 85 slices, each of size 88 × 128 voxels. 
Each participant saw stimuli, each repeated between 4 and 6 times. Here we compute the average brain response per stimulus by combining scans from all these repetitions.




When we describe results, we depict the results of slices 10 through 77 as there is no activation in the remaining slices (the first 9 and the last 8 slices) in a brain volume. 




  
Model Training: Input Representation
We use the publicly available pre-trained word embedding method BERT <cit.> to formulate feature vectors corresponding to the 180 words used as input stimuli. Thus the BERT-based embedding yields 786-dimensional vector for each word.


  
Model Training: Output Representation
The fMRI brain responses (voxel activation) corresponding to every stimulus were used as target output for the subject-specific encoding model. 
The dataset contains average activation (of the 201,011 voxels) for each of the 180 word-stimuli per participant.
We removed the zero intensity values from the voxels resulting in 199,658 voxels as our target output.
We follow the 5-fold cross-validation approach, where 80% of the data (144 words) were used for training and 20% of the data (36 words) for testing.





  
Atlas for ROI Representation
We use the Automated Anatomical Labeling (AAL) atlas <cit.> with a parcellation of 116 brain regions to represent the brain activation response for each stimulus, where each voxel coordinate belongs to a particular region of interest.



 §.§ Choosing the Number of Experts: 








To get the optimal number of experts, along with cross-validation approach, we calculated the BIC scores using the diagonal covariance matrix where we consider the diagonal covariance matrix for each component. 




As shown in Figure <ref>, we calculated the BIC scores using four different types of covariance parameters such as (1) `full': 



Here, we consider the logarithm of BIC values in Figure <ref> for better visualization.
Although, the BIC values in Figure <ref> seem to be  approximately similar, the model with 5 experts has a lower BIC value of 21.19.


We observe a similar pattern across all the 16 subjects, that is, the optimal model is found to be the one with 5 experts. 



§ DATA EXPLORATORY EXPERIMENTS


Before embarking on the training of encoding models, we wanted to investigate the regularities inherent in the original stimulus representations using BERT-embedding as well as the representational similarity of brain responses corresponding to different stimuli. In this section, we describe two exploratory experiments to investigate the nature of the semantic relatedness among the stimulus data as well as among the related brain response data.   
   


 §.§ Semantic Relatedness of the Stimulus Word Vectors



We characterize the semantic relatedness of the words used as stimuli by performing clustering of 180-word stimuli (features extracted from BERT). As seen in in the word-cloud visualization in Figure <ref>, related words tend to appear together.
Here, we choose five clusters based on observation that the optimal model had 5 experts as described in Section <ref>. We use K-means algorithm for clustering.



These word clusters provide insight into how some words are highly correlated when cosine similarity (or correlation) measure was used to investigate semantic relatedness among the word-embedding vectors.
We chose the number of clusters using the BIC & cross-validation methods described above. 
From Figure <ref>, we observe that semantic word pairs such as (“great”,   “smart”, “pleasure”, “charming”, “hurting”  & “feeling") are grouped together in Cluster 1. Similarly in Cluster 4, we have (“mathematical”, “science”, “law”, “professional”, “philosophy”,  & “economy"), (“election" & “nation"), etc. Although, we often find similar pairs in the same cluster, we also find few uncorrelated words in every cluster. 





 §.§ Clustering of the fMRI Brain Activation Vectors




Understanding how the brain represents semantics is still in its relative infancy: how concepts are represented and combined is unclear and how this is manifested in the brain activation when subjects view the words passively is still an enigma. 

In order to understand the representational relatedness of the brain activation, we performed clustering of the fMRI brain activation vectors corresponding to the 180 stimulus words. The results are shown in Figure <ref>. 
Similar to  section <ref>, we use K-means clustering and the number clusters considered is 5, based on the observation that optimal model had 5 experts as described in section <ref>.
Thus we consider the number of clusters as 5 using the BIC method as well as the cross-validation method described above. 


From Figure <ref>, we can observe that words “skin" and “brain" fall in Cluster 5, whereas the word “body" falls in Cluster 3.
This result indicates that the semantic representations reflected in the brain activity might depend on the nature of the input stimuli that are shown to the subject and further might be related to how the subject responds to the image and word combination with his/her imagination and personal experience. One striking observation is that the semantic relatedness captured in Figure <ref> is distinctly different from that in Figure <ref>, except for a few odd words captured in the same cluster (for example, words such as doctor, clothes, religious find themselves in the same cluster in both the cases). This is understandable as the BERT-embedding captures word co-occurrence relationship whereas brain activation is related, among other things, to the personal experience of the subject. Another possibility why these representations are different could be the ambiguity of the word-picture pairs displayed as stimuli, for example if we observe from  Figure <ref> the image is shown for the word “Plant" looks ambiguous. The subject might be thinking of this as “seed/s" that might in future become a plant, and hence the corresponding brain activation might include all of these inner responses evoked based on the word-picture stimulus.
The exploratory analysis suggests that although clusters may be formed based on similar semantic representations, however, each cluster may contain both correlated and uncorrelated words.




In the next section, we present the results of the proposed MoRE models as to how each expert learns the associative relationships among the word embedding and the corresponding brain activation responses. 









§ EXPERIMENTAL RESULTS AND DISCUSSION


Here, we used paradigm-1 of experiment-1, where the target word was presented along with a related visual image (picture).
At the input level, we considered the BERT-based pre-trained embedding for extracting the features (768 dimensions) to train the models corresponding to the  three methods, namely the proposed MoRE model compared with Ridge Regression model and multilayer perceptron (MLP). The target output (brain activations) for all the methods has a very high dimension of 199658 (corresponding to the number of voxels in a brain scan).

Although we use log-likelihood in MoRE method, mean squared error (MSE) in Ridge and MLP methods as loss functions, we also consider the mean absolute error (MAE), R^2-score , precision, recall and F1-score as the metrics to evaluate the performance of our model. We report the the details of model performance in the next subsections.



We split the stimulus data into 144 words used in training and the remaining 36 words as the testing set. The encoding performance was evaluated by training and testing models using different subsets of the 180 words in a 5-fold cross-validation scheme. 
The encoder models were trained until the model reached convergence with a lower error bound of 1e^-10 or till a maximum number of 200 iterations. In order to systematically present the results, the results section is divided into six parts:

Section <ref> discusses the training performance of baseline (classical) methods: linear regression <cit.> and MLP <cit.> for predicting the brain activity patterns.
In Section <ref>, we describe the training results of the proposed MoRE model for predicting the brain activations.
We present the comparison of classical results and the proposed MoRE method in Section <ref>.
Section <ref> illustrates the statistical comparison between MoRE and Baseline methods. After presenting the comparative results, in Section <ref> we investigate the results of the MoRE model to highlight what each expert has actually learned in terms of the brain regions and associated words. Then we present the ROI predictions of the model for unseen words in Section <ref>. 




 §.§ Training of Baseline Methods




  
Ridge Regression Method
In the literature, linear regression has often been successfully used as a simple encoding model by the neuroscience community <cit.>. With the semantic features extracted from BERT-base as input as well as fMRI blood oxygen-level dependent (BOLD) activity collected from each subject as output, we were able to build a regression model to map the association between input (word features) and output (fMRI activations).
To avoid over-fitting, we used ridge regression method to train the model.
During model building, we used MSE as the loss function in the training, whereas mean absolute error (MAE), R^2-score considered as metrics to measure the model performance. 
The ridge regression method reports 5.5 MAE and 0.15 R^2-score.


Figure <ref> shows the predicted fMRI brain activation (fourth row) for the word “Mathematics”.



  
Multi-layer Perceptron Method (MLP)
In this paper, we use a 3-layer neural network to build an MLP method that predicts the neural activation pattern for a given stimulus (word vector) at the input layer.
We use 1000-hidden neurons in the hidden layer, connection weights between the input layer and hidden layer learned through an adaptation process.
The output layer provides the prediction of fMRI activations as a weighted sum of neural activations contributed by each of the hidden layer neurons.


The predicted result for the word “Mathematics” is shown in Figure <ref> (fifth row) depicts the BOLD activations across the slices.
The model predicted brain activations are measured by using MAE, R^2-score as the metrics.
The MLP method yields 4.63 MAE and 0.35 R^2-score, performs better than Ridge regression method.




















 §.§ MoRE Training



Using the approach discussed in Section <ref> and using the insights from the experiments in Section <ref>, we trained a separate mixture of five-regression experts (MoRE) model for each subject.
We performed experiments on the dataset where the stimulus (text) vector extracted from the recently successful neural word-embedding method, namely, BERT, was given as input to the model and estimated the corresponding brain activation response as the output of the model.









 §.§ Comparison of Results of MoRE model with those of the Baseline Models



We predicted the brain activation using classical models as well as the proposed MoRE method.
To statistically verify how our results are reliable and finding the variance in predicted brain activations from the observed average voxel intensity values, we use three-Sigma rule implies that heuristically nearly all values lie within three standard deviations of the mean.
Table <ref> shows the macro/micro-average precision, recall, F1-score of 16 participants obtained using classical methods such as Ridge regression and MLP and compared with those of the MoRE method.
Here, the macro averaging gives equal weight to each class to evaluate the performance of various methods across the two-classes. On the other hand, micro-averaging method calculates the individual true positives, true negatives, false positives, and false negatives of the the binary-class model.


In Table <ref>, we divided the ground truth voxel intensity values of test data into the following distributions such as “μ-3σ”, “μ-2σ”, “μ-1σ”, “μ”, “μ+1σ”, “μ+2σ”, “μ+3σ”, where “μ” is mean and “σ” stands for standard deviation.
For each ground-truth test word, we calculate the “μ” and “σ” from the 199,658 voxel intensity values.
We make the voxel coordinates (199,658 voxels) into 2 classes for each of the above-mentioned distributions for each of the three methods.

We consider the following steps for getting the classification metrics.



    
  * For each test word, we predicted the brain activations from the three methods such as Ridge, MLP, and MoRE.
    
  * We take the predicted brain activations from the previous step and assign the intensity values higher than “μ-3σ” to one class and the remaining voxel coordinates as belonging to the zero class.
    
  * We perfrom a similar analysis on the empirically observed brain activations. We use the observed brain activations with voxel intensity values higher than “μ-3σ” as one class and the other voxel coordinates as belonging to zero class.
    
  * We compare the voxel coordinates of ground-truth and predicted from the previous two steps, and estimate precision, recall, and F1-scores for the three methods shown in Table <ref>.
    
    
  * We repeat the above steps for  other intensity value distributions (corresponding to the values that are 1- or 2- standard deviations above and below “μ”) for the three methods.
    
    


We can observe from the Table <ref> that results with MoRE based method are better compared to classical models. 


The macro average recall score for the MoRE method is higher than the two methods in all the seven distributions.
In contrast, precision for two baseline methods is higher than MoRE, which mainly comprises false positives (some other voxel coordinates that become activate spuriously).
Of the two baseline methods, MLP method results in more false positives as compared to Ridge regression.

We have shown the actual brain responses for the word “mathematics” and predicted brain activation of the three methods (2^nd row MoRE, 3^rd row Ridge, and 4^th row MLP) for the test word “mathematics” in Figure <ref>. As shown in Figure <ref>, we observe that similarities between ground truth and cortical brain responses from the MoRE-based encoding model are better with a near-perfect recall, as described above.  This result encourages us to believe that MoRE  performs prediction based on a good semantic understanding of the stimulus and the associated brain activation. 

Figure <ref> compares the performance of the baseline ridge regression model, MLP, and the proposed MoRE model. 
The proposed MoRE model has a better recall score for all the subjects except for subject-7 as compared to the linear and non-linear baseline models. 



 §.§ Statistical Analysis of the three methods
 

To compare the three methods statistically, one-way ANOVA  was performed on MAE of the three methods. Results confirmed that the three methods were statistically significantly different, with an F-statistic [F(2,3) = 19.99, p = 4.4e-10] 

We use post-hoc Scheffe’s test to the obtain the results between
different pairs such as MoRE vs MLP (p = 0.0263), MoRE vs Ridge (p = 2.16e-8), and Ridge vs MLP (p = 0.0011), again reiterating the superior performance of the proposed model with respect to baseline methods.
Additionally, from Figure <ref> we can also observe that the average MAE error for the models using MoRE, Ridge, and MLP are significantly different, with MoRE performance being superior to other methods.




 §.§ MoRE ROIs Learned by Each Expert



In the previous section comparative study of the proposed MoRE model with baseline methods suggested superior performance of the proposed model. In this section we delve deeper into what the model actually learns.
As we observe the similar set of pair of words for training and testing in each fold, the words illustrated in Table <ref> corresponding to one of the 5-fold data.
Table <ref> displays the words that are categorized by each expert in the train and test datasets and the corresponding highly activated brain regions. 



From Table <ref>, we observe that each expert exhibited distinct associations between word stimuli and the corresponding brain activation in both training and testing experiments. For example, while expert-1 has a higher probability for the word “science", the same expert displays higher probability for a semantically related test word “mathematical". 
To identify the brain regions which are associated with words in each expert, the following steps were taken.

    
  * We generated a matrix of size [#No of high-probable words for each expert × Brain ROIs (regions corresponding to the activated voxels)] for that expert. 

    For example, we can see from Table <ref> that Expert-2 captured 18 words in training (with 116 corresponding ROIs), yielding a matrix of size 18 × 116. 
    
  * To identify the ROIs associated with each expert, we applied principal component analysis (PCA) on the above matrix (#words × #regions) and extracted principal components (PCs) with a maximum explained variance ratio of 85% resulting in a matrix (#words × #components).
    
  * This matrix would enable us to identify the most critical variables in the original feature space that have a maximal contribution. 
    
  * The region's importance is calculated by using the simple matrix multiplication of (#regions × #words) and (#words × #components). Note that the former is the transpose of the matrix whose PCA was performed in the earlier steps.
    
  * By considering the features that have a positive magnitude, from each component we select the regions which have a score greater than 0.2.
    
  * This process is repeated for each of the experts and column-4 in Table <ref> displays the most important regions corresponding to a particular expert.












It can be observed from Table <ref> that the experts correspond to distinct (specialized) ROIs based on joint learning of semantic aspects of the word stimulus and the associated brain regions related to the meaning of the word. 
This aspect can be observed in all the experts, especially in the set of words that have a close correspondence between training and test conditions. 
Further, it has to be noted that the ROIs activated are known from previous studies to have compatibility with the semantics they seem to encode, few of these examples are discussed later. In order to capture all these joint associations in a visually convenient manner, we depict the words and the associated regions of activation on a 3D-rendered depiction on either the left and right cortical surfaces or the dorsal and ventral surfaces of the brain in Figures <ref> -  <ref>. This depiction, the so-called brain dictionary, is inspired from the recent work of Huth et al. <cit.> who suggested that every word activates multiple regions of the brain.




Several brain regions such as “Angular_L,R”, “Lingual_L,R”, “Precentral_L,R”, “Postcentral_L,R”, “Cuneus_L,R”, “Frontal_Sup_L,R”, “Frontal_Mid_L,R”, “Precuneus_L,R”, “Cerebellum_Crus_1L,1R”, “Temporal_Sup_L,R”, and “Temporal_Mid_L,R” 
are commonly activated among all the experts (results not shown in  Table <ref>). 
The common ROIs seem to be related generally to visual-spatial processing, sensory processing, attention, etc. that seem to be shared for all the words and may be related to the processing of the visual stimulus presented along with the lexical input (word). Apart from the common activation, there are also unique regions captured by experts in a semantically appropriate fashion. Some of the notable examples are highlighted here. 

From the Figure <ref>, we observe that Expert1 seems to code for action words such as “play", “spoke", noun-verb co-occurrences such as “event-spoke", “do dance", “dig",  “do music", “play movie", visual-spatial related words, including, camera, picture, texture, etc <cit.>.
The cortical areas associated with movement such as the Supplementary Motor Area, Cerebellum, Putamen, Caudate seem to be active in Expert 1.


It appears that Expert 3 
codes for face recognition words such as “laugh", “emotions", “feelings", “hurting", “sexy", and “angry", problem solving words such as “investigation", “mechanism", “news", etc.  The brain activation in the Fusiform gyrus that lies between the Parahippocampal gyrus and the Lingual gyrus medially seem to be compatible with face processing <cit.>.

Cortical areas associated with movement such as the Supplementary Motor Area, Cerebellum, Putamen, Caudate seem to be active in Expert 4 that seems to code for action words such as “read", “spoke", noun-verb co-occurrences such as “event-spoke", “do dance", “dig",  “applause", “counting", etc <cit.>.


Expert-5 codes for abstract words such as accomplished, pleasure, elegance, dedication, the brain activation in the polar regions of the temporal lobe known for involvement in higher-order language comprehension seems compatible. 
Similar correspondences can be seen in several other words such as read, noise activated in both temporal and frontal area known for hearing and speech functionalities respectively.. 














 §.§ ROI Prediction for Unseen Words










To measure the efficacy of our proposed MoRE model for the word+picture condition, we tried to predict the brain regions for unseen words, i.e., those that are not present in the dataset. We chose four different words such as “physics", “lunch", “generous", and “cat" which are semantically related to words in the existing dataset but not explicitly given while training, model prediction results are displayed in Table <ref>.  For the word “physics", MoRE model chose expert-1 with higher probability among the five experts and expert-1 earlier captured the words science and mathematics in training & testing, respectively as shown in Table  <ref>. The results of this unseen-word experiment give credence to our hypothesis that learning functional differentiation (or specialization) while achieving comparable overall accuracy can be implemented with the mixture of regression experts (MoRE) framework. 




§ STIMULUS PRESENTATION OF TARGET WORD AND A RELATED WORD CLOUD

In this experiment, we used brain activations from paradigm-3 of experiment-1, where word cloud has shown along with the concept word to the participants. We similarly trained this model as described in Section <ref>.
















Table <ref> shows the unique brain regions activated in five different experts.

ROIs activated commonly across all the experts are listed in the last row in Table <ref>. 
These areas seem to be related generally to visual-spatial processing, sensory processing, attention, etc. that seem to be common for all the words observed in the word+picture model shown in Table <ref>.
Interestingly, it can observe from both  Tables <ref> and  <ref> that the model trained on word+picture discriminates all the words associated with brain regions better compared to the other model trained using the word cloud. The visual input rather than the semantic context seems to give rise to better associative learning. Anderson et al <cit.> made a similar distinction between concrete and abstract nouns where visual input seemed to enhance accuracy for concrete nouns, whereas semantic models facilitate abstract nouns.




§ CONCLUSION


In this paper, we present a mixture of experts based model (Expert2Coder) where a group of experts captures brain activity patterns related to particular regions of interest (ROIs) and also shows semantic discrimination across different experts.
Different from previous works, the underlying model depicts that each expert trains on particular brain regions of interest (set of voxels that are significantly activated) based on the semantic category of words that are represented by the model.

Various experiments demonstrated the efficacy and validity of the proposed approach. Notably, the last experiment with words that were not used in training, demonstrates the power of such encoding models that learn a joint association between semantics from linguistic representation and brain responses. These models can potentially predict the brain response corresponding to new words.

In future, we plan to experiment on Spatio-temporal fMRI datasets, with a primary focus on the hierarchical mixture of experts at slice-level instead of voxel-level predictions at each instance. 


























IEEEtranS







§ SUPPLEMENTARY RESULTS



In this section, we report the MoRE model results when tested with a different number of experts.
As seen in the 2-experts MoRE model results visualization shown in Figure <ref>, some related words tend not to appear together.
Besides, the existence of different categories of words present in 2-experts is not sufficient for learning different ROIs.
Similarly, Figure <ref>, <ref> describes the MoRE results with 3-experts and 4-experts.
From Figure <ref>, we observe that expert-1 tries to learn a specific category of words and corresponding brain ROIs.
Also, the observations from Figure <ref> that 3 out of 4 experts categorized the words and learned the unique brain ROIs. 
Overall, we achieved the optimal number as 5-experts discussed in the paper.








