Language Reconstruction with Brain Predictive Coding from fMRI Data
fMRI-to-text decoding;predictive coding theory
> In the regions of interests selection experiment, the authors only consider 'random,' 'whole,' and 'BPC' as the ROIs...

The selected BPC area (superior temporal sulcus, angular gyrus, supramarginal gyrus, and opercular, triangular, orbital part of the inferior frontal gyrus) contributes the most to predictive coding, as indicated in some neuroscience studies [1][2].

The process for selecting "BPC": For the Narratives dataset, Destrieux atlas is applied and the above mentioned ROIs are extracted. For LeBel's dataset, since the fMRI signals are not projected to a standardized space, we use the “Auditory” region provided by the authors', containing parietal-temporal-occipital (PTO) area. The BPC area of both datasets cover highly similar area.

The process for selecting "random": For the Narratives dataset, G_and_S_cingul-Ant, G_and_S_subcentral, G_and_S_transv_frontopol, G_orbital, S_front_middle, S_subparieta are selected. For LeBel's dataset, we randomly choose 1000 voxels from brain surface data.

The process for selecting "whole": We use the whole brain surface data as ROIs for both datasets.

We believe selecting random and whole ROIs as controlled experiments is sufficient for demonstrating the effectiveness of using predictive coding to improve decoding performance: 

1. random vs. BPC demonstrates only ROIs related to predictive coding in human language comprehension can improve decoding.

2. whole vs. BPC not only confirms conclusion in 1, but also shows whole brain surface which contains BPC area still can't contribute to better decoding, because some other brain regions contain too much noise.  

3. none (PREDFT without SideNet) vs. BPC. PREDFT without SideNet is equivalent to not using any ROIs for predictive coding. This comparison shows predictive coding improves decoding accuracy significantly.

**All the above clarifications are included in sec 4.3 and Appendix A.4. We will add more key information in sec 4.3 in the updated version**

[1]. Evidence of a predictive coding hierarchy in the human brain listening to speech. Nature Human Behavior
[2]. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature

> Could the authors provide pseudocode for the method...

We will provide pseudocode in the appendix for edited version of paper. The discussion of time complexity is already in Appendix A.4 (line 845) of original paper.

> The results provided by the authors mostly only include the mean value...

We guess you indicate the experiment of analyzing the impact of prediction length and distance to model performance (sec 4.4), as we have presented results per subject for other experiments. **The per-subject results of analyzing the impact of prediction length and distance are already presented in Figure 16,17,18 in the appendix of original paper.**

> In the methods section, some symbols are not defined...

**A notation table for symbols is presented in Table 3 in the appendix of original paper.**

### Questions

Please refer to clarification for weaknesses.
Rebuttal (part 2)
We sincerely appreciate your effort in reviewing our paper. We will address your concerns point by point.

### Weaknesses

> In Section 3.3, the authors state, 'During the inference stage, as illustrated in Figure 8, the decoder in the side network is abandoned.' However, they do not provide a detailed explanation of why the decoder is discarded or discuss the potential impact of this decision...

What we really want is predictive coding representation, which is produced by side network encoder. The side network decoder is designed to help train the side network encoder (we can't find out how to directly obtain predictive coding representation). 
Specifically, during the training process, the label for side network decoder is predicted words instead of complete sentences (as shown in Figure 3). The side network learns mapping between specific areas of brain (BPC area) and predicted words.
However, the goal of our task is to decode complete sentences. So the side network decoder is useless after training.

**We will add the motivation and reason for discarding the side network decoder in the methodology section in the updated version.**

> As shown in Table 1, PREDFT does not achieve the best performance on ROUGE1-R...

We think the different lengths of generated content might contribute to this factor, since we don't apply a word rate model to control the number of generated words.
Although PREDFT fails to outperform other models in ROUGE1-R, the gap is narrow.
Just like recall and precision to f1 score, ROUGE-Recall measures the extent to which a machine-generated content captures the information contained in a reference content, which is a single perspective assessment. ROUGE-Recall and ROUGE-Precision are characterized by a trade-off. When PREDFT gets a relative low ROUGE-R, it gets a high ROUGE-P.
Instead, ROUGE-F1 is a more comprehensive indicator, combining both ROUGE-P and ROUGE-R. Our model outperforms other models in this metric. 

> As shown in Table 1, PREDFT without SideNet performs similarly to other methods. However...

The SideNet is designed to obtain predictive coding representation.
PREDFT without SideNet can be viewed as traditional deep learning approach which directly applies Transformer to decode text from brain recordings, while PREDFT with SideNet combines deep learning and neuroscience findings (predictive coding). The SideNet provides predictive coding representation to the decoder in Main network, and the decoder incorporates both current fMRI representation and predictive coding representation for text decoding.

The idea of PREDFT is motivated by predictive coding theory in Neuroscience, which indicates human can naturally predict upcoming words. Since predictive coding has been verified to contribute to human language comprehension, we seek to investigate whether such predictive information can help language reconstruction. 
The improvement of incorporating SideNet highlights 1. the effectiveness of our model design 2. predictive coding has potential to improve brain-to-text decoding. **We will provide the illustration of PREDFT without SideNet for better understanding in the updated version.**

> Although the authors provide a detailed description of the hyperparameter selection...

All the hyperparameters are chosen to minimize the training & validation loss as much as possible.
We don't understand which hyperparameter the reviewer is confused about.
The influences of ROIs selection, prediction length, prediction distance,  $ \lambda $ to model performance are detailedly discussed in sec 4.3, sec 4.4, Appendix E. The learning rate is set to stabilize training. We don't test the influence of model layers (e.g. Transformer layers) due to limited computational resources.
Rebuttal (part 1)
Yes
No ethics review needed.
Please see weaknesses. I would need to be convinced that majority of the claimed improvements in the model are not merely from a bias towards outputting high-frequency words, and thereby overfitting the chosen test metrics of BLEU and ROGUE, in order to change my score. Right now, I am fairly convinced that this is the case.
The attempt to use hypothesized predictive coding representations to enable better text decoding is interesting.
The paper describes a decoding method "PredFT" that uses a main decoding network and a side network to perform decoding from fMRI recordings of subjects listening to stories to text. The side network is responsible for obtaining predictive coding representations from specific brain regions and integrating them into the main network, enhancing language decoding. The authors claim that this integration leverages brain regions known for predictive functions (like the parietal-temporal-occipital areas) to better align brain signal decoding with anticipated semantic content. This is supported by results that have claimed the brain performs predictive coding during language stimulation.
My main concern is that the metric does not seem to produce even locally coherent text, which substantially damages the authors' claims that this method is an advancement over prior work, such as Tang et al., which uses an LM to guarantee local coherence. Consider the following example from the case study: "He don’t know my girl you of the eyes but his girl sleep he and he said and he said and the to the and and which I not wrong. But the Guy". Clearly, this has no meaning, and does not even obey basic local grammatical rules (e.g. "and and"). The problem seems to be that the model has merely learned repeat short, high-frequency words like "the", "he" and "and", which improves BLEU/ROGUE score but does not actually move forward towards the goal of better language decoding. I imagine if you just had the model repeatedly and randomly output words sampled from the top 100 most common English words that it would behave fairly similarly. My expectation is that a small percentage of the improvement in BLEU score is genuinely derived from brain signals, with most of the benefit deriving from this output bias. The unreasonably high 5.62 BLEU-3 score when compared to other methods is more of a red flag, because its pretty clear that the model is simply guessing every high frequency trigram in the English language.

 The paper is also quite difficult to read for no reason and pointlessly notational, for example when the self-attention equation is repeated three separate times in only slightly different ways.
Yes
No ethics review needed.
1. In Section 3.3, the authors state that the decoder in the side network is abandoned during the inference stage. Could the authors provide a detailed explanation of why the decoder is discarded and discuss the potential impact of this decision on the overall performance and functionality of the model?

2. As shown in Table 1, PREDFT does not achieve the best performance on ROUGE1-R. Could the authors analyze the potential reasons for this and discuss any factors that may have contributed to the lower performance in this specific model? For instance, how might the model's architecture, training process, or characteristics of the ROUGE1-R metric explain this discrepancy? Did the authors observe any patterns in the types of language constructs where PREDFT underperformed on ROUGE1-R?

3. As shown in Table 1, PREDFT without SideNet performs similarly to other methods, while the inclusion of SideNet leads to a significant performance improvement. Could the authors provide a detailed analysis of this phenomenon to explain how SideNet contributes to the model's enhanced performance?

4. Although the authors provide a detailed description of the hyperparameter selection, could they explain the rationale behind these choices? How do these choices relate to the model's performance or the underlying theory of predictive coding?

5. In the regions of interest selection experiment, the authors only consider 'random,' 'whole,' and 'BPC' as the ROIs. Could the authors clarify whether there are other potential ROIs associated with predictive coding? If so, could they provide supporting neuroscience literature for the selection of BPC? Additionally, can the authors explain the process for selecting these particular ROIs and why they believe these are sufficient to demonstrate the effectiveness of their approach? Did the authors consider any other ROIs, and if so, why were those not included in the study?

6. Could the authors provide pseudocode for the method and an analysis of its time complexity to enhance the reproducibility of the article?

7. The results provided by the authors mostly only include the mean value. Could the authors include the variance and statistical test results in the experimental results?

8. In the methods section, some symbols are not defined. Could the authors compile a list of symbols used in the paper in an appendix to help readers understand better?
1. Integrating predictive coding theory into the decoding process offers a fresh perspective on reconstructing language from brain signals.
2. Experimental results demonstrate that PREDFT outperforms other methods across various evaluation metrics, showing significant improvements.
The paper presents PREDFT (FMRI-to-Text Decoding with Predictive Coding), a novel framework that utilizes predictive coding to translate fMRI signals into continuous language. This approach combines a primary decoding network with an auxiliary network focused on capturing brain predictive coding, aiming to improve the accuracy of language reconstruction from brain signals. The authors conduct experiments on two established naturalistic language comprehension fMRI datasets, showing that PREDFT achieves state-of-the-art performance across multiple evaluation metrics.
1. In Section 3.3, the authors state, 'During the inference stage, as illustrated in Figure 8, the decoder in the side network is abandoned.' However, they do not provide a detailed explanation of why the decoder is discarded or discuss the potential impact of this decision. It is recommended to elaborate on the rationale behind this choice and its implications on the overall performance and functionality of the model.

2. As shown in Table 1, PREDFT does not achieve the best performance on ROUGE1-R. The authors should analyze the potential reasons for this and discuss any factors that may have contributed to the lower performance in this specific model. For instance, the model's architecture, training process, or characteristics of the ROUGE1-R metric that might explain the discrepancy. If the authors observed any patterns in the types of language constructs where PREDFT underperformed on ROUGE1-R.

3. As shown in Table 1, PREDFT without SideNet performs similarly to other methods. However, the inclusion of SideNet leads to a significant performance improvement. The authors should provide a detailed analysis of this phenomenon to explain how SideNet contributes to the model's enhanced performance.

4. Although the authors provide a detailed description of the hyperparameter selection, they do not explain the rationale behind these choices. How these choices relate to the model's performance or the underlying theory of predictive coding.

5. "In the regions of interests selection experiment, the authors only consider 'random,' 'whole,' and 'BPC' as the ROIs, which appears somewhat limited. The paper does not clarify whether there are other potential ROIs associated with predictive coding, nor does it provide supporting neuroscience literature for the selection of BPC. It is recommended to either justify the choice of BPC with relevant references or explore additional ROIs to strengthen the study's validity. If authors can explain the process for selecting these particular ROIs and why authors believe these are sufficient to demonstrate the effectiveness of their approach. Additionally, if authors considered any other ROIs and why those were not included in the study.

6. It is recommended that the authors provide pseudocode for the method and an analysis of its time complexity to enhance the reproducibility of the article.

7. The results provided by the authors mostly only include the meanvalue. The experimental results should provide the mean, variance, and statistical test results.

8. In the methods section, some symbols are not defined. It is recommended that the authors compile a list of symbols used in the paper in an appendix to help readers understand better.
Yes
No ethics review needed.
(1) Why did the author only use BLEU and ROUGE in the experiment? Why doesn't the author use WER, METEOR, and BERTScore which is used in the Tang and MapGuide? BLEU and ROUGE both evaluate the matching degree of n-grams, which can easily lead to surface matching but semantic mismatch. METEOR and BERTScore can better reflect semantic similarity.
(2) Many of the methods compared by the author incorporate LLM, while the author's model is entirely trained with their own transformer. Does this result in the author's method being inferior to the baseline method in terms of semantic similarity?
(3) The author's method was inspired by predictive coding and validated it on LLM using a prediction score. But can the author's own model still observe the same phenomenon on the prediction score? I haven't seen the same experiment evaluating the author's own model.
(4) In some parts of the paper, fMRI is spell as FMRI.
The author provided sufficient experiments to demonstrate the significance of his motivation.
In the submission-6263, the authors  propose PREDFT (FMRI-to-Text decoding with Predictive coding) , which was inspired by predictive coding theory. This theory suggests that when humans listen to a certain speech, their subconscious brain predicts the words they may hear next. Then the author validated this theory through a prediction score. The verification method is to first calculate the correlation coefficient between the features extracted by LLM at the current location and the brain features, and then add the features of an upcoming text segment to the current location features, calculate the correlation coefficient again, and observe the changes in the correlation coefficient. The experimental results show that incorporating upcoming text features can increase the correlation coefficient between LLM features and brain features. Based on the above experimental results, the author designed their own model, which includes the side network to decode upcoming text. In the decoding of current text, the feature from the side network is used to incorporate the predictive coding theory into the method.
Although the author's explanation of motivation is very sufficient, I still have a few major questions about the author's method and list them in the questions part.
Yes
No ethics review needed.
1. What would be the chance-level performance when reconstructing continuous language? Is there a baseline available for comparison? Additionally, what is the percentage of overlap between random ROIs and whole-brain voxels? Did the authors repeat the selection of random ROIs multiple times to ensure robustness, or did they only select a single set of random ROIs?
2. What is the rationale for using 4D volume data from the Narratives dataset while using 2D brain data from the Moth Radio Hour dataset? Since the Narratives dataset includes both smoothed and unsmoothed versions, along with brain masks to select activated voxels from the 4D volume, why did the authors make these choices regarding data representation?
3. There is no interpretation provided for the two encoders used in PREDFT. The authors could project these voxels onto brain maps to verify the quality of their encoders.
4. Figures 3, 4, 6, and 8 appear redundant. The authors could combine these into a single figure with a comprehensive caption, instead of presenting multiple, repetitive figures.
5. What does the y-axis represent in Figure 9?
5. Several major questions are raised in the weaknesses section.

Typos:

1. Line 35: Bhattasali et al. (2019); Wang et al. (2020); Affolter et al. (2020); Zouet al. (2021)  - > (Bhattasali et al. 2019; Wang et al. 2020; Affolter et al. 2020; Zouet al. 2021)
1. The motivation for using predictive coding in continuous language reconstruction is clear and well-explained.
2. The proposed approach aims to improve the reconstruction of narrative stories from fMRI brain data. This is a very interesting research area because reconstructing language is challenging due to the slowness of the hemodynamic response.
3. The authors compared the reconstruction performance using evaluation metrics against recent studies. Additionally, ablation studies were conducted on the proposed approach, with and without the predictive coding component.
Recent brain decoding studies have demonstrated that speech perception can be decoded from fMRI recordings and subsequently reconstructed as continuous language. These studies reconstruct continuous language either from specific regions of interest (ROIs) or from the whole brain, using decoder-based language models like GPT-2. Additionally, recent predictive coding studies reveal that the human brain naturally engages in continuously predicting future words across multiple timescales. Building on recent linguistic brain decoding research and the predictive coding approach, this paper explores predictive coding theory in the context of continuous language reconstruction. To this end, the authors propose PREDFT (fMRI-to-Text decoding with Predictive Coding), which consists of a main decoding network and a side network (the predictive coding component). Experimental results on two naturalistic brain datasets (Moth Radio Hour and Narratives) indicate that PREDFT achieves superior decoding performance when comparing the actual story with the reconstructed story.
1. There are several major weaknesses in this work, particularly concerning the evaluation of reconstruction results:
	- A major concern is that the current study (PREDFT) does not provide a clear evaluation of reconstruction results compared to the baseline paper by Tang et al. (2023).
	- For example, the authors did not evaluate the word rate in the generated narrative story. Since the fMRI data was captured while participants were listening to stories, each word has an onset and offset. Similarly, during decoding, what is the word rate predicted by the proposed model, and does this word rate match the actual word rate of the original stimuli? 
	- Therefore, comparing the reconstructed stimulus to the ground truth (i.e., the actual transcripts of the stimuli) would provide a good sense of whether the outputs are meaningful, as the dataset includes the ground truth of what words participants heard and when they heard them.

2. Furthermore, the authors performed decoding using either random selections of ROIs, the whole brain, or BPC, which includes language-related ROIs. However, prior studies have focused on specific ROIs, such as the language, prefrontal, and auditory association cortices. Therefore, it is unclear how the proposed method compares with prior methods. Since the authors' main research question revolves around how semantic information is embedded in brain signals to improve decoding, they should consider these ROIs, as they maintain a hierarchy of language processing.
	- The random selection of ROIs generally leads to low decoding performance. What are these random ROIs? Do they have any overlap with BPC ROIs?
	- Previous studies have conducted both quantitative and qualitative analyses, reporting what the stimulus decoded at each ROI, including language-related regions in both the left and right hemispheres, as well as using four evaluation metrics. However, this paper does not report any reconstructed stimulus in the main content, nor does it include analysis at the ROI level. Additionally, the authors only used two metrics, and throughout the paper, the focus is more on the scores rather than on the main reconstructed language results.

3. Although the authors report some results on predictive length and distance from the current word in Figure 1, there are no qualitative reconstruction results for these different predictive lengths and distances. What type of information is the model forecasting based on brain data? Is it syntactic information, such as nouns and verbs, or semantic content? This analysis is clearly missing from the paper.

4. All the figures lack detailed captions. The results presented in the figures are difficult to understand. For instance, what is the prediction score in each subplot of Figure 1? What does each line in the top plots represent? What does prediction distance "d" refer to? Without providing clear details in the figure captions or placing the figures appropriately in the text, it becomes challenging for readers to understand the content and what is being conveyed.

5. Since the authors use two encoders and two decoders in the proposed PREDFT, it is unclear which component is primarily responsible for reconstructing the language and which component provides the theme and narrative structure. It would be interesting if the authors reported the generated stimulus from individual components and from PREDFT as a whole, along with the performance metrics. This would help identify the shared and individual contributions of each component during language reconstruction.
@inproceedings{
anonymous2024language,
title={Language Reconstruction with Brain Predictive Coding from f{MRI} Data},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=2hKDQ20zDa},
note={under review}
}
Many recent studies have shown that the perception of speech can be decoded from brain signals and subsequently reconstructed as continuous language. However, there is a lack of neurological basis for how the semantic information embedded within brain signals can be used more effectively to guide language reconstruction. Predictive coding theory suggests the human brain naturally engages in continuously predicting future words that span multiple timescales. This implies that the decoding of brain signals could potentially be associated with a predictable future. To explore the predictive coding theory within the context of language reconstruction, this paper proposes PredFT (FMRI-to-Text decoding with Predictive coding). PredFT consists of a main decoding network and a side network. The side network obtains brain predictive coding representation from related brain regions of interest (ROIs) with a self-attention module. This representation is then fused into the main decoding network for continuous language decoding. Experiments are conducted on two popular naturalistic language comprehension fMRI datasets. Results show that PredFT achieves current state-of-the-art decoding performance on several evaluation metrics. Additional observations on the selection of ROIs, along with the length and distance parameters in predictive coding further guide the adoption of predictive coding theory for language reconstruction.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
fMRI-to-text decoding predictive coding theory
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/b4151d73d5768790c6f7af1d68357d03c85ba298.pdf
applications to neuroscience & cognitive science
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Language Reconstruction with Brain Predictive Coding from fMRI Data
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission