Towards Homogeneous Lexical Tone Decoding from Heterogeneous Intracranial Recordings
brain-computer interfaces; speech decoding; tonal language; Homogeneity-Heterogeneity Disentanglement
Yes
No ethics review needed.
Can you provide the specific list of 407 Mandarin syllables and explain how they represent nearly all Mandarin characters?

How and why are the private and shared codebooks defined and utilized within your framework?

Have you considered the computational cost implications of using separate encoders for each subject, and are there plans to optimize this aspect?

How does your work differentiate from the 2023 study "A high-performance brain-to-sentence decoder for logosyllabic language," which also uses a dataset with 407 syllables?

What is the total duration of the sEEG data in the dataset you provided?
The introduction of the Homogeneity-Heterogeneity Disentangled Learning for Neural Representations (H2DiLR) is a novel approach that effectively addresses the challenge of data heterogeneity in neural decoding. By disentangling shared and subject-specific neural features, the framework enhances the decoding accuracy across multiple subjects.

The study employs a robust experimental setup using stereoelectroencephalography (sEEG) recordings from multiple Mandarin-speaking participants.

The proposed method demonstrates a substantial improvement in tone decoding accuracy over traditional subject-specific models.
This paper introduces the Homogeneity-Heterogeneity Disentangled Learning for Neural Representations (H2DiLR), a framework designed to improve neural decoding by separating homogeneous and heterogeneous components from intracranial recordings across multiple subjects. The study demonstrates that H2DiLR significantly enhances tone decoding performance in Mandarin-speaking participants using stereoelectroencephalography (sEEG) data. The framework outperforms traditional methods by effectively capturing and leveraging both shared and subject-specific neural features.
The statement that a "comprehensive set of 407 Mandarin syllables covers nearly all Mandarin characters" requires further clarification. You need to provide the specific vocabulary list used in the rebuttal materials to support this claim.

The concept of private and shared codebooks is introduced in the paper but lacks a detailed explanation. A more comprehensive rationale and description are needed to clarify how and why these codebooks are defined and utilized.

Sharing the data and code would greatly enhance the validation and reproducibility of the proposed method by the research community. This transparency would contribute significantly to the field and support further advancements.

The claim of being "the first" to work with 407 Mandarin syllables is inaccurate, as the 2023 study titled "A high-performance brain-to-sentence decoder for logosyllabic language" also mentions using a dataset with 407 syllables. Although this work is cited, the related work section lacks a detailed discussion of this study and how the current work differentiates itself.

The paper asserts that it disentangles homogeneity and heterogeneity, yet the evidence provided relies on feature visualization of specific samples. A statistically robust analysis is necessary to substantiate this claim. 

While the framework allows for joint training across multiple subjects, the use of separate encoders for each subject may significantly increase computational costs, which should be addressed and optimized.
Yes
No ethics review needed.
See weaknesses. What areas of the brain are most important to tone decoding? Does the exclusion of data from some electrodes have more of an impact than others? Is there any benefit of this method for neural encoding models that predict neural responses from feature input?
The topic of tone decoding is understudied, if somewhat narrow, and the idea to separate homogeneous and heterogeneous neural representations is interesting and could have use cases outside the specialized setting of tone decoding, if applied to semantic neural representations more generally. Overall, I think the idea is interesting enough to be in the "borderline accept" category of papers, mainly because I think the idea is general enough to have practicable value outside this particular application.
The paper discusses a method called Homogeneity-Heterogeneity Disentangled Learning for neural Representations (H2DiLR) for decoding tone from stereoelectroencephalography (sEEG) data.  H2DiLR disentangles shared (homogeneous) and individual-specific (heterogeneous) neural representations using a two-stage learning paradigm. The first stage involves unsupervised learning of neural features via vector quantization with shared and private codebooks, while the second stage applies these representations to decode tones using a transformer model​. The authors claim that H2DiLR outperforms previous baseline methods by 12% when evaluated on Top-1 accuracy.
I feel that the authors limited themselves somewhat in specializing their approach to tone decoding only. I feel that the problem of disentangling subject specific effects from general ones is a wider problem with many downstream applications, most notably for semantic decoding, but also for problems of encoding as well. I would have liked to see a deeper exploration of this in the work. There is also disappointingly little exploration of the abundant neuroscience questions here - I would like to have seen a location-based ablation in addition to the parametric ablations of the method that showed which regions are most useful for tonal decoding. An analysis studying if there are region-specific differences to representational homogeneity (i.e. some regions tend to be more idiosyncratic to a particular subject) would also have been interesting, although I recognize that the chosen data setting, sEEG, has challenges in answering this question due to coverage differences.
Yes
Yes, Responsible research practice (e.g., human subjects, data release)
1. Table 1: What is the codebook size of Heterogeneous H2DiLR with ($\nu=0$)? Is it `K=32`?

2. Did you use a different signal length for pre-training and downstream classification? Are there any differences between the pre-training dataset and the downstream classification dataset?

3. Line 879: Did you apply normalization to each channel independently (i.e., calculating the mean and standard deviation for each channel separately) or across all channels together?
**Significance**: Open-source sEEG speech datasets are rare. Their promise of publishing the dataset upon acceptance (Line 780) is good news for the community as it will lower the entry threshold for future research. Additionally, they demonstrate how SSL-based pre-training allows improving performance compared to only supervised training.

**Clarity**: The text has a good structure and is well-written. The figures also help in understanding the method.
The authors introduce a neural network architecture, H2DiLR, a self-supervised learning (SSL) method for pre-training models for sEEG tone decoding (4-way classification task) and a novel dataset (407-syllable reading sEEG dataset).

**Dataset**: The patients are asked to read 407 syllables aloud. To make their pronunciation as close to natural speech as possible, carrier words are added to form full sentences. The dataset includes `1221` trials (i.e., 3 trials per syllable). Simultaneously recorded audio signals are then used to precisely identify time stamps for extracting the target syllable from each trial. Each trial is truncated based on these time stamps, resulting in a neural recording of up to `1` second, labeled with one of `4` tones. According to the task paradigm demonstrated in [1], the total recording time for each subject is approximately `5` hours, with each trial lasting around 12 seconds. Therefore, the total amount of pre-training dataset is about `20` hours.

**Model**: They introduce two different codebooks to disentangle the shared and private parts of multiple subjects. During the pre-training stage, they reconstruct the original sEEG signals. After pre-training, they use the frozen pre-trained VQ-Encoder to obtain quantized embeddings of the input sEEG signals. These embeddings are then passed through a temporal Transformer for further temporal integration, followed by a classification head for tone decoding.

**Experiments**: Previous pre-training methods (including BIOT, NeuroBERT, etc.) are compared. Besides, the authors conducted different ablation studies regarding pre-training (partition ratio, # of subjects, etc.).
**Major**
1. The authors don’t mention the introduction of data augmentation (e.g., temporal jittering, additive noise, etc.) to avoid overfitting, a common strategy used in sEEG-based speech decoding [2,3]. Since the number of available trials (i.e., $\sim$1221 trials) within each subject is too small, I’m not sure whether the improvement of pre-training comes from this.

2. The preprocess details are missing. Could you provide a detailed description of sEEG preprocessing (i.e., from the originally collected sEEG signals at a sample rate of 2kHz to the epoched trials at a sample rate of 1kHz)?

3. In [1], their CNN baseline achieves about 40% on average, and their NAR model achieves 45.75% on average. All these methods are supervised (heterogeneous) baselines, and these results make it hard to assess the contribution of H2DiLR.

4. No code and demo dataset is provided. Without the code and data, it's difficult to verify the claims.

5. Some aspects of the method are undefined or unclear. Please see the Questions section below. These aspects need to be clarified in the manuscript.

**Minor**
1. Figure 1: It could mislead readers into thinking that H2DiLR uses Product Quantization. In Figure 3 and Section 3.3, the authors actually quantize each item in the embedding sequence using one of different codebooks. However, in Figure 1, the “disentangled embedding” suggests that different parts of each individual embedding are quantized with different codebooks.

2. Line 155: typo “LaBraM Anonymous (2024)”

3. Line 841: typo “the learning of 5e-5” should be “the learning rate of 5e-5”? Is the only difference between fine-tuning pre-trained models and training non-pretrained baselines the learning rate? Are all other settings the same?

4. Use `\citet` or `\citep` instead of `\cite` when referencing other works.

5. Since you collected a 407-syllable reading sEEG dataset, how does H2DiLR perform on other types of syllable classification tasks, such as initial syllable classification? Including these additional results might further highlight H2DiLR's effectiveness.

**Reference**

[1] Feng C, Cao L, Wu D, et al. A high-performance brain-sentence communication designed for logosyllabic language[J]. bioRxiv, 2023: 2023.11. 05.562313.

[2] Metzger S L, Littlejohn K T, Silva A B, et al. A high-performance neuroprosthesis for speech decoding and avatar control[J]. Nature, 2023, 620(7976): 1037-1046.

[3] Willett F R, Kunz E M, Fan C, et al. A high-performance speech neuroprosthesis[J]. Nature, 2023, 620(7976): 1031-1036.
Yes
The dataset is related to human subjects, and has not released yet.
Yes, Responsible research practice (e.g., human subjects, data release)
1. The usage of non-invasive brain recordings is an emerging topic. What's the performance of H2DiLR in decoding EEG signal?
2. Why the size of codebook and dimension of codebook are set as the same?
1. The motivation of this paper is clear. The authors seek to address a very important problem in brain decoding: dig homogeneity within the brain responses of different subjects, and hope the obtained homogeneity representation can help improve decoding performance. The presentation of this paper is also good.
2. The proposed vq-autoencoding based method outperforms previous models in decoding performance. Additional ablation analysis further confirms the effectiveness of model design.
3. The authors build a new sEEG dataset (promise to release if this paper gets accepted) which has potential to boost researches related to lexical tone decoding.
In this paper, the authors propose H2DiLR to capture both homogeneous and heterogeneous information from intracranial recordings across multiple subjects for lexical tone decoding. The H2DiLR framework first obtains discrete vectors through a reconstruction task (vq-autoencoding manner), then builds shared codebook and private codebook for the decoding of different subjects. Experiments on a newly acquired, not publicly released sEEG dataset show the H2DiLR method outperforms other baseline methods.
1. Since lexical tone decoding is not a very common task in the brain decoding area, I think the task / problem setting needs to be first introduced and defined in the methodology part.
2. The paper lacks analysis of the generalization ability of captured common representation, i.e. if the shared codebook of three subjects well generalize to the fourth subject, and whether any three of the four subjects lead to similar shard codebook. The authors also mention this issue in limitations, but I think the dataset containing four subjects is enough to conduct such experiments.
@inproceedings{
anonymous2024towards,
title={Towards Homogeneous Lexical Tone Decoding from Heterogeneous Intracranial Recordings},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=cWEfRkYj46},
note={under review}
}
Recent advancements in brain-computer interfaces (BCIs) and deep learning have made decoding lexical tones from intracranial recordings possible, providing the potential to restore the communication ability of speech-impaired tonal language speakers. However, data heterogeneity induced by both physiological and instrumental factors poses a significant challenge for unified invasive brain tone decoding. Particularly, the existing heterogeneous decoding paradigm (training subject-specific models with individual data) suffers from the intrinsic limitation that fails to learn generalized neural representations and leverages data across subjects. To this end, we introduce Homogeneity-Heterogeneity Disentangled Learning for Neural Representations (H2DiLR), a framework that disentangles and learns the homogeneity and heterogeneity from intracranial recordings of multiple subjects. To verify the effectiveness of H2DiLR, we collected stereoelectroencephalography (sEEG) from multiple participants reading Mandarin materials containing 407 syllables (covering nearly all Mandarin characters). Extensive experiments demonstrate that H2DiLR, as a unified decoding paradigm, outperforms the naive heterogeneous decoding paradigm by a large margin. We also empirically show that H2DiLR indeed captures homogeneity and heterogeneity during neural representation learning.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
brain-computer interfaces; speech decoding; tonal language; Homogeneity-Heterogeneity Disentanglement
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/8acabb5cf9f46b7aafa9e28205deba3ee9ddda3b.pdf
applications to neuroscience & cognitive science
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Towards Homogeneous Lexical Tone Decoding from Heterogeneous Intracranial Recordings
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission