NeuroLM: A Universal Multi-task Foundation Model for Bridging the Gap between Language and EEG Signals
EEG;large language model;multi-task learning
Yes
No ethics review needed.
1. It would be highly beneficial for the research community if you could share the code for NeuroLM as supplementary material. 

2. The expected benefits of large-scale pretraining are not fully realized on the TUAB dataset, with only minimal improvement observed. I was anticipating that the extensive pretraining would lead to more significant performance gains. Could you provide insights into why the pretraining did not result in better performance on TUAB?
1. The paper presents an innovative method that enables handling multiple diverse EEG tasks within a single unified model without the need for task-specific fine-tuning. the authors advance the capability to perform multi-task learning in EEG signal processing, overcoming limitations of traditional models that require separate fine-tuning for each task.

2. The authors conduct thorough experiments on six diverse downstream EEG tasks, including abnormal detection, event type classification, emotion recognition, sleep stage classification, cognitive workload detection, and slowing event classification. This extensive evaluation provides a robust comparison with existing state-of-the-art models, demonstrating the versatility and potential of NeuroLM across different EEG applications.

3. The paper includes detailed ablation studies examining the effectiveness of various components of the proposed model, such as the EEG-text embedding alignment, multi-channel autoregressive pre-training, and the robustness to instruction variations. These analyses validate the contributions of each component, offer insights into the model's performance, and enhance the understanding of how each part contributes to the overall effectiveness.

4. The authors provide clear and detailed descriptions of the model architecture, training procedures, data preprocessing steps, and experimental setups. This transparency facilitates reproducibility and allows other researchers to replicate and build upon the work more easily.
The paper presents NeuroLM, an innovative multi-task foundation model that bridges the gap between electroencephalogram (EEG) signals and language models. By converting EEG signals into discrete tokens aligned with text embeddings, NeuroLM enables Large Language Models (LLMs) to process EEG data effectively. The model employs a neural tokenizer trained through vector-quantized temporal-frequency prediction to encode EEG signals, and integrates these tokens into an LLM using multi-channel autoregressive pre-training. Through multi-task instruction tuning, NeuroLM adapts to various downstream EEG tasks without requiring task-specific fine-tuning. Evaluated on six diverse EEG datasets, NeuroLM demonstrates its potential to unify multiple EEG tasks within a single model.
Overall the paper has a good contribution and is well structured and clear. Here are some tips for improvements: 

1. Even though NeuroLM's performance is currently lower compared to LaBraM on most tasks, the authors could emphasize the benefits brought by the unified instruction tuning approach. Specifically, since NeuroLM is instruction-tuned, it may exhibit better generalization to unseen tasks or prompts, enabling zero-shot or few-shot learning without additional fine-tuning. This generalizability could make NeuroLM more adaptable to new EEG tasks compared to models like LaBraM that require task-specific fine-tuning, thus presenting a significant advantage of their approach.

2. The section on methodology especially the codebook and its origin from VQ-VAE,  and how it functions within NeuroLM, and the stop gradient operator justification in equation (3), could benefit from some explanation.

3. For the EEG-text alignment, while an appendix mentions visualization, a more detailed analysis quantifying the impact of EEG-text alignment on model performance would strengthen the claims.

4. Change the colormap for Figure 1, currently the color for NeuroLM and LaBraM are very similar and hard to distinguish.
Yes
No ethics review needed.
1. The paper mentions that NeuroLM performs well on several EEG tasks, but is there any evaluation of how well the model generalizes across different datasets?
2. How sensitive is the model to noisy data and how does it perform under varying degrees of EEG signal quality variation?
1. NeuroLM demonstrates for the first time the ability of a model to handle multiple EEG tasks simultaneously with instruction tuning.
2. NeuroLM is evaluated on six different downstream datasets, demonstrating its broad applicability and superior learning capabilities.
3. NeuroLM reduces the need for repeated training through instruction adjustment, making the model more efficient.
This paper introduces a multi-task foundation model called NeuroLM, which is integrated into large language models (LLMS) for electroencephalogram (EEG) signal processing. NeuroLM uses a text-aligned neural tokenizer to encode continuous EEG signals into discrete neural labels by learning through vector-quantized temporal-frequency prediction. These labels are subsequently used as input to the LLM through multi-channel autoregressive pre-training, which enables the model to learn a causal representation of EEG signals. Finally, through multi-task instruction tuning, NeuroLM is able to handle multiple downstream tasks, such as anomaly detection, event type classification, and emotion recognition, in a single model.
1. While NeuroLM performs well in multi-task learning, there are still gaps in its performance on some tasks compared to state-of-the-art models specifically designed and optimized for a single task (e.g., LaBraM).
2. Model performance fluctuates greatly across different tasks and datasets.
3. Although the model reduces the need for a large amount of labeled data through pre-training and multi-task learning, its performance and generalization ability still rely on high-quality training data.
Yes
No ethics review needed.
- How robust is NeuroLM across different LLM architectures?
- What role does adversarial training play compared to alternative strategies?
- Can you show some examples of fine-tuning that lead to SOTA performance?
- Innovative Idea : build the first EEG multi-task foundation model. 
- Trained on large-scale EEG dataset (> 25K hours), diverse data sources enhance model's generalization ability. 
- Consider EEG signals as foreign language and expand scope of LLMs including EEG tasks
- Instruction Tuning for Multi-Tasking: joint multi-task instruction handle various tasks in a unified manner.
Introduction of **NeuroLM**: a multi-task foundational model to process EEG signals. It leverages the capabilities of LLMs, employing a unified approach encoding the signals into discrete tokens. It addresses challenges like the usual need for individual task fine-tuning. 
Model is utilizing a text-aligned neural tokenizer trained using an adversarial strategy and multi-channel autoregressive pre-training to enhance the understanding of the signals. Experimental results demonstrate a competitive performance across 6 different tasks.
- Absence of Fine-Tuning Examples: The authors position NeuroLM as a foundation model and present solid performance across six downstream tasks. However, the study does not include an exploration of fine-tuning capabilities to demonstrate NeuroLM's potential for achieving SOTA performance which could underscore its utility in practical applications.

- Dependency on GPT-2: The authors only test GPT-2 so this study limits insights into the broader potential of this innovative approach. The model aims to bridge EEG and language so evaluating additional language models is necessary for a more comprehensive understanding of the model robustness and adaptability. 

- EEG-Text Alignment: The use of adversarial training is innovative. The absence of ablation studies limits understanding of its specific impact. It could enhance paper's clarity and support the results if it was shown how this approach contributes to the model's performance. 

- Complexity and Reproducibility: The pipeline's complexity may pose challenges for reproducibility. Enhanced clarity in certain sections would better support the community in replicating and extending this work.
Yes
No ethics review needed.
Concern 1:
The paper introduces an EEG-Text Embedding Space Alignment module in Figure 2, built on LaBraM’s neural tokenizer. However, I have concerns regarding the effectiveness and explanation of this alignment.

Fundamentally, EEG and Text have no inherent connection, especially since the EEG signals are not language-evoked. The use of a single gradient reverse layer may only distinguish EEG from text domains rather than fully align their embeddings. To clarify the module's effectiveness, could the authors:

1) Provide visualizations, such as t-SNE plots, comparing EEG and Text embedding spaces before and after alignment to illustrate the degree of alignment achieved.
2) Clarify the source of the Text embeddings and their generation method. The authors do not explain where these embeddings come from or how they are generated, leaving an important gap in understanding.
3) Include an ablation study to quantify the impact of the EEG-Text Embedding Space Alignment module on downstream task performance, showing its necessity and contribution to the overall model.

Concern 2:
The Theory Analysis section in 2.2 of Multi-channel Autoregressive Pre-training provides a high-level explanation by drawing an analogy to VAE, optimizing for reconstruction loss and KL divergence, but it lacks a detailed explanation of how causal relationships between EEG channels are learned or how this approach is theoretically grounded in the autoregressive framework.

To enhance the theoretical foundation of the multi-channel autoregressive approach, could the authors:

Include mathematical formulations demonstrating how this approach models dependencies between EEG channels and captures causal relationships.

Concern 3:
The paper lacks a detailed discussion regarding instruction design and how variations in prompt formulations affect downstream task performance. While instruction tuning addresses multiple tasks, the authors do not examine how different prompts might affect task performance. Could the authors provide an analysis of how different prompts influence task performance?

Concern 4:
The paper does not sufficiently discuss the impact of data curriculum or input order on the results. Although instruction tuning is introduced, there is no exploration of whether sequencing the data in a specific way (e.g., starting with simpler tasks or data) influences model performance across the various downstream tasks. It would be helpful if the authors could provide an analysis of how the ordering of data inputs affects model learning and task outcomes.
- The approach of treating EEG signals as a form of language and integrating them with LLMs is novel and creative, offering a new direction for EEG signal processing.

- NeuroLM shows potential in unifying different EEG tasks within a single model, reducing the need for task-specific fine-tuning and streamlining BCI applications.

- The experiments cover six diverse datasets, demonstrating the model's versatility and robustness in multi-task settings.
The paper introduces NeuroLM, a universal multi-task foundation model for EEG signal processing that consists of three key stages: a text-aligned neural tokenizer, multi-channel autoregressive pre-training, and multi-task instruction tuning. Traditional EEG pre-trained models often require full fine-tuning for each downstream task, resulting in resource inefficiencies. NeuroLM addresses this issue by leveraging Large Language Models (LLMs) and treating EEG signals as a foreign language, demonstrating the potential of this multi-task learning paradigm across multiple EEG-related tasks.
My concerns mainly focus on four areas, which are detailed in the Questions section. These include the effectiveness of EEG-Text Embedding Space Alignment, the theoretical foundation of multi-channel autoregressive pre-training, the lack of discussion on instruction design's impact on downstream tasks, and the absence of an analysis on data curriculum or input order. I hope the authors can address these concerns in the rebuttal stage.
NeuroLM integrates EEG signals and language into a single model, enabling efficient multi-task learning for various EEG applications.
@inproceedings{
anonymous2024neurolm,
title={Neuro{LM}: A Universal Multi-task Foundation Model for Bridging the Gap between Language and {EEG} Signals},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=Io9yFt7XH7},
note={under review}
}
Recent advancements for large-scale pre-training with neural signals such as electroencephalogram (EEG) have shown promising results, significantly boosting the development of brain-computer interfaces (BCIs) and healthcare. However, these pre-trained models often require full fine-tuning on each downstream task to achieve substantial improvements, limiting their versatility and usability, and leading to considerable resource wastage. To tackle these challenges, we propose NeuroLM, the first multi-task foundation model that leverages the capabilities of Large Language Models (LLMs) by regarding EEG signals as a foreign language, endowing the model with multi-task learning and inference capabilities. Our approach begins with learning a text-aligned neural tokenizer through vector-quantized temporal-frequency prediction, which encodes EEG signals into discrete neural tokens. These EEG tokens, generated by the frozen vector-quantized (VQ) encoder, are then fed into an LLM that learns causal EEG information via multi-channel autoregression. Consequently, NeuroLM can understand both EEG and language modalities. Finally, multi-task instruction tuning adapts NeuroLM to various downstream tasks. We are the first to demonstrate that, by specific incorporation with LLMs, NeuroLM unifies diverse EEG tasks within a single model through instruction tuning. The largest variant NeuroLM-XL has record-breaking 1.7B parameters for EEG signal processing, and is pre-trained on a large-scale corpus comprising approximately 25,000-hour EEG data. When evaluated on six diverse downstream datasets, NeuroLM showcases the huge potential of this multi-task learning paradigm. The codes and weights will be publicly available.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
EEG large language model multi-task learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/4325bf31b9030056289ce95b81b7afa47031fdd3.pdf
applications to neuroscience & cognitive science
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
NeuroLM: A Universal Multi-task Foundation Model for Bridging the Gap between Language and EEG Signals
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission