Reading Your Heart: Learning ECG Words and Sentences via Pre-training ECG Language Model
Electrocardiogram;ECG;Cardiac signal;Self-supervised learning;ECG language processing
Yes
No ethics review needed.
It is mentioned that the QRS-Tokenizer is used to segment the raw ECG signals into ECG sentences but as per my understanding the QRS tokenizer should be used to segment into each ECG beats aka words?
The paper is well-written with good clarity and coherence making the paper easy to read and comprehend. Most work on ECG either focus on beats or on rhythms making it challenging to learn both intra-beat and inter-beat features together which is where this contribution shines. The contribution is significant and interest with the concept of beats as words and rhythms as sentences which would allow learning more comprehensive latent features about the beats itself as well as the relation between the beats similar to human language. The masked training approach although not novel is a novel application with the concept of beats as words and rhythms as sentences and achieved significantly improved performance on popular datasets.
The paper introduces HeartLang, a self-supervised learning framework for ECG language processing that conceptualizes heartbeats as "words" and rhythms as "sentences" to better capture ECG signal semantics. The framework includes the QRS-Tokenizer, which segments ECG signals into meaningful units, and, a transformer-based model for learning spatiotemporal and semantic representations. The author's proposed approach, which uses heartbeat reconstruction and masked sentence pre-training and conducts experiments on six ECG datasets, outperforming other models in tasks such as rhythm and form classification. This work positions ECG data processing within a language-like framework, aiming to enrich model interpretability and generalizability for clinical applications.
It is mentioned that the QRS-Tokenizer is used to segment the raw ECG signals into ECG sentences but as per my understanding the QRS tokenizer should be used to segment into each ECG beats aka words which is confusing. The application of HeartLang at this point is limited and would require investigation on downstream tasks along with comparative studies.
Yes
No ethics review needed.
The approach of this research, which treats each heartbeat as a word and the rhythm of the time series as a sentence, is a very good idea for bringing the success of large-scale language models in natural language to electrocardiograms, and as described in this paper, it has also been successfully implemented. Continuing from the above weakness section, it would be even better if there was a clear discussion of the limitations in the main text.
This study on the HeartLang self-supervised learning framework has several strengths that make it a valuable contribution to the field of ECG analysis:
- Innovative ECG Language Perspective: The approach of treating ECG signals as a language, where heartbeats are "words" and rhythms are "sentences," is novel. This shift from conventional time-series analysis captures the unique form and rhythm characteristics of ECG signals, which can be critical for diagnosing cardiac conditions.
- Introduction of the QRS-Tokenizer: This component is specially designed to segment ECG signals into semantically meaningful units, allowing the model to focus on clinically relevant patterns in the data. The QRS-Tokenizer aligns the structure of the data with natural language processing (NLP) techniques, enhancing the depth and utility of the learned representations.
- Transformer-Based Architecture with ST-ECGFormer: The framework’s backbone, ST-ECGFormer, is a transformer-based model specifically designed to capture spatio-temporal features in ECG data. This model helps HeartLang learn complex temporal and spatial dependencies in the signals, enhancing the quality of the extracted features for further tasks.
- Flexibility in Downstream Applications: HeartLang’s pre-trained representations can be adapted for various downstream tasks, such as classification, providing a flexible tool for researchers and clinicians to build specific diagnostic models based on general ECG representations.

These strengths make HeartLang a promising advancement for ECG data analysis, particularly in enhancing the interpretability and applicability of ECG-based machine learning models in clinical diagnostics.
This paper introduces HeartLang, a self-supervised learning framework designed for analyzing electrocardiogram (ECG) signals by treating them like a language, with heartbeats as "words" and rhythms as "sentences." This approach diverges from conventional ECG analysis by using a language model perspective that preserves the form and rhythm characteristics of ECG signals, enhancing representation learning. Key contributions include:

1. QRS-Tokenizer: This tool segments ECG signals into meaningful "sentences" by identifying individual heartbeats (treated as words).
2. ST-ECGFormer: A spatio-temporal network designed to capture and process the temporal and spatial features of ECG data.
3. ECG Vocabulary: The largest heartbeat-based vocabulary to date, with 5,394 distinct heartbeat types for diverse cardiac conditions.
4. Masked ECG Sentence Pre-Training: A pre-training method that learns rhythm-level representations by masking parts of ECG "sentences."

HeartLang was tested on three public datasets, showing improved performance over other ECG self-supervised learning methods in areas such as general representation learning and downstream tasks, such as classification. This approach opens up a language-based approach for ECG research.
This approach would be effective in cases where the QRS waveform is recorded on an electrocardiogram, i.e. in cases of ST abnormalities due to ischemic heart disease or bundle branch block. However, many heart diseases cause significant irregularities in the heart rhythm itself, and these conditions are generally more directly linked to life. It is difficult to think of the HeartLang approach as being suitable for these conditions, and it is thought that it will be necessary to supplement it with other approaches in order to cover the whole of electrocardiography or heart disease.
The scope of this study may be narrow than the general interest of ICLR main conference.
Yes
No ethics review needed.
- To what extent does the current manuscript differ to the manuscript accepted at KDD-AIDSH 2024? What are the novel findings?
- Original idea to treat ECG signal as language in form of sentences and words
- Straightforward approach to tokenize the ECG signal into words and sentences (QRS-Tokenizer)
- Large ECG vocabulary - also useful and interpretable from a clinical perspective
- Systematic comparison of fixed-size and fixed-step time windows vs. language approach
- Benchmarking of results (using standard settings of MERL)
- Informative ablation study regarding spatial / temporal embeddings, pretraining and vocabulary set
This work addresses limitations in deep learning for ECG analysis (usually based on fixed-size and fixed-step time windows) by proposing a novel approach that treats ECG signals like language, where heartbeats are "words" and rhythms are "sentences". The authors introduce the QRS-Tokenizer to segment ECG signals into meaningful "sentences" and present HeartLang, a self-supervised framework that learns ECG representations at the form and rhythm levels. A comprehensive heartbeat-based ECG vocabulary is provided. This approach shows strong performance across 6 ECG data sets.
- A similar version of this manuscript has already been accepted: https://openreview.net/pdf?id=sdczcOS3n9 (at KDD-AIDSH 2024)
Yes
No ethics review needed.
Overall, the paper could be a moderate ECG representation learning contribution, with some practical and experimental issues requiring clarification. Given these clarifications in an author response, I would be willing to increase the score.

The ECG vocabulary was learned which shows similar semantic information, however, its consistency, as well as its randomness needs to be rigorously explored. It's necessity seems questionable due to the fact that normal heartbeats from a single recording may have turned into separate ECG words in the vocabulary which does not make much sense. Although the methodological process can generate it, its utility needs to be justified first. This thus undermines one of the claimed novelty.  An analysis of vocabulary stability across different recordings or subjects seems necessary.

12-lead ECGs were considered as data where some leads can be directly deduced from lead II and III including I, aVR, aVL, and aVF - such channel redundancies may have impact on ECG vocabulary generation. The framework does not show generalisability for less number of channels or even a single channel ECG where there is no spatial items. Experiments with reduced lead sets or single-channel ECGs would be beneficial that would demonstrate the framework's generalizability.

Data split lacks clarity if the train, test and validation splits combine segments across recordings or subject wise splits were considered. A clarification of data splitting strategy in the methodology seems necessary. This is because, if subject wise splits were not considered, then there will be data leakage.

The proposed framework is a variant of EEG based framework, however, the contribution is apparently for ECG modality and the data slicing philosophy. Authors should clarify the claim of framework based novelty accordingly.
A self-supervised framework was proposed for ECG representation learning considering an ECG signals as a sequence of beats, thus slicing them to include each of them in order to learn their form and a sequence of them to learn rhythms. A ECG vocabulary was created  which shows similar semantic information in terms of heartbeat representation.
This paper proposes a self-supervised learning framework for ECG language processing considering ECG beats as words and sequence of of beats as sentences mimicking learning ECG representation at form and rhythm levels. Instead of segmenting ECG signal into fixed-size and fixed-step windows view, as done by other contemporary studies, in this study, ECG beats (area consisting of PQSRT waves) are isolated by locating R-peaks and area around it and repeating it to retrieve sequence of beats for analysis. Transformer-based backbone architecture was used as encoder named as ST-ECGFormer to input ECG sentences which consists of ECG word (token) embedding and spatio-temporal and position embedding. Using vector quantisation, ECG vocabulary was auto generated in order to identify and cross-match similar ECG beats (words). A reconstruction training was adopted by masking parts of ECG words and using encoder/decoder architecture to predict the collective ECG word indices of the masked parts based on the unmasked individual ECG words. Authors have done experimentation on 3 publicly available ECG datasets and the proposed method achieved superior performance on heartbeat class and rhythm class classification downstream tasks after pre-training followed by fine-tuning which shows superior performance in several cases. ECG vocabulary was created and visualisation shows ECG words with the same index exhibit similar semantic information in terms of heartbeat representation.
The study lacks novelty in the framework itself since it is based on an existing one published in past ICLR venue, titled "LARGE BRAIN MODEL FOR LEARNING GENERIC REPRESENTATIONS WITH TREMENDOUS EEG DATA IN  BCI". However, the novelty lies in considering another modality of data, ECG. The way ECG beats are curated to formulate the problem in this study is interesting. 

- The performance difference compared to other self-supervised methods in the literature were discussed, however, it is unclear if the difference is due to the proposed framework or slicing ECG precisely to consider each beat in forming a sequence of beats, or just for considering ECG segments fixed-size and fixed-step. Applying the beat slicing mechanism used in this study to another study where fixed-size and fixed-step windowing were used would help.

- The generated ECG vocabulary is interesting to see, however, its motivation is unclear considering the fact that normal beats of a single recording may yield to show variance in the vocabulary without any clinical significance. It is not shown in this study how diverse these beats are for each category such as how many words the model comes up with for normal beats of a single recording. A domain specific discussion seems necessary to understand the utility of ECG vocabulary.
We propose HeartLang, a novel ECG self-supervised learning framework that treats heartbeats as words and rhythms as sentences, building the largest ECG vocabulary for ECG language processing to date.
@inproceedings{
anonymous2024reading,
title={Reading Your Heart: Learning {ECG} Words and Sentences via Pre-training {ECG} Language Model},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=6Hz1Ko087B},
note={under review}
}
Electrocardiogram (ECG) is essential for the clinical diagnosis of arrhythmias and other heart diseases, but deep learning methods based on ECG often face limitations due to the need for high-quality annotations. Although previous ECG self-supervised learning (eSSL) methods have made significant progress in representation learning from unannotated ECG data, they typically treat ECG signals as ordinary time-series data, segmenting the signals using fixed-size and fixed-step time windows, which often ignore the form and rhythm characteristics and latent semantic relationships in ECG signals. In this work, we introduce a novel perspective on ECG signals, treating heartbeats as words and rhythms as sentences. Based on this perspective, we first designed the QRS-Tokenizer, which segments ECG signals into semantically meaningful ECG sentences. Building on these, we then propose HeartLang, a novel self-supervised learning framework for ECG language processing, learning general representations at form and rhythm levels. Additionally, we construct the largest heartbeat-based ECG vocabulary to date, which will further advance the development of ECG language processing. We evaluated HeartLang across six public ECG datasets, where it demonstrated robust competitiveness against other eSSL methods. Our data and code will be publicly available after acceptance.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Electrocardiogram ECG Cardiac signal Self-supervised learning ECG language processing
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/c862120b71b390f6b68506ff685088f627c77cc4.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reading Your Heart: Learning ECG Words and Sentences via Pre-training ECG Language Model
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission