Hyperbolic Genome Embeddings
genomics;representation learning;hyperbolic geometry
Yes
No ethics review needed.
- Is the CNN/HCNN model pretrained with a MLM task similar to DNABERT models? If so, on what dataset? If not, why?
Hyperbolic architectures is a promising avenue for training models on genomic sequences, as they are inherently related through phylogenetic tree structures. This paper provides interesting initial results in this direction.
The authors introduce a hyperbolic CNN for genomic sequences. They compare their model to a standard CNN on both synthetic and real benchmarks for genomic sequence classification (GUE, GE, and their transposable element benchmark). The authors find that the HCNN model outperforms the standard CNN on most tasks.
- How does the CNN/HCNN performance scale with number of parameters? Figure 5 seems to indicate that performance gets worse with increasing hidden dim? Additionally figure 5 should include error bars. Also the cumulative improvement y-axis is unclear. It should just be the average MCC value. 

- Figure 1 shows that sequences which obey a phylogenetic tree structure are embedded into a hyperbolic space which learns this structure. However I do not see any results which indicate that the phylogenetic structure is being learned. The authors could provide an experiment to validate this by looking at correlation between embedding distances and phylogenetic distances (for example EDS task from https://www.biorxiv.org/content/10.1101/2024.07.10.602933v1)

- The results would be stronger if the authors provided a baseline column for a standard transformer architecture in Table 1, as CNNs are used less frequently for genomic tasks. Additionally, the DNABERT/NT performance should be included in Table 1, not in supplementary Table 4, and would be much more clear by averaging a single value for each task. 

- Figure 2 shows the decision boundary for a 2D model, and shows better separation for the hyperbolic embeddings. However, it is not clear that this result extends to higher dimensions. Instead, the figure would be stronger by using a PCA/UMAP plot with a higher dimensionality model.
Yes
No ethics review needed.
1. Can you compare the cost of different models by the wall-clock time of fitting it on the same dataset with the same devices?
1. The paper is well-written, with good backgrounds and clear visualization.
2. The introduced Transposable Elements Benchmark is valuable to the research community.
3. The hyperbolic CNN demonstrates consistent improvements over the Euclidean CNN. It could serve as a baseline choice for genome classification models.
This paper adopts the hyperbolic CNNs to genome representation learning. Empirical results suggest the advantage of hyperbolic CNN to its Euclidean equivalents in genome learning. Moreover, the hyperbolic CNNs for genome sequences can outperform existing genome foundation models on 7 out of 27 datasets on the GUE benchmark.
1. Comparing the number of parameters between CNN-based models and Transformer-based models is not convincing. Due to the underlying difference between these two model architectures, fewer parameters do not guarantee efficiency. A more suitable comparison would be the time and memory usage of training the models on the same dataset.
2. There is a lack of baselines. As a CNN-based model, the model should also be compared with HyenaDNA and Caduceus.
3. The empirical study is not well-presented. In the main text, this paper dedicates most of the experimental sections to the comparison of hyperbolic and Euclidean CNN, which is less relevant to the genomics audience. On the one hand, this conclusion has already been drawn in the HCNN paper [1]. On the other hand, the Euclidean CNN is not the SOTA in genome classification. Compared with existing genomics foundation models, as shown in Table 4, it is more important, in my opinion.
4. The contribution is limited from both methodology and application perspectives.
    - Methodologically, HCNN is introduced in ICLR 2024, and the advantage of hyperbolic over Euclidean equivalents is also shown in that paper. This work adopts the same model (with tiny modifications on two versions ) and validates the same observations in genome embedding. 
    - From an application perspective, this work's contribution is limited. Though the authors do not report the wall-clock time / memory usage of the training the model, given that GFMs like NT and DNABERT only require 3-5 epochs to converge and this model is trained for 100 epochs, this model is likely to be more costly than existing ones. Moreover, it does not deliver better performances than GFMs. From an application perspective, users do not care much about whether a model is pre-trained / randomly initialized. The cost of fitting a model to a specific dataset is more meaningful.

[1] Bdeir, Ahmad, Kristian Schwethelm, and Niels Landwehr. "Fully Hyperbolic Convolutional Neural Networks for Computer Vision." arXiv preprint arXiv:2303.15919 (2023).
Yes
No ethics review needed.
- If the hierarchical relationship is between sequences then why do the authors note that a single point would be an individual nucleotide and a sequence would be a set? Isn't a sequence then exponential in the number of individual nucleotides thus resulting in an exponential growth of possibility space even without the hyperbolic space inductive bias?
- There is no mention regarding the parameter count of the eucledian CNN. I assume it is the same as the hyperbolic version but would be good to get a confirmation. Can the authors also include FLOP count per forward / backward pass for each of the models?
- The work can benefit from an expanded more thorough explanation in the appendix of mapping between tangent space and Lorentz manifold especially when describing the hypebolic layers. 
- Can the authors mention how the confidence intervals were generated in table 1. 
- Do the authors find it strange that for covid variant classification, arguably in a very clear cut case where there is abundant information regarding evolution of the virus the hyperbolic architecture doesn't perform so well? Do the authors have a hypothesis for why that is the case?
- The authors analyze an existing problem through a novel lens suggesting a new possible inductive bias to an important problem. 
- The authors perform a comprehensive evaluation across a wide variety of tasks
- Authors propose a new dataset which can evaluate ability of models to classify transposable elements. This class of sequences are widely abundant however it is unclear as to their biological significance compared to sequences encoding mature RNAs.
The authors propose an architecture that is better suited for genomic property prediction from DNA sequences. They implement hyperbolic convolutional neural networks (HCNN) and demonstrate improvements across a number of tasks. They compare against a matched CNN in Euclidean space. They motivate the hyperbolic embeddings due to the tree like underlying generative process of genomic sequences. In addition the authors propose an additional Transposable Elements Benchmark. In addition they use a measure of $\delta$ hyperbolicity or a measure of tree-likeness in genetic data.
- This work can benefit from additional  justification for why the hierarchical tree structure is a good inductive bias for genomic data. Although the original data generating process can take on a tree structure is it necessarily the case that we would want to add this as an inductive bias to the model?
- Is there an experiment that the authors can do that would confirm the validity of this inductive bias? Do the authors find it strange that the synthetic task that was explicitly designed to test performance of the inductive bias the baseline falls within the confidence interval of the best performing model? 
- In genomic sequence modeling it is common to do homology splitting where sequences related through their homologous relationships are fully left out and the model's capacity is assessed on whether it's able to generalize to unseen homology branches. Do the authors expect that this inductive bias helps or hurts the performance of the model in this domain? Can they perform an experiment validating this?
Yes
No ethics review needed.
- I would appreciate if the authors could address the above points. 

- Could selection of GUE as a benchmark cause underperformance? From my understanding, the benchmark does not control for homology in the train / test splits. Conceivably, the benefit of the HCNN would be improved generalization in evolutionary / phylogenetic contexts, and potential data leakage in the GUE benchmark could mask the performance benefit in this aspect. 

- I would appreciate if the colour of the DNABERT-2 distribution in supplemental figure 12 could be changed to be more distinct for folks with colour vision deficiency!
- Paper presents a creative application of HCNNs to the field of genomics, and serves as a useful preliminary paper to explore the benefits of representing genomic data in hyperbolic space. Given the current community interest in representation learning for genomics, this new approach could be of high interest and significance. Authors also provide new dataset for benchmarking a relevant task.

- The authors make an effort to construct high quality baselines for their experiments. In addition to demonstrating the benefits of an HCNN vs a Euclidean CNN, the method is also benchmarked against genomic foundation models. The authors also take care during their empirical evaluation of dataset hyperbolicity to account for dataset dimensionality by calibrating against simulated data.

- Experimental results indicate the hyperbolic representation offers benefit on some classes of tasks. This shows that the HCNN or other hyperbolic representations could have promise in genomic representation learning. Visually, it appears that the tasks where HCNNs offer improvement also correlate with the underlying data hyperbolicity, supporting the core hypothesis of the work.

- Experiments are well executed, with replication across random seeds and confidence intervals + statistical significance reported. Experimental setup is well described, and code is available, indicating that results are likely to be reproducible.
The paper proposes the application of fully hyperbolic CNNs (HCNNs) for representation of biological data, in specific DNA sequences. The authors note that genomic data is often generated by evolutionary processes, and result in phylogenetic hierarchies that may be better represented in hyperbolic space as opposed to Euclidean space. The authors apply HCNNs on several genomic element classification benchmarks, and additionally propose a new dataset, the Transposable Elements Benchmark, in which models are tasked with classifying the type of transposable element that exists in a context window. The authors find that hyperbolic CNNs outperform their Euclidean counterparts in most of the benchmark tasks, and can occasionally outperform genomic foundation model performance. They also perform an empirical analysis on the hyperbolicity of the selected datasets.
- I believe the analysis could be improved by investigating exactly how the HCNN represents sequences related through evolution. A common intuition I have seen for hyperbolic embeddings in computer vision is that datum closer to the origin tend to represent high level features and vice versa. It would be convincing if some analogy could be made for the genomic sequences through the experiments. For example, while Figure 2 demonstrates that hierarchical structure exists in the HCNN representation, it would be interesting to see whether the sequences placed at the "top" of the learned hierarchy reconcile with biological intuition.

- It's surprising that the HCNN representation is not consistently better for datasets that seem highly related to evolutionary processes. For example, it is strange that the covid variant classification is so poor given that the observed variation is likely driven by selection, and that the HCNN-M is outperformed by the Euclidean CNN on the synthetic dataset which is generated using a evolutionary simulator. Similarly, it is interesting that the HCNN does not significantly outperform the Euclidean models on promoter classification tasks despite the results in the empirical study in Figure 3. I would appreciate any deeper insights from the authors on these issues.

- Tying into the last point, given the lack of consistent improvement and correlation in improvement with seemingly evolutionary data generating processes, its unclear under what circumstance the HCNN should be used compared to existing approaches.

- To my knowledge, there is very limited methodological novelty in the paper, although I believe the current paper may be sufficient for an applied paper.
A hyperbolic geometry-based approach for genomic sequence representation learning
@inproceedings{
anonymous2024hyperbolic,
title={Hyperbolic Genome Embeddings},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=NkGDNM8LB0},
note={under review}
}
Current approaches to genomic sequence modeling often struggle to align the inductive biases of machine learning models with the evolutionarily-informed structure of biological systems. To this end, we formulate a novel application of hyperbolic CNNs that exploits this structure, enabling more expressive DNA sequence representations. Our strategy circumvents the need for explicit phylogenetic mapping while discerning key properties of sequences pertaining to core functional and regulatory behavior. Across 37 out of 43 genome interpretation benchmark datasets, our hyperbolic models outperform their Euclidean equivalents. Notably, our approach even surpasses state-of-the-art performance on six GUE benchmark datasets, consistently outperforming many DNA language models while using 13-379$\times$ fewer parameters and avoiding pretraining. Our results include a novel benchmark dataset - the Transposable Elements Benchmark - which explores a significant but understudied component of the genome with deep evolutionary significance. We further motivate our work by constructing an empirical method for interpreting the hyperbolicity of dataset embeddings. Throughout these assessments, we find persistent evidence highlighting the potential of our hyperbolic framework as a robust paradigm for genome representation learning.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
genomics representation learning hyperbolic geometry
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/0018422e16c3f8c18539f34a2dc679e2347ccdb2.pdf
applications to physical sciences (physics, chemistry, biology, etc.)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/dd89214d86984b16f2bcbe9c2bd24628f40fc544.zip
Hyperbolic Genome Embeddings
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission