Genomic Foundationless Models: Pretraining Does Not Promise Performance
genomics;biology;deep learning;pretraining
Yes
No ethics review needed.
1. Could different pretraining tasks or loss functions enhance mutation sensitivity or biotype classification, specifically for detecting single-nucleotide variants?
2. The paper indicates certain tokenization changes improved performance. Would further model-specific optimizations, such as vocabulary adaptations or alternate pooling strategies, yield better results?
3. Given that pretrained models in other domains often benefit from more extensive data and compute, is there a threshold or scaling that could make pretraining beneficial for GFMs?
1. The paper proposes an interesting perspective in challenging the assumed superiority of pretrained models in genomics;
2. The experiments were thorough in comparing pretrained models with randomly initialized counterparts across various genomic tasks;
3. The writing is clear and easy to follow.
The paper questions the value of pretraining in genomic foundation models (GFMs), demonstrating that randomly initialized models can often match or even surpass pretrained GFMs on several tasks.
1. Although the results suggest current methods may not work, the paper could explore alternative pretraining objectives or paradigms that might better align with biological complexity.
2. There lacks an explanation or analysis on why specific models, such as Mistral or DNABERTv2, might perform better or worse on certain tasks, which would provide better insight.
Yes
No ethics review needed.
1. For NT, why use the 1000G instead of NTv2-multispecies? The v2 one is much better.
2. It's nice to compare the models in both finetuning and feature extraction. Why don't you use the same datasets for finetune/feature extraction? I would suggest the same feature extraction experiments on the same selected portion of the NT benchmark, like the histone datasets, to provide more convincing observations on the feature extraction of GFMs.
1. The paper is well-written and easy to follow.
2. The experiments are comprehensive and well-presented, except for a few ones (see Weaknesses). 
3. The mutation experiments are very interesting.
This work benchmarks the effectiveness of the Genomics Foundation Models (GFM) compared to randomly initialized models. Their experiments indicate that GFMs offer limited or even no advantages over randomly initialized models on many tasks.
1. Findings 1 is not well-supported
    - Figure 1 is missing leading. Given that different GFMs differ significantly in the number of parameters (e.g., 450k v.s. 500M), I don't understand what the authors aim to convey with this figure. A randomly initialized large model achieving better performance than a pre-trained small model does not suggest the pre-training is not helpful. 
    - A pairwise comparison of the pre-trained and random versions of the same model, as shown in Figure 2, is much more meaningful. In fact, as shown in Figure 2, the pre-trained model performs better than the random counterparts in most cases, which actually suggests the effectiveness of pre-training. The improvement of pre-training over randomized is more significant on more distinguishable datasets like histone (i.e., different models achieve distinct performances) and less significant on promoter/enhancer. I would attribute this insignificance mainly to the datasets (See below).

2. As shown in Figure 2, all the models ranging from 450k to 500M parameters achieve near-identical performance on many datasets of the NT benchmark, especially on promoter and enhancer. This indicates this benchmark may not be suitable for differentiating different models' capabilities. If the dataset could not differentiate different pre-trained models, it's hard to believe it can effectively differentiate pre-trained/random models. I would suggest more experiments on other benchmarks/datasets that can clearly differentiate the performance of different models. You may consider the datasets used in HyeneDNA (species classification), DNABERT-2 (GUE), and Caduceus (Genomics Benchmark).


Overall, I appreciate the authors' efforts in conducting this amount of experiments. Yet the results are not convincing, and the main conclusion (findings 1) is not well-supported. I will consider increasing my scores if more convincing results are presented.
Yes
No ethics review needed.
See Weaknesses.
1. The study includes a detailed analysis across several GFMs with random initialization methods, providing a broad view of the pretrained effect on genomic benchmarks. 
2. The observation that pretrained models often fail to capture clinically relevant mutations, particularly for tasks that require sensitivity to sequence variation.
3. The authors conducted extensive and convincing experiments on genomic datasets to validate their hypotheses.
The paper critically evaluates the assumption that pretraining offers significant advantages in Genomic Foundation Models (GFMs). Through extensive comparisons, the authors show that randomly initialized models can achieve performance comparable to or exceeding that of pretrained models across a variety of genomics tasks.
1. Lack of further analysis on model behavior. For instance, why do random models outperform pretrained models on some tasks? Why does Mistral perform better in mutation sensitivity analysis?
2. The experiment on ClinVar uses a limited sample size (only one gene), which may limit the generalizability of the conclusion that pretrained models lack sensitivity to mutations.
3. The single nucleotide sensitivity analysis lacks comparative experiments. There is no direct comparison between random and pretrained models in the mutation sensitivity analysis and the ClinVar experiments.
Yes
No ethics review needed.
- It's not that pre-training doesn't work. It's that gene sequences haven't found a more appropriate pre-training strategy.

- It doesn't make sense to me to disregard the EVO model, which has the largest pre-training scale, doesn't it?
- The exploration of pretraining in genomic models is an important area of research, especially given the growing interest in applying deep learning to genomics.
- Potential Impact: Understanding the limitations of pretraining in this context can provide valuable insights for future research.
This paper investigates the performance of genomic foundationless models, particularly focusing on the implications of pretraining. While the topic is pertinent, the paper presents several critical issues that undermine its contributions and clarity.
- The findings largely reiterate known limitations of pretraining without offering new insights or approaches. The paper does not present a compelling argument for how its contributions advance the field.

- The experimental validation lacks rigor. There is a failure to adequately compare the proposed models with established methods, making it difficult to evaluate the significance of the results. A large number of genetic tasks that have been established have not been mentioned and tested for relevance [1]. 

- The conclusions drawn from the experiments are overly broad and not sufficiently supported by the data presented. The authors claim that pretraining does not promise performance without adequately accounting for various contextual factors.

[1] GenBench: A Benchmarking Suite for Systematic Evaluation of Genomic Foundation Models
Current pretrained Genomic Foundation Models offer little to no advantage over randomly initialized models on many genomic tasks.
@inproceedings{
anonymous2024genomic,
title={Genomic Foundationless Models: Pretraining Does Not Promise Performance},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=kDZKEtDnT1},
note={under review}
}
The success of Large Language Models has enabled the development of Genomic Foundation Models (GFMs) through similar unsupervised pretraining techniques. To assess the effectiveness of pretraining in genomics, we conducted a comprehensive evaluation of seven different GFMs across various benchmarks, comparing them to their counterparts with randomly initialized weights.  Surprisingly, we found that randomly initialized models can match or even surpass the performance of pretrained GFMs in finetuning and feature extraction tasks. We also discovered that pretrained GFMs fail to capture clinically relevant genetic mutations, which are crucial for understanding genetic disorders and phenotypic traits. Our results indicate that most of the current pretrained GFMs lack a ``foundational'' understanding of genomics and offer minimal utility, if any, even for basic tasks such as sequence classification. These findings highlight the need for developing novel architectures and training methods tailored for genomic applications. The code is available through this anonymous link https://github.com/nxifemwt/GFMs.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
genomics biology deep learning pretraining
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/036a301b0d88417d738a92c909f59a00e4294ddd.pdf
applications to physical sciences (physics, chemistry, biology, etc.)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Genomic Foundationless Models: Pretraining Does Not Promise Performance
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission