CLIBD: Bridging Vision and Genomics for Biodiversity Monitoring at Scale
Multimodal Learning;Contrastive Learning;DNA Barcodes;Taxonomic Classification;Fine-grained Classification;Biodiversity Monitoring
Yes
No ethics review needed.
1) In Tab. 1, what is the point of comparing non-aligned embeddings? It seems intuitive that there should be no correspondence in the learned representations.

2) The attention visualization is not discussed in detail. Also, as mentioned above, what information comes from checking classification before alignment? Wouldn’t correct predictions be random in that case?
1) The paper is well-written and and organized. The proposed approach and the experiments are clearly described.

2) The proposed approach is effective in tackling the problem at hand, while being simpler than common alternatives in the field.
The authors present a paper on a CLIP-based approach combining images, DNA barcodes and textual taxonomic description for biological classification (specifically of insects) in an open-set setting. Experimental results show that the method is effective and outperforms dual-modality contrastive learning approaches, as well as other approaches from the literature on this specific task.
1) The paper presents no methodological novelty, and mostly applies existing techniques in a standard way to a particular use case.
Yes
No ethics review needed.
Some suggestions and questions,

1.	Comparison with BioCLIP at different taxa levels: Going by the works claim, incorporation of DNA embeddings could help with classifying unseen species up to species level, but I’m guessing at higher taxonomic levels the BioCLIP performance should be comparable to CLIBD. It would be interesting to see a comparison.
2.	From Table 1, Image-to-DNA performance on seen species reduces going from (I+D) to (I+D+T), why is this happening? I would have expected the performance to improve.
3.	I’m also curios to know why BIOSCAN-5M was not used, considering that foundational models such as these can greatly benefit from more data.
4.	I personally felt Figure 7 was more informative to understand the data partitioning than Figure 2 (Just to consider in the future revisions).
1.	Incorporation of DNA as a modality to align the image embedding against instead of text is well motivated.
2.	Extensive experiments and ablations are provided.
The approach proposes multimodal contrastive learning between image, DNA and text (taxonomic labels) as opposed to Image+TaxonomicLabel only approach followed by previous work (BioCLIP). The usage of DNA is claimed to be better because, (1) classifying unseen species would be difficult with a taxonomic-label-only model because the species name would not have been seen during training, (2) DNA is easier to obtain compared to taxonomic label which requires careful examination by human experts.
1.	The accuracy when doing image to DNA on unseen species is not quite significant although it is better than BioCLIP’s approach of doing image to text. This indicates the image encoder is still not strong enough to generate a good DNA aligned embedding just from the image. Perhaps this can improve with more data.
Yes
No ethics review needed.
Please see weaknesses for primary questions. 

Minor comments / questions (no need to respond):
* This work focuses on the cases where aligned data is available: images and DNA barcodes from the same individuals. It might be useful to extend the method to also take advantage of abundant unpaired data: images and DNA barcodes that are not paired. 
* Figure 2 could be clearer. Why are there different shades of blue and orange? Why don't the numbers in the boxes sum up to 36729? Generally, this figure did not aid my understanding (though many of my confusions were clarified in later text and figures). 
* It would be nice to have a chance-level baseline in Table 1. 
* Down the road, it might be interesting to try to integrate additional modalities, e.g. geospatial location (https://arxiv.org/abs/1906.05272).
* The idea of jointly embedding DNA barcodes with images and taxonomic information is interesting.
* The experiments in the paper are extensive - there is a lot of technical content, and it's clear that a lot of effort went in to this work. There are many quantitative results, in addition to interesting qualitative results (e.g. Fig. 3, Fig. 5).
* The paper is very well-written. 
* The hyperparameters and training procedures are clearly spelled out.
This paper proposes a CLIP-based method to jointly embed images, DNA barcodes, and taxonomic strings for different insect species. The method is evaluated on cross-modal matching and species classification, with qualitative and quantitative results provided. The paper compares against one external method (BIOCLIP) in one set of experiments and one external method (BZSL) in a second set of experiments. The key claim of the paper is that jointly learning from the modalities of images, DNA, and taxonomic information leads to stronger representations for downstream use.
My major issue with the paper is missing baselines:
* The paper compares their multimodal representation learning approach against the unimodal pretrained models they start with. They show that their method is better, and conclude that multimodality is important. However, this claim is not justified - couldn't the benefit be from the additional training each modality received? It seems to me that the fair comparison would be to take the unimodal models and run unimodal CLIP-style training (with a similar computation / # steps budget) for each. If the multimodal model beats these CLIP-fine-tuned unimodal models, then that provides stronger evidence of the benefit of multimodal learning. 
* For the image encoder, wouldn't a model pretrained on BIOSCAN-1M be a more appropriate starting point / baseline than an ImageNet-pretrained model? 

The paper should address a few items that were not discussed:
* Are there confounders we need to worry about in this data, e.g. the facility the data was collected by? 
* The DNA to DNA matching results are very high, but shouldn't we expect this? Would simple homology methods based on string matching do a very good job at this task? What is the benefit of using deep embeddings to solve this problem? 
* Doesn't adding a modality increase the number of steps of training the model receives? Couldn't this be partly responsible for the differences between 1, 2, and 3 modalities in Table 1? 
* Image-DNA and Image-text matching don't seem very good based on Fig. 4, with the average being pulled up by a few good cases. What are those cases, and why are they different? Is there any insight to be gained there? 

A few claims made in the paper were not clear to me:
* The paper claims that "BIOCLIP... requires taxonomic labels to be available in order to obtain text descriptions. These labels can be expensive and time-consuming to obtain" and that "DNA barcodes can be obtained at scale more readily than taxonomic labels". These claims are not intuitive to me. How much does DNA barcoding cost per individual, compared to the cost of having an expert inspect an image to identify the species? 
* The training data for BarcodeBERT is claimed to be "different from, but highly similar to" the data used in this paper  - can you expand on what this means, and the implications for the results presented?
Yes
No ethics review needed.
Details about the reference database is missing. How many images and DNA barcodes are present in the reference database during inference?
- The motivation and problem formulation is sound and interesting.
- The proposed model fuses images, text and DNA into a contrastive embedding space enabling zero-shot image classification of unseen species.
- The flow of paper and writing is good in general.
The paper proposes CLIPBD, a tri-modal embedding space consisting of image, text and DNA of insect specimens. CLIPBD is trained using a three-way contrastive learning objective between image, text and DNA on the BIOSCAN-1M dataset. Using a reference database of images and DNA, CLIPBD is able to classify images of seen and unseen species. Results reported in the paper show superior performance as compared to BioCLIP and other state-of-the-art DNA encoder models.
- The abstract claims the paper is the first to use contrastive learning to fuse DNA and image. However, there are existing works [1, 2, 3] which have done this for other applications and they should be discussed in the related works.
- The claim that *DNA is a better target than taxonomic labels* (**Line 316**) is highly questionable. This claim is reiterated in **Lines 359-362**. The paper clearly mentions that majority of the BIOSCAN-1M dataset does not have taxonomic labels. In fact only 3.36% pretraining data has labels upto the species level (**Line 321**). It is clearly seen from **Table-1** that aligning with taxonomic labels outperforms aligning with DNA at the order level. I believe if the authors used an **unbiased dataset** containing the same proportions of DNA labels and taxonomic labels, the results would have been similar if not worse. This is more of a problem of the dataset and not the modality itself. 
- Following the previous point, why does Image-to-DNA retrieval performance improve at the species level when aligning all three modalities (**Table 1**)?
- For inference to work on unseen species during training, the framework assumes that their DNA and/or images are available in the lookup database. This is an unrealistic assumption. If the images and DNA are already available for unseen species, they might as well should have been used for training.
- Limited technical novelty considering no new representation learning technique has been proposed. The paper uses an existing dataset containing 1M insect specimens and unbalanced DNA and taxonomic labels, raising questions on the effectiveness of the method on other real-world datasets. Majority of the experiments and evaluations are only shown for a single dataset.
- The authors correctly pointed that BioCLIP was trained on diverse set of species including natural images. However, one easy way to utilize the BioCLIP embedding space would be to align the DNA modality with frozen BioCLIP vision and text encoders. Have the authors compared their method with this ImageBind-style training?


References

[1] Taleb, Aiham, et al. "Contig: Self-supervised multimodal contrastive learning for medical imaging with genetics." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[2] Xie, Ronald, et al. "Spatially Resolved Gene Expression Prediction from Histology Images via Bi-modal Contrastive Learning." Advances in Neural Information Processing Systems 36 (2023).

[3] Min, Wenwen, et al. "Multimodal contrastive learning for spatial gene expression prediction using histology images." arXiv preprint arXiv:2407.08216 (2024).
CLIBD uses a multimodal machine learning approach with images, DNA, and text to enhance insect classification, outperforming traditional methods in identifying known and unknown species without specific fine-tuning.
@inproceedings{
anonymous2024clibd,
title={{CLIBD}: Bridging Vision and Genomics for Biodiversity Monitoring at Scale},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=d5HUnyByAI},
note={under review}
}
Measuring biodiversity is crucial for understanding ecosystem health. While prior works have developed machine learning models for taxonomic classification of photographic images and DNA separately, in this work, we introduce a multi-modal approach combining both, using CLIP-style contrastive learning to align images, barcode DNA, and text-based representations of taxonomic labels in a unified embedding space. This allows for accurate classification of both known and unknown insect species without task-specific fine-tuning, leveraging contrastive learning for the first time to fuse DNA and image data. Our method surpasses previous single-modality approaches in accuracy by over 8% on zero-shot learning tasks, showcasing its effectiveness in biodiversity studies.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Multimodal Learning Contrastive Learning DNA Barcodes Taxonomic Classification Fine-grained Classification Biodiversity Monitoring
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/fa63c75bf63d80c45fecebd2c1d3bf35570383fd.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
CLIBD: Bridging Vision and Genomics for Biodiversity Monitoring at Scale
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission