Towards Understanding Domain Adapted Sentence Embeddings for Document Retrieval
Sentence Embeddings;Question Answering;Technical Domains;Retrieval Augmented Generation;Embedding Models;Isotropy
Yes
No ethics review needed.
1. This paper does not provide sufficient details on the ISOTROPY SCORES.
2. The paper does not sufficiently clarify the motivation behind its approach, especially regarding the "COMPUTATION OF THRESHOLDS". It lacks a detailed discussion on why existing methods struggle with this issue and how this paper effectively addresses it. it requires more illustrations and provements about this to help understanding the issue of existing methods and the motivation of the proposed new method in the paper.
1. The paper is generally understandable and clearly explains the technical parts to a certain extent.
2. The figures and charts in the manuscript are exceptionally clear and well-presented.
This paper reported mean bootstrapped retrieval accuracies along with confidence intervals for various SOTA embedding models with and without domain-adaptations. The results show that domain specific embedding show improved isotropy scores and move away from general domain embeddings. Overall, this is a solid study with reasonable technical contributions.
1. This paper does not provide sufficient details on the ISOTROPY SCORES.
2. The paper does not sufficiently clarify the motivation behind its approach, especially regarding the "COMPUTATION OF THRESHOLDS". It lacks a detailed discussion on why existing methods struggle with this issue and how this paper effectively addresses it. it requires more illustrations and provements about this to help understanding the issue of existing methods and the motivation of the proposed new method in the paper.
Yes
No ethics review needed.
1. How does the test or dev set accuracy compare to bootstrapped accuracy? could you compare it with existing QA model results as a relaxed version of exact answer retrieval?
1. The paper proposes novel metrics (CI, COE and ROE) over batched samples to evaluate embeddings for technical domains.
2. The methodology and metrics introduced are well-explained.
This paper investigates different aspects of sentence encoders for domain-specific question-answer retrieval. It aims to address the limitations of cosine similarity in capturing true semantic similarity, especially for frequent words with homonyms or dependence on regularization. To improve retrieval evaluation, the authors use confidence intervals (CIs) over bootstrapped samples for models before and after pre-training and fine-tuning on specific domains. They introduce COE and ROE metrics to measure distributional overlaps between top-K, correct, and random document similarities with the query, showing these metrics correlate with accuracy and threshold values. The paper visualizes how domain adaptation shifts embeddings away from domain-agnostic spaces. It also shows isotropy scores have limited correlation with retrieval performance.
1. The paper lacks specifics on the pre-training corpus for each domain. If models are pre-trained solely on training sets, this limits the utility of pre-training, affecting the reliability of conclusions. For instance, in the health domain, pre-training on a large corpus like PubMed abstracts is common to ensure domain knowledge.
2. Comparisons are primarily with pre-trained models from BAAI and OpenAI; adding more state-of-the-art domain-specific baselines could strengthen the evaluation.
3. While the work presents interesting findings, the novelty is limited. Observations like tighter CIs with fine-tuning are expected since task-specific fine-tuning generally increases confidence for a specific task while potentially reducing generalizability.
Yes
No ethics review needed.
The paper could benefit from a more cohesive narrative and clearer articulation of the benefits of the proposed assessment approach. While the common theme of domain adaptation links the various experiments, it is unclear what specific advantages the presented methodology offers.

To strengthen the paper, I would suggest addressing the following points:

Clearly define the end goal of the work. For example, if the aim is to provide guidance on setting appropriate retrieval thresholds, what is the current standard practice in this area? How does the proposed method improve upon existing approaches?

Explicitly highlight the key advantages of the new assessment technique. What performance gains or other benefits does it provide compared to baseline methods? 

Structure the paper to guide the reader seamlessly from the problem statement, to the methodological details, and finally to a clear conclusion about the merits and recommended use cases of the new assessment strategy.
The authors define a relevant problem for RAG applications and show that fine-tuning can help to not only improve the retrieval performance but also increase the confidence intervals tightens. Thus, supporting that domain-adaptation is likely to also reduce the expected performance variance in that domain.
The paper presents an empirical study of domain adapted embedding models for QA. The authors evaluate a range of existing models and their domain-adapted variants on point accuracy and confidence intervals of retrieval tasks. The main contributions of this study are using the following aspects to provide a recommendation to select retrieval thresholds for domain adapted models: 

1. A methodology for determining appropriate similarity score thresholds for different embedding models.
2. Introduction of new metrics that capture the distributional overlap between top-ranked, correct, and randomly selected document with respect to the input question.
3. Analysis of the relationship between embedding isotropy (uniformity of vector lengths) and retrieval accuracy.
4. Observations that domain-adapted embeddings exhibit little overlap with their domain-agnostic counterparts.
While the problem addressed is relevant and some of the findings hold intrinsic interest, the paper also presents several notable limitations that should be addressed.

First, the confidence interval (CI) estimation techniques employed are direct applications of common methodologies to measure CI for a given metric (in this case accuracy). 

Second, the paper does not provide a compelling justification for the proposed assessment to select the retrieval threshold, nor does it rigorously compare their performance to simpler approaches such as hyper-parameter cross-validation which is relatively inexpensive assuming the similarity matrix is computed only once. The added value of the new techniques is therefore unclear from the current presentation.

Third, while the paper cites prior studies on the concept of isotropy, there is insufficient explanation of how the experimental setup and findings in this work relate to or build upon those earlier investigations. The connection to the broader academic context is not well established.

Finally, the overall narrative lacks cohesion, as the sections are not clearly tied together, and the suggested methodology for assessing retrieval thresholds is not systematically contrasted against relevant baseline approaches. This makes it difficult for the reader to fully grasp the unique contributions of the work.
Yes
No ethics review needed.
- It is unclear why the CI is set to 95%. Is there any supporting evidence?
- The ndcg is widely used to evaluate information retrieval performance. Why not use it?
- This paper proposes a systematic method to introduce thresholds to improve document retrieval performance.
- Extensive experiments are conducted to demonstrate that fine-tuning and pretraining-then-fine-tuning help improve retrieval accuracy.
This paper investigates domain-adapted sentence embeddings for document retrieval. It proposes a systematic method to obtain thresholds for similarity scores for different embeddings, and proves that fine-tuning and pretraining-then-fine-tuning help improve retrieval accuracy through extensive experiments.
- Lack of novelty: Much existing work has shown that fine-tuning or continued pretraining helps to improve domain performance [1]. The isotropy problem has been well studied in [2][3] and many other contrastive learning-based sentence embeddings.
- There are too many research questions in this paper. It looks like the RQ2 and RQ3 don't have a strong connection to the research topic, i.e., domain-adapted sentence embeddings.
- Poor organization. The current introductory section functions as a literature review. It is suggested to clarify the research problem, research gap and contribution in the introduction.


**Reference:**

[1] Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020). Don't stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004.10964.

[2] Kawin Ethayarajh. 2019. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 55–65, Hong Kong, China. Association for Computational Linguistics.

[3] Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910. Association for Computational Linguistics.
The paper presents challenges in selecting embeddings for technical domains, metrics for domain adaptation and the impact of fine-tuning on accuracy and confidence intervals, and lack of correlation of isotropy of embeddings & retriever performance.
@inproceedings{
anonymous2024towards,
title={Towards Understanding Domain Adapted Sentence Embeddings for Document Retrieval},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=p7K3idvKTQ},
note={under review}
}
A plethora of sentence embedding models makes it challenging to choose one, especially for technical domains rich with specialized vocabulary. In this work, we domain adapt embeddings using telecom, health and science datasets for question answering. We evaluate embeddings obtained from publicly available models and their domain-adapted variants, on both point retrieval accuracies, as well as their (95\%) confidence intervals.  We establish a systematic method to obtain thresholds for similarity scores for different embeddings. As expected, we observe that fine-tuning improves mean bootstrapped accuracies. We also observe that it results in tighter confidence intervals, which further improve when pre-training is preceded by fine-tuning. We introduce metrics which measure the distributional overlaps of top-$K$, correct and random document similarities with the question. Further, we show that these metrics are correlated with retrieval accuracy and similarity thresholds. Recent literature shows conflicting effects of isotropy on retrieval accuracies. Our experiments establish that the isotropy of embeddings (as measured by two independent state-of-the-art isotropy metric definitions) is poorly correlated with retrieval performance. We show that embeddings for domain-specific sentences have little overlap with those for domain-agnostic ones, and fine-tuning moves them further apart. Based on our results, we provide recommendations for use of our methodology and metrics by researchers and practitioners.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Sentence Embeddings Question Answering Technical Domains Retrieval Augmented Generation Embedding Models Isotropy
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/6aae9e09b7c2b74c3a86bd9f1a4c56ba074e45c4.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Towards Understanding Domain Adapted Sentence Embeddings for Document Retrieval
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission