Contextual Document Embeddings
text;embeddings;retrieval;context;contrastive
Yes
No ethics review needed.
1. Can the authors provide more information about how the model is trained in the two-stage pass? Also, is the final model M1 or M2, or the both of them?
2. When evaluating on MTEB, how the in-context documents are selected for each task in the second-stage encoding?
1. The method is well-motivated, although it is unclear whether without the method, current state-of-the-art embedding models are not able to provide good nuanced representations. 
2. The authors interpret the training of dense embedding methods and the method itself from a statistical perspective which is convincing.
The paper hypothesizes that dense embeddings lose the benefit of statistical embedding approaches in using corpus statistics, and proposes a two-step method to leverage the corpus information (e.g., documents in the same domain). The method first uses an extra embedding model to rearrange the training set by using their embeddings for clustering, such that similar documents can be in same batches for contrastive learning; in inference time, it further uses similar documents to get embeddings contextualized on these given documents.
The authors claim that no hard negative mining is required to achieve state-of-the-art. However, the first step of the method (grouping similar documents) is essentially hard negative mining and is shown to be a key contribution to the performance. At the end, it is mentioned that an extra hard negative per query is used to achieve the best performance.
Yes
No ethics review needed.
- The paper motivates the work by introducing context into document embeddings. However, the context is collected by clustering and therefore it is more similar to increase the difficulty of the in-batch negatives.
- I also don't fully get how the batch of documents can capture the domain shift at inference time.
- A new batch sampling technique.
  - State-of-the-art on the MTEB benchmark.
The paper proposed a batch sampling and architecture change to allow for contextual document embeddings. They achieve state-of-the-art results on the MTEB benchmark.

They embed a document in a two-stage approach. They first use clustering on embeddings generated by existing text embedding model to form a batch. Then they generate a single embedding by taking both the input document and the context documents in the same batch.
Even though the paper is motivated by adapts the model to out-of-domain corpus, it's not evaluated on the domain-shift paradigm.
Yes
No ethics review needed.
See above.
(1) Overall, the paper works on an important problem and provides two clean improvements that seem well-motivated and effective.

(2) The paper is well-written and clear: each section is well-organized, the methods are well-motivated, and each aspect of the method is explained clearly.

(3) The experiments are thorough and include useful ablations and analysis.
The paper proposes two improvements for document embedding models: (1) using clustering to group similar documents together to make contrastive batches more difficult, and (2) making document embedding contextual through an alternate architecture that takes in not only the document itself, but also neighboring documents. Both improvements lead to better performance on document retrieval benchmarks.
(1) The contextual architecture seems more expensive than its non-contextual counterpart, so the experiments would be clearer if they also reported wall clock training time. For example, from what I understand, all four methods in table 1 were trained for the same number of steps; would the results change if they were trained for the same amount of time instead?

(2) The paper states that hyperparameters were chosen based on the small-scale experiments, but later also states "For our final model (anon-model-v1), we select the best of the supervised models, which comes from finetuning on the BGE dataset." What is the set of models you are selecting from, and what metric are you using to perform selection?

(3) The paper mentions that the "largest improvements [were] in ArguAna and SciFact, two of the smaller and more out-of-domain datasets," but I wasn't able to find a breakdown of the BEIR results for individual datasets in Table 1 or the supplementary material.
Yes
No ethics review needed.
See weaknesses section for questions and concerns.

Few suggestions below:

1. Equation on page 4 in the manuscript uses symbol "m" in two different contexts which could make the notation a bit confusing. It might be better to use distinct symbols to avoid ambiguity.

2. The contextual architecture resembles to be somewhat similar to RETRO in terms of inspiration. It would be useful to add some discussion in the section 4.2.

3. Section 6 mentions that using single hard negative per query achieves better performance compared to without using any hard negatives. The authors should also report the results for both variants in Figure 5.
1. The proposed method incorporates corpus-level statistics through statistical retrieval method into dense neural biencoders to better contextualize the embeddings by conditioning on the neighboring documents. This effectively address the lack of contextual information not captured by existing neural biencoders and makes the contextualized document embeddings to be particularly beneficial in domain-specific retrieval tasks.

2. Most of the prior works on training dense neural biencoders often employ strategies like hard negatives or large batch sizes to achieve effective performance. It is interesting that the proposed method achieves competitive or superior performance without relying on such strategies.

3. The authors conduct extensive ablation studies to understand the impact of different components on the downstream performance, and these findings will be valuable for future research in this area.

4. The paper mostly list all the hyperparameters, datasets and models used in this work, ensuring the reproducibility of work.
The paper explores contextual document embeddings by factoring in the information from surrounding documents in context in order to effectively handle retrieval tasks in out-of-domain scenarios that can be extremely challenging for existing dense neural encoders.

The authors propose two complementary methods to improve the dense neural encoders in such settings: a) contrastive learning objective to explicitly incorporate neighboring documents into the contextual batch loss b) architectural modification to explicitly encode information from neighboring documents.

The results on MTEB benchmark demonstrate SOTA performance without the need of sophisticated strategies like hard negative mining, intra-GPU example sharing or large batch sizes which are often employed by prior works. The proposed method is agnostic and can be applied to any contrastive learning dataset or dual encoder models.
1. The proposed method relies on the availability of relevant document neighbors for contextualization in the second stage of proposed architecture. The authors provide some evidence (Figure 7) on how the model performance varies in scenarios with limited context by simulating using random documents. However, it is not clear how the model trained with the proposed method would perform in absence of context (using null tokens in place of contextual tokens). Does the method makes the model overly reliant on context?

2. Following on the previous point, the contextual tokens might need to be augmented with the null tokens and this may potentially lead to unnecessary computational overhead.

3. The effectiveness of the proposed method heavily depends on accurately clustering documents into fine-grained pseudo-domains using GTR model, as these clusters are later used for contextual training. I am uncertain if using a simpler embedding model would result in obtaining difficult batch configurations, as it might fail to account for nuances. Do you have any ablations with alternative embedding models?

4. The authors mention that they set the dataset input tokens to zero during the self-attention phase in the section 9.4 in the appendix ("position-agnostic embedding" para in the manuscript). However, self-attention plays a critical role in contextualizing and routing necessary information to the subsequent modules. I am not sure how contextualization will be achieved if this is true. I would appreciate clarification from the authors.
A new way to do contrastive learning for text embedding models
@inproceedings{
anonymous2024contextual,
title={Contextual Document Embeddings},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=Wqsk3FbD6D},
note={under review}
}
Dense document embeddings are central to neural retrieval. The dominant paradigm is to train and construct embeddings by running encoders directly on individual documents. In this work, we argue that these embeddings, while effective, are implicitly out-of-context for targeted use cases of retrieval, and that a contextualized document embedding should take into account both the document and neighboring documents in context - analogous to contextualized word embeddings. We propose two complementary methods for contextualized document embeddings: first, an alternative contrastive learning objective that explicitly incorporates the document neighbors into the intra-batch contextual loss; second, a new contextual architecture that explicitly encodes neighbor document information into the encoded representation. Results show that both methods achieve better performance than biencoders in several settings, with differences especially pronounced out-of-domain. We achieve state-of-the-art results on the MTEB benchmark with no hard negative mining, score distillation, dataset-specific instructions, intra-GPU example-sharing, or extremely large batch sizes.  Our method can be applied to improve performance on any contrastive learning dataset and any biencoder.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
text embeddings retrieval context contrastive
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/132db5066db5c576c41a3b464c7000008038007b.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/10d3b9708368c7e045c36520cd4dd7f90dda2427.pdf
Contextual Document Embeddings
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission