Semi-Parametric Retrieval via Binary Bag-of-Tokens Index
information retrieval;efficient retrieval;retrieval-agumented applications;RAG
Yes
No ethics review needed.
- what is the purpose of section 3.1? I guess the argument is around the relationship between MLM and vocabulary expansion - see e.g. https://dl.acm.org/doi/10.1145/3634912

- Unless I miss something, there is no justification of the second par of $L_{semi-para}$, i.e. $L(V_{BoT}(q), V_\theta(p))$ since it is never used. And no ablation shows the importance of this factor

- What is $VDR_{\beta}$ in table 1? I could not find this model in the VDR paper.

- I do not agree on the difference in efficiency (lines 499-506) between your model and BM25. While it is true that leveraging GPU is useful in your case, BM25 could actually be implemented on GPU, so the reported speedup does not mean much.
The main strength of the paper lies in the static representation of documents. As far as I know, this is the first model that relies on a simple index structure, that of representing the documents as a binary bag-of-token. This allows for potentially fast retrieval (although this can be debated, see weaknesses) of potential candidates that have then to be re-ranked.
This paper main contribution is a retrieval model where documents are represented by the presence (boolean) of tokens (i.e. a binary bag of tokens). The advantage of such an approach is that the index has to be computed just once (it does not depend on parameters). However, the precision of such a model being low, the authors show that it can be successfully trained with a dense model (used as a second-stage ranker) - and combined with it at inference time. Experiments are conducted on the Wiki21m and BEIR benchmarks and retrieval latency is evaluated - showing that this approach offers a good compromise between effectiveness and efficiency.
One of the main weaknesses is related to the number of collisions (i.e. number of documents for one token) increases with the size of the collection. It is unclear how this approach performs when the number of documents, their length, or both, increase. It would be important thus to investigate using the model on a larger collection (e.g. MS-Marco).

Another point is that the authors state (l. 329) that SOA training "techniques are orthogonal to the retrieval model and have not been applied in our works.". It is not clear if this is truly the case here, and no experiments have been conducted to check the potential of e.g. simple techniques like knowledge distillation. 

Other things:
- The authors propose to use the elu1p (p. 4) - it is not clear why softplus has not been used, the function is quite close to this and more "standard".
- The negative mining proposed for the Wikipedia collection is very specific to this collection (l. 292-293). In my opinion, this invalidates the results reported on the Wikipedia collection (table 1 p. 7) and would justify the use of the dismissed SOA training techniques.
Yes
No ethics review needed.
BM25 as a token retrieval baseline has stood the test of time, being competitive for three decades. Why do the authors think SiDR_beta handily outperform BM25? What can we learn from this? Could this be because it's not an apples to apples comparison? Are there other underlying factors? 

Why did the authors not consider hybrid retrieval systems a more natural baseline?
The paper is well motivated. The problems identified with constructing and refreshing vector based document indices are consistent with prevalent challenges in the industry. While most SOTA retrieval systems, especially those used in the industry, are hybrid, re-using representations across the two forms of retrieval seems novel. There is a good set of baselines, though the most relevant baselines are those that perform hybrid retrieval with vector based and token based approaches. The paper will also benefit from inclusion of more qualitative analysis, especially on how the aligned BoT query representations tackle classical IR problems like polysemy and synonymy.
This paper extends the VDR encoder (which also seems to be under review), aligning masked term prediction and bag of tokens representations. The paper shows that hybrid retrieval outperforms either vector based or token based retrieval alone, and the same parametric representation used in vector based retrieval can be "aligned" with bag of tokens to perform non-parametric retrieval.
One of the strengths of bag-of-token based retrieval is that it is easily interpretable. The paper misses an opportunity to demonstrate how going from a parametric representation (which is considered semantic retrieval) allows us to tackle the standard problems in IR such as polysemy and synonymy. How do the lack of weights (a la BM25) on the document side affect retrieval? 

A nit: since the paper uses a hybrid retrieval system, comparison against either vector retrieval alone is not an apples to apples comparison. For example, in Table 1, this system seems to handily outperform BM25. But BM25 doesn't have the benefit of query side synonyms. While the index is non-parametric, the query is not (it's an "aligned version" of the parametric embedding). This doesn't invalidate the results. Rather it provides a channel for further investigation into the kinds of representations that are produced.  

Comparison against other hybrid retrieval systems (Gao et al 2021, Kuzi et Al 2020) would make the paper a lot stronger. (These are ones that come to mind, but there may be more). 

Luyu Gao, Zhuyun Dai, Tongfei Chen, Zhen Fan, Benjamin Van Durme, and Jamie Callan. Complement lexical retrieval model with semantic residual embeddings. In European Conference on Information Retrieval, 2021. URL https://api.semanticscholar.org/CorpusID:232423090

Saar Kuzi, Mingyang Zhang, Cheng Li, Michael Bendersky, and Marc Najork. Leveraging semantic and lexical matching to improve the recall of document retrieval systems: A hybrid approach, 2020. URL https://arxiv.org/abs/2010.01195.
Yes
No ethics review needed.
How many threads does the retrieval use?
Does retrieval on GPUs use all 4 GPUs or a single one?
L146-150 the discussion of in-training retrieval is not understandable without explaining that in-training retrieval is required to mine hard negatives.
See the summary for more details.
* An interesting approach
* A semi-parameteric index retrieval is easy to carry out on a GPU
* A substantial evaluation using the BEIR datasets (plus additional QA datasets)
* Promising results
The paper focuses on sparse neural retrieval using a SPLADE-like model. It proposes to train a neural sparse bi-encoder V for both queries Q and documents D in such a way that it works well when computing similarity in the standard bi-encoder fashion, i.e., as the inner product of V(Q) and V(D) as well as through the inner product of V(Q) and the binary bag-of-tokens representation of a document.

This can be quite useful since it permits using a neural encoder together with non-parametric document representations (that do not required encoding) for effective and efficient retrieval.

Although I like the idea, but I have mixed feelings about the paper for several reasons:
1. The paper is somewhat confusing and hard to follow (even for people with deep expertise on various retrieval approaches).
2. The proposed semi-parametric approach SIDR_beta does not uniformly outperforms BM25 and on average it is worse on BEIR. What happens if we take BM25 and re-rank using VDR or SPLADE or other neural bi-encoder. There is prior work doing exactly this: Leonhardt, Jurek, et al. "Efficient neural ranking using forward indexes." Proceedings of the ACM Web Conference 2022. 2022.
3. Latency comparison with BM25 is unfair. We are comparing CPU and GPU while claiming that BM25 cannot run on GPU. This is not true. GPUs are worse at sparse inner product compared to dense ones, but they can still do it (e.g., I found one implementat of bm25 using Pytorch: https://github.com/jxmorris12/bm25_pt)

Moreover, the efficiency comparison needs to ensure we use a comparable CPU in terms of the price **AND** we use all threads. It is not clear to me if this is the case: the listed system uses 4 A100 GPUs which cost about 4x compared the Intel Xeon Platinum 8358. Are four GPUs used during retrieval?

On top of that, we need to make sure we use a very efficient BM25 implementation such as the one in PISA and not a Java-based one!

https://github.com/pisa-engine/pisa

Another unfairness "point" is that GPU has more limited memory than a CPU system, which further complicates comparison. You cannot see the difference when testing on tiny BEIR datasets, but this can be an issue when much larger datasets are used.
 
4. The paper uses a seemingly SPLADE-like bi-encoder VDR, but it is not clear why it was chosen over SPLADE. The paper avoids comparison of the methodology in the main part of the paper (I think authors should say that it is very much like SPLADE). In fact, for all people familiar with sparse lexical retrieval this is very confusing, b/c a lot of people know about SPLADE, but few are familiar with VDR.
please see the summary
Yes
No ethics review needed.
1. How does SiDR compare with other late parametric methods, and what are the reasons for its success in the late parametric scenario?
1. The proposed SiDR integrates the advantages of non-parametric and parametric retrieval, achieving competitive performance while maintaining high efficiency.
2. The paper provides clear explanations of the SiDR method, experiments, and analysis, making it easy to understand.
This paper introduces a Semi-parametric Disentangled Retrieval (SiDR) strategy that integrates the advantages of non-parametric and parametric retrieval, achieving competitive performance through the late parametric mechanism while maintaining high efficiency.
1. In Table 1, the performance of SiDR in parametric retrieval scenario does not surpass ANCE, even with the late parametric mechanism, yet the authors still report it as the best result with the bolded values.
2. In Table 2, under the non-parametric retrieval scenario, SiDR fails to surpass BM25 in most of datasets.
3. The late parametric mechanism is not novel, and the experiments should include more comparisons to various methods in this category.
4. Because of the comparison results with SiDR and ANCE/BM25, the experimental analysis should include an examination of the reasons for SiDR's better performance in the late parametric scenario.
Our paper presents a semi-parametric neural retrieval system that supports a non-parametric index to achieve efficiency and cost-effectiveness, meeting the growing demands of retrieval-augmented applications.
@inproceedings{
anonymous2024semiparametric,
title={Semi-Parametric Retrieval via Binary Bag-of-Tokens Index},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=l0fn10vSyM},
note={under review}
}
Information retrieval has transitioned from standalone systems into essential components across broader applications, with indexing efficiency, cost-effectiveness, and freshness becoming increasingly critical yet often overlooked. In this paper, we introduce SemI-parametric Disentangled Retrieval (SiDR), a bi-encoder retrieval framework that decouples retrieval index from neural parameters to enable efficient, low-cost, and parameter-agnostic indexing for emerging use cases. Specifically, in addition to using embeddings as indexes like existing neural retrieval methods, SiDR supports a non-parametric tokenization index for search, achieving BM25-like indexing complexity with significantly better effectiveness. Our comprehensive evaluation across 16 retrieval benchmarks demonstrates that SiDR outperforms both neural and term-based retrieval baselines under the same indexing workload: (i) When using an embedding-based index, SiDR exceeds the performance of conventional neural retrievers while maintaining similar training complexity; (ii) When using a tokenization-based index, SiDR drastically reduces indexing cost and time, matching the complexity of traditional term-based retrieval, while consistently outperforming BM25 on all in-domain datasets; (iii) Additionally, we introduce a late parametric mechanism that matches BM25 index preparation time while outperforming other neural retrieval baselines in effectiveness.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
information retrieval efficient retrieval retrieval-agumented applications RAG
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/89b269c02d7bc34440633c272c4856794bc7a7ee.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Semi-Parametric Retrieval via Binary Bag-of-Tokens Index
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission