ESE: Espresso Sentence Embeddings
sentence embeddings;semantic textual similarity;information retrieval;retrieval-augmented generation
We would like to thank Reviewer 5KSE for your helpful comments. Below are our answers to your concerns.

## Concern 1: Lack of Comparison with Strong Baselines like Llama 3.1

**Answer:**

Our work focuses on scalable sentence embeddings. **Our sentence embedding model builds upon powerful transformers**, including both bi-encoder transformers (BERT, RoBERTa) and large language models (LLaMA, Qwen). 
To establish our model's competitive performance, **we have carefully selected strong and well-recognized baselines**, including:

1. bge-base-en-v1.5 (>3M monthly downloads on HuggingFace)
2. UAE-Large-V1 (Outperforming OpenAI embeddings on MTEB, >1M monthly downloads on HuggingFace)
3. Popular LLM-based backbones: Qwen and LLaMA2

Using these powerful backbones, we conducted comprehensive comparisons with RAW and MRL to demonstrate the scalability and effectiveness of our proposed model.

Regarding LLaMA 3.1: We acknowledge its recent release and **appreciate the suggestion to include it in our comparisons**. While the rapid development of LLMs makes it challenging to update to the latest backbones continuously, **we are currently working on extending our experiments to include LLaMA 3.1. It will take some time. We expect to complete these additional experiments and update our results within next few days.** **Please stay tuned!**

---

## Concern 2: The paper does not include comparisons with OpenAI's embedding models, which are considered significant benchmarks in the field of NLP.

**Answer:**

The reasons why we did not compare with OpenAI's embeddings are as follows:

First, OpenAI's embedding models are closed-source, making it impossible to apply our proposed ESE training methodology to them for a direct comparison.

Second, OpenAI's API only provides access to the final layer embeddings, preventing any analysis or comparison of shallow layer representations. This limitation is particularly relevant since improving shallow layer performance is a key contribution of the proposed ESE approach.

Third, our chosen backbone **UAE-Large-V1 is a strong open-source model that outperforms OpenAI's text-embedding on the MTEB benchmark [1].** Following your suggestion, we extend our Table 1 to add the last layer performance of OpenAI text-embedding-3, as follows:

| Model                         | STS12 | STS13 | STS14 | STS15 | STS16 | STSB  | Sick-R | Average |
|-------------------------------|:-----:|-------|-------|-------|-------|-------|--------|---------|
| OpenAI text-embedding-3-large | 71.98 | 85.82 | 80.5  | 87.51 | 84.48 | 82.34 | 79.18  | 81.69   |
| UAE-Large-V1                  | 79.09 | 89.62 | 85.02 | 89.51 | 86.61 | 89.06 | 82.10  | 85.86   |
| ESE-Large                     | 79.64 | 90.40 | 85.76 | 90.33 | 86.64 | 88.54 | 81.09  | **86.06**   |


The results demonstrate that ESE can outperform strong baselines. Due to the aforementioned API limitations regarding shallow layer access, a comprehensive comparison of shallow layer emebddings with OpenAI's model remains unfeasible.

---


Thank you again for your very kind effort and professional input! We will include the comparison between our results and OpenAI’s embedding in our final version.

**We hope Reviewer 5KSE could consider raising the score accordingly if we have addressed your concerns. If you have any further concerns, please don't hesitate to let us know.**


---


**Reference:**

- [1] Muennighoff, N., Tazi, N., Magne, L., & Reimers, N. (2022). MTEB: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316.
Response to Reviewer 5KSE
Yes
No ethics review needed.
Please refer to the weakness section.
1. The paper is well-organized and easy to follow. 
2. Experimental results are clearly presented, demonstrating the effectiveness of the proposed ESE method.
The paper proposes an approach for generating sentence embeddings that are scalable both in terms of model depth and embedding size. The proposed Espresso Sentence Embeddings (ESE) model incorporates two key processes: the learn-to-express process and the learn-to-compress process. The learn-to-express process aims to encode crucial latent representations in shallow layers, while the learn-to-compress process uses Principal Component Analysis (PCA) to condense essential features into the initial dimensions of the embeddings. The paper demonstrates that ESE can produce high-quality embeddings that are competitive with or surpass existing methods on standard benchmarks for semantic textual similarity (STS) and retrieval-augmented generation (RAG) tasks, particularly at reduced model depths and embedding sizes.
1. Lack of Comparison with Strong Baselines like Llama 3.1
2. The paper does not include comparisons with OpenAI's embedding models, which are considered significant benchmarks in the field of NLP.
Yes
No ethics review needed.
## Q1:

> The possible reason is the ESE's high-quality embeddings in varying scaled sizes allow easier and more effective indexing, thus improving RAG efficiency

I do not see how this affects the indexing process. Can you elaborate a bit more?

## Q2:

Could you share the code for the inference used in the case of MRL or point me to the part of the code that does this in the anonymized repository?
1. The authors fuse two existing ideas (model truncation and embedding truncation) into a single training process, which is a somewhat limited novelty. However, this provides much-needed flexibility for real-world use cases. 
2. The method is backed up by empirical results on seven STS datasets, one RAG dataset, and ablation studies that justify method design. 
3. The paper uses figures to better explain concepts, which is nice. However, I often found it lacking details for experiments and ablation studies.
4. This paper introduces the idea of training multiple configurations of a model at once. This is very useful for efficiency-critical and memory-constrained environments. The method can be handy for engineering use cases.
This work introduces a method for fine-tuning sentence embeddings while simultaneously providing flexibility in model size and embedding size. The method consists of two parts: learn-to-express, which trains different model depths, and learn-to-compress, which focuses on compressing information into a sub-embedding. The authors provide extensive experimental results on STS datasets and RAG on QA, including visualizations for the quality of the embeddings as well as results on inference efficiency. For the ablation studies, they investigate PCA on different dependencies, the effect of embedding sizes for PCA, and weighting present in the loss function.
## 1. Unfocused contribution statement

> To the best of our knowledge, we are the first to learn sentence embedding with information compression, presenting scalable embedding inference to both model depths and embedding dimensions

> We are the first to employ the information compression technique to scale sentence embeddings"

If by "information compression" the authors are referring to model depth, there are already existing papers on this topic, some of which are even cited (e.g., "BeLLM: Backward Dependency Enhanced Large Language Model for Sentence Embeddings" and "Are ELECTRA's Sentence Embeddings Beyond Repair? The Case of Semantic Textual Similarity"). Additionally, other cited works focus on reducing the embedding size (e.g., "Matryoshka Representation Learning"). Please clarify your wording and specify precisely how your method differs from these approaches.

## 2. Incomplete related work

The related work section is quite incomplete, as it includes only a few well-known references, most of which are not closely connected to the paper's topic. For example, a few papers that address closely related topics (excluding the aforementioned references) are:

- Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings
- WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach
- What does BERT learn about the structure of language?
- How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings
- BERT Has More to Offer: BERT Layers Combination Yields Better Sentence Embeddings

## 3. Unmotivated design choices

The paper is very empirical in nature, which is not a bad thing. While most of the method's design is supported by intuitive explanations, I find the rationale for choosing the "align" loss function insufficiently explained. Ideally, a more theoretical approach and justification for introducing the loss functions would significantly strengthen the paper.

> Specifically, we use Euclidean distance, i.e., IndexFlatL2 in faiss, as the similarity measurement

Why use the Euclidean distance if you are optimizing for cosine similarity?

> Note that we leave the last layer unweighted due to its critical role in capturing sentence-level semantics

The reference paper mentioned only tests this on LLMs, so what is the design justification for using it on MLMs?

## 4. Potentially incorrect/misleading conclusions

> It allows the direct use of the first k-dimensional sub-embeddings at inference without repeatedly performing PCA and improves inference efficiency

I am not convinced by the second part of this statement. I can see that this improves efficiency ONLY if you consider using PCA. If you don't use any compression, the efficiency would be roughly the same, wouldn't it?

Furthermore, Figures 3, 4, and ≺ Avg. columns for Tables 1 and 2 provide an unfair comparison -- RAW and MRL do not tune the inner transformer layers. This comparison is repeated a couple of times throughout the paper as an insight: 
 
> RAW and MRL exhibit inferior performance in the shallow layers

> This shows the effectiveness of ESE's learn-to-express process in encoding the more important information into shallow layers

> Notably, ESE achieves a significant improvement at shallow layers, demonstrating its superior scalability

This leads an unwary reader to an incorrect conclusion. It would make more sense to tune each layer separately and compare it with your method.

Furthermore, Table 5 is a somewhat unfair comparison. Truncating the model depth does not mean the model has the same number of parameters as BERT_smal or BERT_tiny (due to different hidden dimensions). I would advise the authors to add the information on number of parameters and mention this while commenting on the results.

The comparison of distances after using TSNE is debatable. TSNE preserves the global structure, but measuring distances after TSNE does not make sense (cf. [https://datascience.stackexchange.com/questions/17314/are-t-sne-dimensions-meaningful](https://datascience.stackexchange.com/questions/17314/are-t-sne-dimensions-meaningful)). For more rigor, I would instead use MDS for this visualization and comparison.

## Minor remarks

- Figure 7 seems to lack details on which layer was used to create the embeddings on the subfigures and which 128 dimensions were chosen for the RAW method.

- For Table 1, I would advise to put the links into the footnotes.

- I know you use CLS pooling in your experiments, but the implementation for mean pooling is incorrect.

```
    if pooling_strategy == 'avg':
        outputs = torch.sum(
            outputs * inputs["attention_mask"][:, :, None], dim=1) / torch.sum(inputs["attention_mask"])
```

- Is Equation 6 missing a sum over k?
Yes
No ethics review needed.
1) Is embedding dimension k also a hyperparameter? If so, does this mean that achieving different types or lengths of compressed representations requires multiple training iterations?

2) Does "PCA on sentence embeddings" in Table 2 mean directly applying PCA on the high-dim representations? If so, from the storage angle, the advantage of the proposed method is limited?
1. The paper is well-written, showing clearly the gains from the inner layers upon raw and MRL design.
2. I appreciate their experiments with different model scales and compression strategies, which clearly demonstrate the ablation and provide realistic comparison results.
This paper follows the motivation of Matryoshka Representation Learning (MRL) to learn sentence representation at different granularities, allowing a single embedding to adapt to the computational constraints of downstream tasks. The main idea is to use the PCA compressed representation to supervise representation learning at each grain. They show some positive results on STS tasks, more specifically, compared to previous works, their method makes the embedding extracted in the inner layers better, which is aligned with their design. Also, some ablation studies are provided to support the plausibility of their design. Basically, the paper is clear and well-written. However, I hold some concerns with their motivation and design, see the weaknesses part.
1) The design is complex: there are a lot of hyperparameters to tune with, like the weight for Loss_le, Loss_lc, the combination of MSE and KL-div, and joint training. It is hard to provide insights into where the main gain comes from.

2) Is embedding dimension k also a hyperparameter? If so, does this mean that achieving different types or lengths of compressed representations requires multiple training iterations?

3) I doubt real-world applications for this method. Their main motivation is to reduce inference latency, where the most straightforward way is to use a smaller backbone. However, as shown in Table 5, the gain compared with BERT_small is marginal.
Yes
No ethics review needed.
1. Why do MRL and RAW show consistent results across compression sizes in Figure 5? Does compression not cause any performance change for these models?
2. In Section 3.3, the compression dimension $k$, to which the vector is reduced, appears to be fixed. However, compressed embedding sizes vary in Figures 3 and 4. How is this possible? Are these models using different compression dimensions at training and testing time, or are they trained with varying k values each time?
**Motivation**

- The proposed work addresses the scalability challenges in sentence embedding that arise from fixed embedding sizes and a fixed number of layers. The training objective (learn-to-compress and learn-to-express) is well-motivated and effectively demonstrated throughout the paper.

**Methods and Experiments**

- The approach of directly training shallow layer embeddings (Section 3.2) significantly enhances embedding quality. Experimental results in Figure 3 show that even embeddings from shallow layers can achieve performance comparable to deeper layers in other models, highlighting the effectiveness of this approach.
- The novel compression method (Section 3.3) successfully captures interactions between vector dimensions, improving upon traditional PCA techniques, as shown in Table 2.

**Practicality**

- The framework presents a representation learning method that can be applied to low-resource scenarios.

**Writing**

- The paper's motivation is clearly written and easy to follow. The figures, particularly Figure 2, are well-designed and effectively illustrate the proposed methodology.
The paper proposes "Espresso Sentence Embedding" (ESE) for effective sentence embedding learning with two key processes "learn-to-express" and "learn-to-compress.” In the "learn-to-express" phase, the loss is computed using both the final layer and intermediate layer embeddings. In "learn-to-compress," ESE focuses on compressing embeddings by applying the softmax of the outer product of attention values. Experimental results demonstrate that ESE outperforms baselines like RAW (original model) and MRL (Matryoshka embedding; Kusupati et al., 2022) on STS and QA benchmarks. Furthermore, ablation studies confirm the robustness of ESE across various language models and embedding sizes.
The experimental setup is generally well constructed, but I have little concerns about following issues:

1. Kusupati et al. (2022) conduct extensive experiments across various classification and retrieval tasks using the ImageNet-1K dataset. In contrast, the paper under review **restricts its experimental scope solely to STS tasks,** which raises **concerns about the generalizability of the proposed method.** If the authors intended to focus exclusively on the NLP domain, I recommend including experiments in areas where sentence embeddings are widely applicable, such as information retrieval (e.g. BEIR benchmarks[1]). Although the paper includes results from HotpotQA (Table 4), it omits key metrics for evaluating the quality of embeddings from the retriever, such as supporting facts EM and F1. As a result, it is challenging to evaluate the sole impact on the retriever (i.e., embedding) within the overall pipeline.
2. It is uncertain whether **comparing ESE’s intermediate layer performance with that of other models is entirely fair.** Obviously, these models are intended to use the final layer embedding for inference, so a comparison based on shallow-layer performance would be inappropriate. For practical significance, it would be better to **emphasize that “non-final-layer embeddings (e.g., <12th or <24th layer) of ESE outperform the final-layer embeddings (e.g., 12th or 24th layer) of MRL or RAW” ** This would highlight ESE’s potential time-saving benefits by reducing forward passes while guaranteeing the fair comparison. However, in the BGE-based results shown in Figure 3, ESE’s performance at layer 11 does not appear to surpass that of MRL or ESE’s own performance at layer 12, which raises concerns about claimed practical advantages of  ESE.

[1] Thakur, Nandan, et al. "Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models." arXiv preprint arXiv:2104.08663 (2021).
Yes
No ethics review needed.
- Why, in Figure 4a, spearman correlation is degraded along the increment of embedding size?
- The proposed espresso sentence embeddings are indeed the first of its kind in this area.
- The experimental results denote good performance-efficiency tradeoffs have been achieved.
This paper is concerned about sentence embeddings. This paper claims that previous optimizations on the efficiency of sentence embeddings largely lie in the embedding size yet ignore the model depth. However, both the model depth and the embedding size could have a large impact on the efficiency. So in this paper, espresso sentence embeddings are proposed to consider both two facets in a joint optimization framework including both learning-to-express and learning-to-compress. The learning-to-express attempts to compress the model depth while the learning-to-compress attempts to compress the embedding size. Experimental results show that espresso sentence embedding enjoy better performance-efficiency tradeoffs than compared baselines.
- The idea, though not previous studied, is not that novel. And both two facets, i.e., the mode depth and embedding size, have been thoroughly studied in existing literature.
We are the first to learn sentence embedding with information compression, presenting scalable embedding inference to both model depths and embedding sizes.
@inproceedings{
anonymous2024ese,
title={{ESE}: Espresso Sentence Embeddings},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=plgLA2YBLH},
note={under review}
}
High-quality sentence embeddings are fundamental in many natural language processing (NLP) tasks, such as semantic textual similarity (STS) and retrieval-augmented generation (RAG). However, most existing methods leverage fixed-length sentence embeddings from full-layer language models, which lack the scalability to accommodate the diverse available resources across various applications. Viewing this gap, we propose a novel sentence embedding model Espresso Sentence Embeddings (ESE) with two learning processes. First, the learn-to-express process encodes more salient representations to shallow layers. Second, the learn-to-compress process compacts essential features into the initial dimensions using Principal Component Analysis (PCA). This way, ESE can scale model depth via the former process and embedding size via the latter. Extensive experiments on STS and RAG suggest that ESE can effectively produce high-quality sentence embeddings with less model depth and embedding size, enhancing inference efficiency.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
sentence embeddings semantic textual similarity information retrieval retrieval-augmented generation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/c4ec78f7a4f9f64fa9ac201437604d0b80b5e834.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
ESE: Espresso Sentence Embeddings
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission