with respect to model ranking?
To go further than the correlation analysis among
datasets regarding their topics (see section 3.1.5),
subsequent analysis will be conducted regarding
how they rank models. Additionally, complemen-
tary insights will be derived from examining cor-
relations of models relative to their strengths and
weaknesses across different datasets.
4 Results and discussion
In this section, we present the results through the
prism of our research questions.
Q1: Is there a model that outstands on all
tasks?
Models performances for each task are presented
in appendix Tables 9, 10, 11, 12 and 13. Figure
1 shows the critical difference diagram of average
score ranks.
As in MTEB (Muennighoff et al., 2022), no
model claims state-of-the-art in all tasks even if
the text-embedding-3-large model is in first place
on average on all tasks (see Table 9). It ranks
first for the classification and reranking tasks. For
the clustering task, text-embedding-ada-002 is the
best model. The models voyage-code-2, text-
embedding-3-small and mistral-embed share the
top positions in the retrieval task ranking. For the
pair classification task, laser2 is ahead of its com-
petitors. Finally, sentence-camembert-large leads
on the STS task and multilingual-e5-small has the
best results for summarization.
Figure 1 shows a global model comparison
across all datasets. The models are arranged hori-
zontally according to their performance, with the
best models on the left. The black bars repre-
sent the statistical equivalence between the mod-
els’ performances. The statistically equivalent
top performers for this benchmark are OpenAI’s
models text-embedding-3-large, text-embedding-3-
small and text-embedding-ada-002. Interestingly,
many models do not show a significant perfor-
mance gap between their base and large flavours.
Some French models stand out among the multi-
lingual models, such as Solon-embeddings-large-
0.1, sentence_croissant_alpha_v0.3 and sentence-
camembert-large.
Q2: Are there any links between model
characteristics and performance?
The Spearman correlations between the average
rank of the models and their characteristics are the
following:
• Tuned for sentence similarity: 0.727
• Finetuned vs pretrained: 0.544
• Model number of parameters: 0.49
• Embedding dimension: 0.452
• Closed source: 0.449
• Max sequence length: 0.336
• Multilingual: 0.103
• English: 0.025
• English but tuned on other languages: -0.025
• French: -0.134
• Bilingual: -0.135
Additionally, all cross-correlations between charac-
teristics are reported in appendix Figure 10.
As expected, the score most strongly correlates
with whether the evaluated models were trained on
a sentence similarity task. Of course, this criterion
is connected to the more general Finetuned one.
The only top-performing models solely pre-trained
are from the E5 family, where the pre-training is,
in fact, contrastive and optimized for similarity.
Conversely, models pre-trained on token-level tasks
and generating embeddings via pooling appear less
well-suited for the benchmark tasks.
Furthermore, we observe a performance correla-
tion with the embedding dimension and the model’s
number of parameters, which are often correlated
themselves. This appears very clearly on the rela-
tive ranking of E5 and T5 models (see Figure 1).
However, some small models perform very well
on the benchmark, such as the standard version
of the multilingual universal sentence encoder or
Solon-embeddings-base-1.0. Notably, the maxi-
mum sequence length, while an important criterion
for generative tasks with LLMs, is less correlated
with performance than the other dimensions. This
can be explained by many datasets containing rel-
atively small texts (see appendix Table 3 showing
that 14 datasets have less than 50 tokens).
Regarding language, it is surprising that good
performance is not particularly correlated with
French models in particular. In reality, the other