tation and, in practical applications, the under-
lying storage and compute costs. We selected
models with embedding dimensions ranging
from 384 to 4096.
• Sequence length: Being the number of to-
kens that a model can consider as input, the
sequence length is important as it impacts the
unit that can be encoded (sentence, paragraph,
document). However, encoding overly long
sequences requires efficiently storing the rele-
vant information into a single vector. Among
the selected methods, this criterion varies
from 128 tokens to 32768.
• Model parameters:Often correlated with the
two first characteristics, parameter count is im-
portant for practical applications as it affects
usability on resource-efficient machines. The
selected models have a number of parameters
ranging from 20 million (∼100Mb in float32)
to 7 billion (∼28Gb).
• Language: This is a major feature of lan-
guage models. Some are monolingual, and
others are multilingual. Language is usually
acquired during pre-training, but sometimes,
models familiarize themselves with new lan-
guages at tuning. For the benchmark, we
selected French models, as well as bilingual
or multilingual models. We also included a
few ones that claimed to be English (e.g. all-
MiniLM-L12-v29).
• Model types:There are several strategies to
generate text embeddings such as aggregat-
ing (e.g. with average pooling) token-level
embeddings from raw pre-trained models, or
adding an extra contrastive learning step on a
sentence similarity task with, optionally, ad-
ditional transformation layers. We included
models of all types in our benchmark, summa-
rizing the model type information under two
relevant criteria: finetuned vs pretrained, and
trained for sentence similarity or not.
The selected models are visible in Figure 1, and
all of their characteristics are summarized in ap-
pendix Table 7. Overall, the selection includes the
best models from the sentence transformers frame-
work (Reimers and Gurevych, 2019), the most pop-
ular French NLP models (Le et al., 2020; Martin
9https://huggingface.co/sentence-transformers/
all-MiniLM-L12-v2
et al., 2019), their variants optimized for semantic
similarity (Reimers and Gurevych, 2019), numer-
ous multilingual models performing at the top on
MTEB (e.g E5 and T5), Bloom variants (Zhang
et al., 2023), models based on very recent power-
ful LLMs (Wang et al., 2023; Faysse et al., 2024)
and finally the proprietary models of OpenAI, Co-
here and V oyage. Certain models were selected in
multiple sizes to isolate the dimensionality effect
effectively. We provide information on the mod-
els’ licenses as reported in the Hugging Face hub10.
However, we encourage readers to conduct further
research before utilizing a model.
3.3 Evaluation
For the sake of homogeneity, models are evalu-
ated using the same metrics per task as in MTEB
(Muennighoff et al., 2022): Classification (Accu-
racy), Bitext mining (F1 score), Pair classification
(AP), Clustering (V measure), Reranking (MAP),
Retrieval (NDCG@10), Summarization and STS
(Spearman correlation based on cosine similarity).
BitextMining tasks are excluded from the aver-
age performance scores and therefore the figures,
as this task evaluates 2 languages instead of one,
and this benchmark focuses only on one language
(French). We present the results for both DiaBlaBi-
textMining and FloresBitextMining in Table 12.
Using the overall benchmark results, our goal
will be to answer the following research questions:
Q1: Is a model outstanding on all tasks?
As we are trying to find out whether one embed-
ding model is statistically better than the others for
French, the objective will also be to analyze the
performance of the models by tasks to facilitate
model choice for specific applications.
Q2: Are there any links between the model charac-
teristics and performance?
In section 3.2, we undertook the substantial task of
gathering the characteristics of all evaluated mod-
els. The goal here will be to analyze their impact