MTEB-French: Resources for French Sentence Embedding Evaluation and
Analysis
Mathieu Ciancone
Wikit, France
mathieu@wikit.ai
Imene Kerboua
Esker, France
imene.kerboua@esker.com
Marion Schaeffer
Wikit, France
marion@wikit.ai
Wissam Siblini
wissam.siblini92@gmail.com
Abstract
Recently, numerous embedding models have
been made available and widely used for var-
ious NLP tasks. The Massive Text Embed-
ding Benchmark (MTEB) has primarily sim-
plified the process of choosing a model that
performs well for several tasks in English, but
extensions to other languages remain challeng-
ing. This is why we expand MTEB to propose
the first massive benchmark of sentence em-
beddings for French. We gather 15 existing
datasets in an easy-to-use interface and create
three new French datasets for a global evalua-
tion of 8 task categories. We compare 51 care-
fully selected embedding models on a large
scale, conduct comprehensive statistical tests,
and analyze the correlation between model per-
formance and many of their characteristics. We
find out that even if no model is the best on all
tasks, large multilingual models pre-trained on
sentence similarity perform exceptionally well.
Our work comes with open-source code, new
datasets and a public leaderboard1.
1 Introduction
Embeddings are dense vector representations that
capture the semantics of an input. The first emblem-
atic example is Word2Vec, introduced by Mikolov
et al. (2013). It consists of neural architectures
trained to learn high-quality word representations
from contextual relationships in vast amounts of
text. Other models were proposed since then, lever-
aging the transformer architecture (Vaswani et al.,
2017) to produce both generic and contextualized
word embeddings using self-attention. Many mod-
els now exist with various architectures, mono-
lingual or multilingual, pre-trained or fine-tuned
(Naseem et al., 2021; Ding et al., 2023).
In this work, our primary objective is to in-
troduce a large-scale embedding benchmark for
1French table on: https://huggingface.co/spaces/
mteb/leaderboard
French to enable the research community and indus-
try to select the most relevant embedding methods
based on one’s specific needs, such as being open-
source, versatile or targeted toward a particular task,
having a small embedding dimension, the ability to
process long texts or their performance. To achieve
this goal, we undertake significant efforts in col-
lecting datasets to conduct a broad comparison of
models. We ensure that the datasets cover various
tasks within a common, easy-to-use framework,
and we create three new quality-checked datasets
to enhance this collection. We select a diverse
range of models, including prominent French and
multilingual models deemed most efficient. The re-
sults of our study already enable the community to
make informed model selections, whether for gen-
eral purposes or specific tasks. Additionally, our
implementation is open to the community and fea-
tures a public leaderboard, allowing the results to
evolve with new models or datasets. With this first
large-scale comparison, we perform an in-depth
analysis of the results, confirming well-known find-
ings such as the correlation between performance
and model/embedding dimensions and uncovering
interesting nuances.
2 Related Work
Sentence Embeddings Sentence embeddings are
required for many language tasks, such as Semantic
Textual Similarity (STS) and knowledge retrieval.
Many models have been proposed in the litera-
ture, leveraging pooling strategies (Devlin et al.,
2019; Muennighoff, 2022) or similarity fine-tuning
(Reimers and Gurevych, 2019) using a contrastive
framework (Gao et al., 2021; Neelakantan et al.,
2022; Ni et al., 2021; Wang et al., 2022; Zhang
et al., 2023), leveraging prompts (Wang et al., 2023)
or a two steps training process (Chen et al., 2024;
Lee et al., 2024). Few French-language models
have been proposed in the literature (Martin et al.,
1