2019; Le et al., 2020). Most French models for
sentence embeddings have been developed by the
open-source community2, by fine-tuning models
like CamemBERT(Martin et al., 2019) or Crois-
santLLM(Faysse et al., 2024).
Benchmarks Embedding models are generally
compared on specific tasks, such as information
retrieval, STS or reranking (Thakur et al., 2021;
Agirre et al., 2016; Wang et al., 2021). Other
works evaluate embedding models on multiple
tasks (Wang et al., 2018; et al., 2022; Conneau and
Kiela, 2018) or compare meta-embeddings (García-
Ferrero et al., 2021). The most comprehensive
benchmark to date is MTEB (Muennighoff et al.,
2022). MTEB still has a critical limit: it mainly
focuses on English. Some initiatives already ex-
tended this benchmark to other languages, such as
Chinese (Xiao et al., 2024) and German (Wehrli
et al., 2024). Our work comes with the same am-
bition for French. It relies on the MTEB structure
that provides a solid basis for analysis and extends
it to a new language.
3 MTEB for French
In this section, we describe the datasets and the
models that we propose for the French extension
of MTEB. We also list the research questions we
want to discuss with the results.
3.1 New Datasets
We identified 7 datasets relevant to French in the ex-
isting MTEB, which we assume are of good quality.
We complemented these with 8 external relevant
datasets proposed in the literature, such as BSARD
(Louis and Spanakis, 2022) and Alloprof (Lefebvre-
Brossard et al., 2023), which are proven to be good
quality. We created 3 new ones presented in Table 1
and assessed their quality with various procedures
and metrics. In addition to all performed checks,
we run multiple models on these datasets and pro-
vide results to show that they are neither trivial nor
impossible to solve (see Tables 10, 11, 12 and 13).
Therefore, as of today, our French MTEB
runs on 18 datasets. Some datasets are framed
differently according to the task category they
are used with. For example, MasakhaNEWS
dataset (Adelani et al., 2023) is used for
both Classification (MasakhaNEWSClassification)
and Clustering (MasakhaNEWSClusteringS2S and
2Models on the HuggingFace hub: sentence-camebert,
sentence_croissant_alpha_v0.3, Solon-embeddings-large-0.1.
MasakhaNEWSClusteringP2P). Table 3 shows de-
tails of each task data used for running the bench-
mark.
This section describes the 3 new datasets we in-
troduce, quality checks performed and an analysis
of the semantic similarities between datasets.
3.1.1 Syntec (Retrieval)
The Syntec French collective bargaining agree-
ment3 comprises around 90 articles. Despite its
topic, the language used does not feature the speci-
ficity of the legal vocabulary, making the data
suitable for benchmarking general-purpose mod-
els. The articles have been scraped for use as doc-
uments. Four annotators were divided into two
groups. Each group was given half of the articles
and asked to choose an article and write a question
about it. Each annotator wrote 25 questions. Thus,
a hundred questions have been manually created
and paired with the articles containing the answer4.
Examples of the dataset are available in the ap-
pendix Figure 5. This dataset could also be used
for text classification, clustering or topic modeling.
Regarding quality checks, every article’s integrity
has been reviewed while manually creating ques-
tions. We also manually checked that the questions
could only be answered using the annotated article.
3.1.2 HAL (Clustering)
Hyper Articles en Ligne (HAL) is a French open
archive of scholarly documents from all academic
fields. Scrapping this resource, we fetched 85,000
publications in French5. We extracted IDs, titles
and the author’s choice among domain labels. The
last 2 are provided by authors when submitting
their papers to HAL. Since domain annotations are
provided, the dataset can be used for many tasks,
such as topic modeling or text classification. To en-