Q4: Are there any correlations between
datasets with respect to model ranking?
The datasets correlation w.r.t model ranking are
presented in appendix Figure 12. Except for
two datasets (MasakhaNEWSClusteringP2P, Sum-
mEvalFr), the correlations, on average, are high.
There is still enough diversity to make each dataset
interesting for the French MTEB benchmark. Two
groups (SyntecReranking/ SyntecRetrieval, Mas-
siveScenarioClassification/ MTOPDomainClassi-
fication/ MassiveIntentClassification) exhibit no-
tably high correlations ( ∼0.97). It is interesting
to point out some sub-diagonal correlation blocks.
The datasets being arranged by task indicate that
models behave slightly more similarly within the
same task than between two different tasks. This
underscores the importance of having multiple
tasks in the benchmark to select general-purpose
models. For readers interested in specific tasks,
it is more relevant to examine task-specific rank-
ings rather than the overall one. The complemen-
tary results of model correlations w.r.t to strengths
and weaknesses on datasets are displayed in ap-
pendix Figure 11. Strong correlations in behavior
emerge among the variants of the same models
(e.g. DistilBERT, sentence-croissant, sentence-t5,
e5, etc.). Correlations are also generally observed
among numerous models trained using the sentence
transformers framework (Reimers and Gurevych,
2019), as well as proprietary models, e.g. from
Cohere and OpenAI. Conversely, these models fine-
tuned for sentence similarity, show minimal cor-
relation with pre-trained models for which token-
embedding pooling techniques are employed.
5 Conclusion and perspectives
In this work, we introduce a large-scale embed-
ding benchmark for French to enable the research
community and industry to select the most relevant
embedding methods based on their specific needs.
We undertake significant efforts in collecting 15
datasets and create 3 new quality-checked ones to
enhance this collection. The whole French bench-
mark runs on 26 tasks. We select a diverse range of
51 models, including prominent French and multi-
lingual models deemed most efficient to conduct a
broad comparison. Our implementation is open to
the community and features a public leaderboard,
allowing the results to evolve with new models or
datasets. After an in-depth analysis of the results,
OpenAI models perform significantly better than
the other models. However, other models should be
considered for their performance on specific tasks,
being open source or having a small embedding
dimension.
This work opens several doors for future im-
provements. By examining dataset diversity in
terms of topics and model ranking, we observe
that the benchmark would benefit from additional
datasets that introduce higher diversity. Beyond
classification, many tasks focus on semantic simi-
larity, explaining the strong performance of models
trained for similarity. Exploring novel tasks in the
generative spectrum or evaluating token embed-
dings (contextualized or not) on tasks like Named
Entity Recognition could be an interesting path
for future exploration. There are also opportuni-
ties for improvements on the model side. With
numerous existing models that could be added to
the leaderboard and many new proposals awaiting.
For instance, we can already see the promising ca-
pabilities of early variants of recent models (Faysse
et al., 2024) and expect that future proposals will
come to compete strongly with closed-source mod-
els. Ultimately, we hope to see the emergence of
other language-specific MTEB variants (e.g. for
high-resource languages like Spanish and German),
enabling a more comprehensive evaluation of mul-
tilingual model performance.
6 Limitations
Native French resources unavailability The
availability of resources natively in French is an
obvious limitation of our work. Regarding mod-
els, there are far fewer options than with more
widespread languages such as English. Indeed,