0.2 0.4 0.6 0.8
text-embedding-3-large (0.087)
text-embedding-ada-002 (0.15)
text-embedding-3-small (0.17)
mistral-embed (0.19)
bge-m3 (0.22)
voyage-code-2 (0.24)
e5-mistral-7b-instruct (0.24)
Solon-embeddings-large-0.1 (0.25)
sentence_croissant_alpha_v0.3 (0.26)
sentence-t5-xxl (0.27)
embed-multilingual-v3.0 (0.27)
sentence-camembert-large (0.29)
bge-m3-custom-fr (0.3)
sentence_croissant_alpha_v0.2 (0.31)
multilingual-e5-large (0.31)
Solon-embeddings-base-0.1 (0.34)
multilingual-e5-base (0.34)
sentence-t5-xl (0.36)
voyage-2 (0.41)
sentence-croissant-llm-base (0.42)
paraphrase-multilingual-mpnet-base-v2 (0.43)
embed-multilingual-light-v3.0 (0.43)
multilingual-e5-small (0.44)
sentence-t5-large (0.45)
sentence-flaubert-base (0.46)
universal-sentence-encoder-multilingual-3 (0.49)
(0.94) flaubert_large_cased
(0.92) flaubert_base_uncased
(0.91) xlm-roberta-base
(0.86) xlm-roberta-large
(0.86) flaubert_base_cased
(0.86) udever-bloom-560m
(0.84) camembert-base
(0.78) bert-base-multilingual-cased
(0.75) distilbert-base-25lang-cased
(0.75) camembert-large
(0.74) distilbert-base-en-fr-cased
(0.74) distilbert-base-fr-cased
(0.71) multi-qa-MiniLM-L6-cos-v1
(0.69) all-MiniLM-L12-v2
(0.69) all-MiniLM-L6-v2
(0.67) laser2
(0.64) bert-base-multilingual-uncased
(0.64) udever-bloom-1b1
(0.62) text2vec-base-multilingual
(0.59) sentence-camembert-base
(0.56) distiluse-base-multilingual-cased-v2
(0.54) sentence-t5-base
(0.54) paraphrase-multilingual-MiniLM-L12-v2
(0.52) universal-sentence-encoder-multilingual-large-3
(0.51) LaBSE
Statistically equivalent performance
Lower 
performance
Better 
performance
Figure 1: Critical difference diagram representing the significant rank gaps between models. The axis represents the
normalized average rank of the models (lower is better). The black bars indicate that the difference in models’ rank
is not statistically significant, i.e. lower than the critical difference.
for similarity, prevail. Nevertheless, we can high-
light the excellent performance of a few French
models such as sentence-camembert and sentence-
croissant and Solon-embeddings.
Lastly, we emphasize that closed-source models
perform well on this benchmark (text-embeddings,
mistral-embed and voyage), but we lack informa-
tion about their characteristics. As more open-
source well-performing models get added in the
future, we could expect this correlation to decrease.
Note that the correlation between sequence length
and performance could be dragged by closed-
source models that have generally larger sequence
lengths.
Q3: Do monolingual models have multilingual
capabilities?
Multilingual French English +
tuning on
other languages
Bilingual English
Language
0.2
0.4
0.6
0.8Average performance
Model perfromance vs language
Figure 2: Model performance depending on the lan-
guage of the data they have been trained on.
We also studied the capabilities of models on the
French language when the language of the training
data varies. It is surprising to note the absence of a
clear correlation between the language the model
is trained on and its performance on French, as
shown by the large standard deviation in Figure 2.
Furthermore, monolingual models trained exclu-
sively on English such as voyage-code-2 show
very good results on French datasets compared
to models trained exclusively on French such as
flaubert derivatives and distilbert-base-fr-cased
(see Table D.1).
This is explained by the fact that a large part of the
selected French models generate embeddings using
a pooling strategy. Only a few are sentence trans-
former models, for which the pooled representation
is part of the model and trained with it, leading to
higher-quality embeddings. This is endorsed by
the excellent results of sentence-camembert-large,
a sentence transformer model trained on French
corpus and confirms the recent findings in terms of
model architecture (Gao et al., 2021).
Finally, it should be noted that a significant portion