MassiveScenario
MassiveIntent
MasakhaNEWS
MTOPIntent
MTOPDomain
AmazonReviews
PawsX
OpusparcusPC
Classification PairClassification
bge-m3 0.73 0.67 0.77 0.62 0.89 0.45 0.60 0.93
distilbert-base-25lang-cased 0.44 0.35 0.68 0.35 0.62 0.29 0.51 0.86
distilbert-base-en-fr-cased 0.44 0.35 0.68 0.35 0.62 0.29 0.51 0.86
distilbert-base-fr-cased 0.44 0.35 0.68 0.35 0.62 0.29 0.51 0.86
sentence-camembert-large 0.70 0.64 0.74 0.61 0.87 0.38 0.61 0.94
sentence-flaubert-base 0.63 0.59 0.71 0.53 0.79 0.40 0.58 0.93
Solon-embeddings-base-0.1 0.70 0.65 0.75 0.62 0.87 0.41 0.59 0.93
Solon-embeddings-large-0.1 0.71 0.67 0.76 0.69 0.89 0.42 0.60 0.94
sentence-croissant-llm-base 0.65 0.59 0.79 0.63 0.86 0.35 0.63 0.91
bert-base-multilingual-cased 0.44 0.37 0.64 0.38 0.64 0.29 0.53 0.87
bert-base-multilingual-uncased 0.44 0.38 0.76 0.39 0.64 0.29 0.53 0.87
camembert-base 0.39 0.31 0.66 0.29 0.58 0.30 0.52 0.83
sentence-camembert-base 0.61 0.52 0.70 0.43 0.77 0.36 0.57 0.92
sentence-camembert-large 0.69 0.63 0.81 0.59 0.86 0.38 0.60 0.95
embed-multilingual-light-v3.0 0.59 0.56 0.83 0.50 0.81 0.39 0.57 0.91
embed-multilingual-v3.0 0.67 0.63 0.83 0.61 0.86 0.42 0.61 0.94
flaubert_base_cased 0.11 0.07 0.71 0.09 0.26 0.25 0.52 0.82
flaubert_base_uncased 0.11 0.06 0.63 0.09 0.28 0.24 0.53 0.82
flaubert_large_cased 0.23 0.16 0.56 0.10 0.24 0.22 0.54 0.75
e5-mistral-7b-instruct 0.70 0.60 0.75 0.53 0.82 0.44 0.60 0.92
multilingual-e5-base 0.66 0.61 0.80 0.56 0.85 0.41 0.57 0.93
multilingual-e5-large 0.68 0.64 0.79 0.59 0.86 0.42 0.59 0.94
multilingual-e5-small 0.61 0.56 0.78 0.46 0.81 0.40 0.56 0.93
udever-bloom-1b1 0.50 0.43 0.81 0.51 0.69 0.35 0.62 0.86
udever-bloom-560m 0.22 0.15 0.68 0.16 0.35 0.27 0.60 0.82
laser2 0.59 0.53 0.66 0.57 0.76 0.34 0.70 0.94
bge-m3-custom-fr 0.75 0.67 0.70 0.61 0.90 0.42 0.61 0.93
sentence_croissant_alpha_v0.2 0.70 0.64 0.76 0.61 0.89 0.38 0.67 0.93
sentence_croissant_alpha_v0.3 0.70 0.65 0.76 0.59 0.88 0.36 0.65 0.93
mistral-embed 0.70 0.63 0.81 0.66 0.90 0.42 0.62 0.93
LaBSE 0.65 0.60 0.77 0.62 0.84 0.39 0.55 0.94
all-MiniLM-L12-v2 0.54 0.45 0.72 0.39 0.76 0.28 0.56 0.87
all-MiniLM-L6-v2 0.51 0.43 0.74 0.40 0.75 0.27 0.55 0.87
distiluse-base-multilingual-cased-v2 0.67 0.60 0.77 0.56 0.85 0.36 0.51 0.92
multi-qa-MiniLM-L6-cos-v1 0.50 0.43 0.76 0.37 0.73 0.27 0.57 0.88
paraphrase-multilingual-MiniLM-L12-v2 0.65 0.58 0.76 0.48 0.78 0.37 0.57 0.92
paraphrase-multilingual-mpnet-base-v2 0.68 0.62 0.78 0.52 0.80 0.40 0.58 0.93
sentence-t5-base 0.60 0.51 0.81 0.44 0.75 0.37 0.55 0.89
sentence-t5-large 0.64 0.57 0.80 0.48 0.80 0.41 0.60 0.91
sentence-t5-xl 0.66 0.61 0.80 0.54 0.85 0.44 0.63 0.92
sentence-t5-xxl 0.69 0.66 0.79 0.58 0.86 0.46 0.64 0.94
text2vec-base-multilingual 0.58 0.52 0.74 0.45 0.72 0.34 0.66 0.92
text-embedding-3-large 0.76 0.71 0.82 0.74 0.93 0.46 0.65 0.96
text-embedding-3-small 0.73 0.68 0.76 0.68 0.91 0.43 0.61 0.94
text-embedding-ada-002 0.71 0.65 0.82 0.64 0.89 0.44 0.60 0.94
voyage-code-2 0.70 0.63 0.82 0.59 0.88 0.42 0.61 0.93
universal-sentence-encoder-multilingual-3 0.70 0.61 0.82 0.54 0.85 0.34 0.52 0.91
universal-sentence-encoder-multilingual-large-3 0.73 0.66 0.72 0.64 0.88 0.35 0.54 0.93
xlm-roberta-base 0.23 0.14 0.60 0.19 0.44 0.27 0.51 0.85
xlm-roberta-large 0.24 0.16 0.66 0.15 0.37 0.27 0.53 0.84
Table 10: Performance of each model for Classification and Pair Classification.
22