2023) where given the original human summary
in English and its translation in French, the model
rates the quality of the translation from 0 to 10,
with 0 being of very bad quality and 10 being ex-
cellent. The prompt is available in Figure 8. Ad-
ditionally, we manually check random translations
with ratings between 9 and 10 to ensure the rating
is relevant. We do the same for all translations with
a score less than 9 and correct them7 (see the rating
distribution in Table 6).
Dataset BLEU ROUGE-1 ROUGE-2 ROUGE-L
SummEval 0.205 0.292 0.099 0.193
SummEvalFr 0.276 0.302 0.117 0.194
Correlation En-Fr 0.70 0.85 0.80 0.84
Table 2: Average ROUGE and BLUE scores computed
between machine summaries and human summaries
for the original English SummEval and its translation
to French. The correlations of the individual scores
between English and French are also reported.
3.1.4 Data for the Reranking task
The reranking task, as evaluated in MTEB, requires
datasets composed of a set of queries, each as-
sociated with relevant and irrelevant documents.
Despite our efforts, we found no French dataset
that natively exhibits such a structure. Thus, to
evaluate this task, we built data for the reranking
task based on the Syntec and Alloprof (Lefebvre-
Brossard et al., 2023) datasets. These already fea-
ture queries and labeled relevant documents. Irrele-
vant ones were added using the following process:
• To avoid bias, we use the BM25 algorithm
(Robertson and Jones, 1976) (which is a deter-
ministic method) to rank documents in terms
of relevance regarding each query.
• The top 10 documents that are not labeled as
relevant constitute the negative samples.
We recognize that this process leads to a high cor-
relation between the retrieval and reranking tasks.
We still think it is essential to make the latter avail-
able, with an open door to future improvement8.
7SummEvalFr available at: https://huggingface.co/
datasets/lyon-nlp/summarization-summeval-fr-p2p
8SyntecReranking available at: https:
//huggingface.co/datasets/lyon-nlp/
mteb-fr-reranking-syntec-s2p and AlloprofRerank-
ing available at: https://huggingface.co/datasets/
lyon-nlp/mteb-fr-reranking-alloprof-s2p
3.1.5 Similarity analysis
We investigate the proximity between the datasets’
topics to give insights about the benchmark con-
tents. The methodology introduced by Muen-
nighoff et al. (2022), i.e. computing an average
embedding of samples from each dataset, is used to
build a dataset-similarity matrix (displayed in ap-
pendix Figure 3). The distances between averaged
embedding vectors of each dataset (which range
from 0.89 to 1 in Figure 3) remain hard to interpret
into a dataset semantic proximity. Thus, we com-
plement this by observing the dataset’s clouds of
embedding in a 2D plane using PCA in Figure 4.
Figures 4 and 3 seem to correlate, showing high
similarity between two datasets when the same
underlying data is used in different tasks. Dataset
topics are pretty close, with some exceptions, such
as the Syntec dataset. As more datasets are added
to the benchmark, this analysis will help select new
data that do not produce redundant results. It may
also help to understand the link between the results
and the datasets’ topics.
3.2 Models
For comparison on our benchmark, we selected
various models to fulfil three objectives.
• Quantity: The aim was to compare a substan-
tial number of models (51 in total) to provide
comprehensive results, facilitating the com-
munity in selecting effective French models.
• Relevance: It was imperative to include
top performers from the MTEB benchmark
(Muennighoff et al., 2022). We mainly se-
lected multilingual models and some English
models to asses their language-transferring
abilities. Additionally, we integrated natively
French transformer-based models such as
CamemBERT (Martin et al., 2019),FlauBERT
(Le et al., 2020) and even the very recent
CroissantLLM (Faysse et al., 2024).
• Variety: Diverse model types were included