Summary
type
Original
(SummEval)
Translated
(Sum-
mEvalFr)
Human
summary
The whale,
Varvara, swam
a round trip
from Russia to
Mexico, nearly
14,000 miles.
The previous
record was set
by a humpback
whale that
migrated more
than 10,000
miles.
La baleine,
Varvara, a
parcouru à la
nage un trajet
aller-retour
entre la Russie
et le Mexique,
soit près de
14 000 milles.
Le précédent
record avait
été établi par
une baleine
à bosse qui
avait migré sur
plus de 10 000
miles.
Machine
summary
north pacific
gray whale has
earned a spot
in the record
for the longest
migration of a
mammal ever
recorded . the
whale , named
varvara , swam
nearly 14,000
miles from
the guinness
worlds records
. the record
was set by a
whale whale
whale that
swam a mere
10,190-mile
round trip . the
north coast
of mexico is
russian for
"barbara".
la baleine
grise du paci-
fique nord a
obtenu une
place dans le
record de la
plus longue
migration d’un
mammifère
jamais en-
registrée. la
baleine, nom-
mée varvara,
a nagé près
de 14 000
miles depuis
les records
du monde
guinness. le
record a été
établi par une
baleine baleine
qui a nagé
un voyage
aller-retour
de seulement
10 190 miles.
la côte nord
du mexique
est le nom
russe pour
"barbara".
Figure 9: Extracts of SummEvalFr dataset.
Quality Rating # samples
Good quality
10.0 186
9.5 661
9.0 193
Not good enough
8.5 16
8.0 5
7.5 7
7.0 3
6.0 3
5.0 2
4.0 1
3.0 1
2.0 3
N/A 19
Table 6: Ratings provided by the LLM judge for
the quality of human summaries translations of Sum-
mEvalFr from English to French.
• Summarization: Spearman correlation based
on cosine similarity
D.1 Average performance per task type
Table 9 presents the average performance of each
model on each task type.
D.2 Evaluation results per task
Tables 10, 11 12 and 13 present the models’ perfor-
mance on each task type. Table 10 presents the per-
formance on classification and pair classification
tasks. Table 11 presents the reranking and retrieval
performance. Table 12 presents the performance
on bitext mining, semantic textual similarity and
summarization. Table 13 presents the performance
on the clustering tasks.
16