A Supplementary materials for datasets
A.1 All datasets
Table 3 displays the size of each dataset along with
the average number of tokens per sample and their
references. The dataset’s content was tokenized
using cl100k_base encoding. For Retrieval, the
two numbers refer to the queries and the docu-
ments. For Reranking, the three numbers refer to
the queries, the pairs of queries with relevant docu-
ments and the pairs of queries with irrelevant ones,
respectively. The pairs of queries and documents
are obtained from the 90 documents extracted. For
SummEvalFr, the three numbers refer to the texts,
human and machine summaries, respectively.
Figure 3 represents the semantic similarity be-
tween each dataset. The methodology was as fol-
lows: 90 random samples per dataset are embedded
using the multilingual-e5-large model. The embed-
dings of each dataset’s samples are averaged. The
similarity between each dataset is then calculated
using cosine similarity as in (Muennighoff et al.,
2022).
We complement this analysis by observing the
dataset’s clouds of embedding in a 2D plane using
PCA in Figure 4.
A.2 Created datasets
Syntec Figure 5 shows an extract from the Syntec
dataset with a document and a query relative to this
document.
HAL Figure 6 is an extract from the HAL
dataset. Table 4 lists the distribution of classes
(domain field) for the HAL dataset on raw
subset and mteb_eval subset, which is used
for MTEB evaluation. Labels descriptions
can be found at this URL: https://api.archives-
ouvertes.fr/ref/domain/?q=*:*&rows=393 or in Ta-
ble 4. After pre-processing, mteb_eval covers titles
from 10 domains as classes with less than 500 sam-
ples were removed. In the MTEB evaluation subset
of the dataset, titles composed of 2 words or less
have been removed (371 samples), resulting in an
average word count of 13.4. Figure 7 shows the
word count distribution per title. Furthermore, the
dataset has been cleaned up by manually remov-
ing all non-French titles. Additionally, it can be
observed in Table 4 that in the original raw dataset,
the shs and sdv classes represent by far the majority
of the dataset samples with respectively 58706 sam-
ples (73%) and 11049 samples (13%). In order to
mitigate the class imbalance while preserving the
majority of those classes, they have been randomly
subsampled to 6701 and 4803 samples. Further-
more, baseline models have been trained and tested
to assess the usability of this dataset in other tasks,
such as classification and topic modeling. Table 5
shows the results obtained.
SummEvalFr Extracts of humans and machine
summaries translated in French from SummEvalFr
and the original ones in English from SummEval
(Fabbri et al., 2021) are shown in Figure 9. As ex-
plained in section 3.1.3, we use a LLM to evaluate
the quality of translations for human summaries,
we provide the prompt used with GPT-4 for this
evaluation in Figure 8.
Table 6 shows the distribution of ratings given
by the LLM. With the scale being 10, we man-
ually verify random samples rated above 9. We
verify all samples with ratings under 9 and those
with no provided rating (N/A) due to the triggering
of the OpenAI content management policy. The
LLM suggests that 60 samples are not correctly
translated. These were verified manually, and after
checking, less than 10 samples only needed to be
corrected.
B Supplementary materials for
correlation analysis
This section presents various correlations computed
based on the model results on the proposed bench-
mark.
Figure 10 represents cross-correlations between
models’ performances and their studied character-
istics as a heatmap.
Figure 11 represents the Spearman correlations
in terms of performance across models.
Figure 12 represents the Spearman correlations
in terms of performance across datasets.
C Supplementary materials for models
We present in this section the model characteristics
we collected for the 46 evaluated models.
For evaluating prompt-based models such as