Dataset Syntec HAL SummEvalFr
Samples 100 queries
90 documents
26233 samples
10 classes
100 texts
1100 human summaries
1600 machine summaries
Creation process Scraping of Syntec col-
lective bargaining agree-
ment with articles as doc-
uments. Writing queries
corresponding to articles.
Scraping of HAL arti-
cles with id, title and do-
main. Further cleaning
with deduplication, lan-
guage filtering and class
subsampling.
Translation from English
to French with Deepl of
the SummEval dataset.
Annotation process 4 annotators divided into
2 groups. Each group was
given half of the articles
and asked to choose an ar-
ticle and ask a question
about it. Each annotator
wrote 25 questions.
Annotations provided by
authors when submitting
their paper. They choose
the domainbetween exist-
ing academic fields.
Detailed annotation pro-
cess provided in Fabbri
et al. (2021).
Quality checks Human verification of an-
notations.
Baseline models for clas-
sification and topic model-
ing.
Correlation between
BLEU and ROUGE
scores of the French
and the original English
datasets. LLM as-a-judge
translation rating and
human verification.
Table 1: New datasets details with the number of samples, the creation process, the annotation process and the
quality checks. All datasets are test splits.
• Samples belonging to domain classes with
less than 500 samples were removed, which
leads us to keep only 10 classes.
• Subsampling was performed on 2 classes con-
taining more than 10k samples each to lower
the number of samples and mitigate the unbal-
ance of the dataset.
More details about this process are provided in the
appendix A.2 along with some extracts in Figure
6. We make the dataset publicly available in both
their raw and clean versions. We use this dataset in
a clustering setup to cluster publications by their
title and use the domain as ground truth. To ensure
the quality of this dataset, we run 3 baseline mod-
els for classification: TF-IDF + SVM, a fine-tuned
Camembert (Martin et al., 2019) and GPT-4 lever-
aging In-Context Learning (ICL). Furthermore, we
run one baseline model for topic modeling: Latent
Dirichlet Allocation (LDA) (Blei et al., 2003) and
report scores in the appendix A.2.
3.1.3 SummEvalFr (Summarization)
The original SummEval dataset (Fabbri et al., 2021)
consists of 100 news articles from the CNN/Dai-
lyMail dataset. Each article has 11 human-written
summaries and 16 machine-generated summaries
annotated by 8 people with a score for coherence,
consistency, fluency, and relevance. We trans-
lated it from English to French using DeepL API6.
Since MTEB evaluation is based on the embedding
similarity between machine-generated and human-
generated summaries, we propose to compute the
ROUGE (Lin, 2004) and BLEU (Papineni et al.,
2002) metrics between machine and human sum-
maries for both French and English version. In Ta-
ble 2, we report the average of the scores as well as
their correlations between the two languages. The
correlation is high (above 0.7), showing that the
word and n-gram overlap between human and ma-
chine summaries is highly preserved in the French
version. One may argue that computing the met-
ric on fully translated texts (human and machine
summaries are both translated from English) may
introduce biases and not assess the quality of the
translations. For this purpose, we ensure the French
human summaries are correctly translated from En-
glish. We use an LLM as-a-judge (Zheng et al.,
6https://www.deepl.com
3