Figure 4: 2D projection of tasks’ data. 90 random samples per task’s data are embedded using multlingual-e5-small
model (Wang et al., 2022). The embeddings are reduced to 2 dimensions using PCA. The centroid of each task’s
data is represented, along with the ellipse showing the standard deviation along each axis.
Label # raw #mteb_evalDescription
shs 58706 6701 Human and social sciences (Sci-
ences humaines et sociales)
sdv 11049 4803 Life science [Biology] (Sciences du
vivant [Biologie])
spi 3601 3451 Engineering science (Sciences de
l’ingénieur [Physics])
info 3446 3263 Computer Science (Informatique)
sde 2830 2754 Environment science (Sciences de
l’environnement)
phys 2003 1926 Physics ( Physique)
sdu 1177 1158 Planet and Universe [Physics]
(Planète et Univers [Physique])
math 862 824 Mathematics ( Mathématiques)
chim 764 734 Chemistry ( Chimie)
scco 652 619 Cognitive sciences (Sciences cogni-
tives)
qfin 183 N/A Economy and quantitative finance
(Économie et finance quantitative
stat 52 N/A Statistics ( Statistiques)
other 18 N/A Other ( Autre)
stic 14 N/A N/A
nlin 12 N/A Non-linear Science [Physics] (Sci-
ence non linéaire [Physique])
electromag 3 N/A Electro-magnetism (Electro-
magnétisme)
instrum 2 N/A Instrumentation [Physics] (Instru-
mentation [Physique])
image 1 N/A Image
Table 4: Distribution of classes in HAL the raw and
mteb_eval subsets of the dataset.
Task type Model Score
Classification (F1-score)TF-IDF + LR 0.60 (±0.002)TF-IDF + SVC 0.61 (±0.001)CamemBERT (fine-tuned)*0.6 (±0.008)GPT-4 (ICL)** 0.30
Topic Modeling TF-IDF + LDA 0.49 (Coherence)-8.23 (Perplexity)
Table 5: Baselines results for HAL on a classification
task and topic modeling.
* CamemBERT was finetuned for 5 epochs with learn-
ing rate of 1e−4 (+ lr scheduler) and a batch size of 64.
** Due to limited budget, we evaluate GPT-4 ICL ca-
pabilities on a limited subset of our dataset (600 first
samples from the test set that is generated using the
same seed as for other experiments).
• Bitext Mining: F1 score
• Classification: Accuracy
• Clustering: V measure
• Pair Classification: Average Precision (AP)
• Reranking: Mean Average Precision (MAP)
• Retrieval: Normalized Discounted Cumula-
tive Gain at k (NDCG@k)
• STS: Spearman correlation based on cosine
similarity
14