correlated (see Figure 12). We preferred to propose
datasets even if they could introduce biases rather
than not address the task in the benchmark. Note
that each task type can be considered individually.
We hope additional resources will be developed
in the French-speaking community to enrich our
comparison.
Benchmark validity over time As with all
benchmarks, their reliability over time can be dis-
cussed as the field evolves fast. The models se-
lected for the analysis conducted in this paper are
those available at this time, new outperforming
models will be created and shall be evaluated. Our
work extends MTEB and thus simplifies the ad-
dition of new datasets for evaluation and allows
running new models. With this effort, we hope
this will simplify the evaluation of new models pro-
posed by the community to keep our work up to
date.
Data contamination issues Bias may exist for
models that use the training sets of the provided
evaluation datasets for their training. It consider-
ably improves their performance on the benchmark,
favouring them over other models. This is particu-
larly worrying for models that do not communicate
about the datasets used during training, such as pro-
prietary models. Generally speaking, it would be
interesting to calculate the similarity between the
datasets used to train the models and those used to
test them to check that they are far enough apart to
draw general conclusions.
Focus on sentence embeddings Finally, like the
original version of MTEB, the comparison focuses
mainly on sentence embeddings. Other tasks could
be added to cover word embeddings and, therefore,
more NLP tasks.
Acknowledgements
We would like to thank Wikit 11 and Esker12 for
providing compute and funding this research.
References
David Ifeoluwa Adelani, Marek Masiak, Israel Abebe
Azime, Jesujoba Oluwadara Alabi, Atnafu Lam-
bebo Tonja, Christine Mwase, Odunayo Ogun-
depo, Bonaventure F. P. Dossou, Akintunde
Oladipo, Doreen Nixdorf, Chris C. Emezue,
11https://www.wikit.ai/
12https://www.esker.com/
Sana Al-Azzawi, Blessing K. Sibanda, Davis
David, Lolwethu Ndolela, Jonathan Mukiibi,
Tunde Oluwaseyi Ajayi, Tatiana Moteu Ngoli, Brian
Odhiambo, Abraham Toluwase Owodunni, Nnae-
meka Obiefuna, Shamsuddeen Hassan Muham-
mad, Saheed Salahudeen Abdullahi, Mesay Gemeda
Yigezu, Tajuddeen Rabiu Gwadabe, Idris Abdulmu-
min, Mahlet Taye Bame, Oluwabusayo Olufunke
Awoyomi, Iyanuoluwa Shode, Tolulope Anu Ade-
lani, Habiba Abdulganiy Kailani, Abdul-Hakeem
Omotayo, Adetola Adeeko, Afolabi Abeeb, An-
uoluwapo Aremu, Olanrewaju Samuel, Clemen-
cia Siro, Wangari Kimotho, Onyekachi Raphael
Ogbu, Chinedu E. Mbonu, Chiamaka Ijeoma Chuk-
wuneke, Samuel Fanijo, Jessica Ojo, Oyinkansola F.
Awosan, Tadesse Kebede Guge, Sakayo Toadoum
Sari, Pamela Nyatsine, Freedmore Sidume, Oreen
Yousuf, Mardiyyah Oduwole, Ussen Kimanuka,
Kanda Patrick Tshinu, Thina Diko, Siyanda Nx-
akama, Abdulmejid Tuni Johar, Sinodos Gebre,
Muhidin A. Mohamed, Shafie Abdi Mohamed,
Fuad Mire Hassan, Moges Ahmed Mehamed, Evrard
Ngabire, and Pontus Stenetorp. 2023. Masakhanews:
News topic classification for african languages. In
International Joint Conference on Natural Language
Processing.
Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab,
Aitor Gonzalez-Agirre, Rada Mihalcea, German
Rigau, and Janyce Wiebe. 2016. SemEval-2016
task 1: Semantic textual similarity, monolingual
and cross-lingual evaluation. In Proceedings of the
10th International Workshop on Semantic Evaluation
(SemEval-2016), pages 497–511, San Diego, Califor-
nia. Association for Computational Linguistics.
Arthur Barbosa, Máverick Ferreira, Rafael Fer-
reira Mello, Rafael Dueire Lins, and Dragan Ga-
sevic. 2021. The impact of automatic text transla-
tion on classification of online discussions for social
and cognitive presences. In LAK21: 11th Interna-
tional Learning Analytics and Knowledge Confer-
ence, LAK21, page 77–87, New York, NY , USA.
Association for Computing Machinery.