text: MasakhaNEWSClusteringP2P). Table 3 shows de- 
  tails of each task data used for running the bench- 
  mark. 
  
  2019; Le et al., 2020). Most French models for 
  sentence embeddings have been developed by the 
  open-source community2, by fine-tuning models 
  like CamemBERT(Martin et al., 2019) or Crois- 
  santLLM(Faysse et al., 2024). 
  
  This section describes the 3 new datasets we in- 
  troduce, quality checks performed and an analysis 
  of the semantic similarities between datasets. 
  
  Benchmarks Embedding models are generally 
  compared on specific tasks, such as information 
  retrieval, STS or reranking (Thakur et al., 2021; 
  Agirre et al., 2016; Wang et al., 2021). Other 
  works evaluate embedding models on multiple 
  tasks (Wang et al., 2018; et al., 2022; Conneau and 
  Kiela, 2018) or compare meta-embeddings (García- 
  Ferrero et al., 2021). The most comprehensive 
  benchmark to date is MTEB (Muennighoff et al., 
  2022). MTEB still has a critical limit: it mainly 
  focuses on English. Some initiatives already ex- 
  tended this benchmark to other languages, such as 
  Chinese (Xiao et al., 2024) and German (Wehrli 
  et al., 2024). Our work comes with the same am- 
  bition for French. It relies on the MTEB structure 
  that provides a solid basis for analysis and extends 
  it to a new language. 