text: Native French resources unavailability The 
  availability of resources natively in French is an 
  obvious limitation of our work. Regarding mod- 
  els, there are far fewer options than with more 
  widespread languages such as English. 
  Indeed, 
  most of the existing French embedding models we 
  found are trained using either older architectures 
  or methods, unlike most recent multilingual mod- 
  els such as NV-Embed-v1 (Lee et al., 2024) or e5- 
  mistral-7b-instruct (Wang et al., 2023). Comparing 
  models by family would be beneficial, particularly 
  for evaluating French models against multilingual 
  models on the same architecture using the same 
  training technique. Resource limitations also ap- 
  ply to datasets. For example, the summarization 
  task dataset is translated, which can be less relevant 
  than a natively French dataset. We have also built 
  datasets for reranking tasks using existing ones 
  from retrieval task because we could not find any 
  in French. This construction process introduces a 
  bias as the model performance on both tasks may be 
  
  5 Conclusion and perspectives 
  
  In this work, we introduce a large-scale embed- 
  ding benchmark for French to enable the research 
  community and industry to select the most relevant 
  embedding methods based on their specific needs. 
  We undertake significant efforts in collecting 15 
  datasets and create 3 new quality-checked ones to 
  enhance this collection. The whole French bench- 
  mark runs on 26 tasks. We select a diverse range of 
  51 models, including prominent French and multi- 
  lingual models deemed most efficient to conduct a 
  broad comparison. Our implementation is open to 
  the community and features a public leaderboard, 
  allowing the results to evolve with new models or 
  datasets. After an in-depth analysis of the results, 
  OpenAI models perform significantly better than 