text: • Quantity: The aim was to compare a substan- 
  tial number of models (51 in total) to provide 
  comprehensive results, facilitating the com- 
  munity in selecting effective French models. 
  
  • Relevance: 
  
  • To avoid bias, we use the BM25 algorithm 
  (Robertson and Jones, 1976) (which is a deter- 
  ministic method) to rank documents in terms 
  of relevance regarding each query. 
  
  • The top 10 documents that are not labeled as 
  
  relevant constitute the negative samples. 
  
  We recognize that this process leads to a high cor- 
  relation between the retrieval and reranking tasks. 
  We still think it is essential to make the latter avail- 
  able, with an open door to future improvement8. 
  • Variety: Diverse model types were included 
  to offer an insightful analysis across vari- 
  ous model characteristics (dimension, training 
  strategy, etc.). 
  
  In line with the third objective, we explicit below 
  the studied characteristics of embedding models 
  that will be discussed with the results. 
  
  7SummEvalFr available at: https://huggingface.co/ 
  datasets/lyon-nlp/summarization-summeval-fr-p2p 
  https: 
  8SyntecReranking 
  
  at: 
  //huggingface.co/datasets/lyon-nlp/ 
  mteb-fr-reranking-syntec-s2p 
  ing available at: 
  lyon-nlp/mteb-fr-reranking-alloprof-s2p 
  
  available 
  
  and AlloprofRerank- 
  https://huggingface.co/datasets/ 
  
  • Embedding dimension: This critical element 
  influences the expressiveness of the represen- 