text: 3.3 Evaluation 
  
  • Model parameters: Often correlated with the 
  two first characteristics, parameter count is im- 
  portant for practical applications as it affects 
  usability on resource-efficient machines. The 
  selected models have a number of parameters 
  ranging from 20 million (∼100Mb in float32) 
  to 7 billion (∼28Gb). 
  
  For the sake of homogeneity, models are evalu- 
  ated using the same metrics per task as in MTEB 
  (Muennighoff et al., 2022): Classification (Accu- 
  racy), Bitext mining (F1 score), Pair classification 
  (AP), Clustering (V measure), Reranking (MAP), 
  Retrieval (NDCG@10), Summarization and STS 
  (Spearman correlation based on cosine similarity). 
  BitextMining tasks are excluded from the aver- 
  age performance scores and therefore the figures, 
  as this task evaluates 2 languages instead of one, 
  and this benchmark focuses only on one language 
  (French). We present the results for both DiaBlaBi- 
  textMining and FloresBitextMining in Table 12. 
  
  • Language: This is a major feature of lan- 
  guage models. Some are monolingual, and 
  others are multilingual. Language is usually 
  acquired during pre-training, but sometimes, 
  models familiarize themselves with new lan- 
  guages at tuning. For the benchmark, we 
  selected French models, as well as bilingual 
  or multilingual models. We also included a 
  few ones that claimed to be English (e.g. all- 
  MiniLM-L12-v29). 