text: lyMail dataset. Each article has 11 human-written 
  summaries and 16 machine-generated summaries 
  annotated by 8 people with a score for coherence, 
  consistency, fluency, and relevance. We trans- 
  lated it from English to French using DeepL API6. 
  Since MTEB evaluation is based on the embedding 
  similarity between machine-generated and human- 
  generated summaries, we propose to compute the 
  ROUGE (Lin, 2004) and BLEU (Papineni et al., 
  2002) metrics between machine and human sum- 
  maries for both French and English version. In Ta- 
  ble 2, we report the average of the scores as well as 
  their correlations between the two languages. The 
  correlation is high (above 0.7), showing that the 
  word and n-gram overlap between human and ma- 
  chine summaries is highly preserved in the French 
  version. One may argue that computing the met- 
  ric on fully translated texts (human and machine 
  summaries are both translated from English) may 
  introduce biases and not assess the quality of the 
  translations. For this purpose, we ensure the French 
  human summaries are correctly translated from En- 
  glish. We use an LLM as-a-judge (Zheng et al., 