text: We also studied the capabilities of models on the 
  French language when the language of the training 
  data varies. It is surprising to note the absence of a 
  clear correlation between the language the model 
  is trained on and its performance on French, as 
  shown by the large standard deviation in Figure 2. 
  Furthermore, monolingual models trained exclu- 
  sively on English such as voyage-code-2 show 
  very good results on French datasets compared 
  to models trained exclusively on French such as 
  flaubert derivatives and distilbert-base-fr-cased 
  (see Table D.1). 
  This is explained by the fact that a large part of the 
  selected French models generate embeddings using 
  a pooling strategy. Only a few are sentence trans- 
  former models, for which the pooled representation 
  is part of the model and trained with it, leading to 
  higher-quality embeddings. This is endorsed by 
  the excellent results of sentence-camembert-large, 
  a sentence transformer model trained on French 
  corpus and confirms the recent findings in terms of 
  model architecture (Gao et al., 2021). 
  Finally, it should be noted that a significant portion 
  of the French data used to train the selected French 
  models actually comes from English datasets 
  that have been machine translated (May, 2021). 
  Despite the tremendous progress of machine 
  translation, it is well known that the generated 
  data may be unrepresentative of the language 
  used by native speakers and cause a reduced final 
  performance (Barbosa et al., 2021). 