
## Usage

For usage instructions, refer to: https://github.com/Muennighoff/sgpt#asymmetric-semantic-search-be

The model was trained with the command
```bash
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 accelerate launch examples/training/ms_marco/train_bi-encoder_mnrl.py --model_name bigscience/bloom-7b1 --train_batch_size 32 --eval_batch_size 16 --freezenonbias --specb --lr 4e-4 --wandb --wandbwatchlog gradients --pooling weightedmean --gradcache --chunksize 8
```

## Evaluation Results


`{"ndcgs": {"sgpt-bloom-7b1-msmarco": {"scifact": {"NDCG@10": 0.71824}, "nfcorpus": {"NDCG@10": 0.35748}, "arguana": {"NDCG@10": 0.47281}, "scidocs": {"NDCG@10": 0.18435}, "fiqa": {"NDCG@10": 0.35736}, "cqadupstack": {"NDCG@10": 0.3708525}, "quora": {"NDCG@10": 0.74655}, "trec-covid": {"NDCG@10": 0.82731}, "webis-touche2020": {"NDCG@10": 0.2365}}}`

See the evaluation folder or [MTEB](https://huggingface.co/spaces/mteb/leaderboard) for more results.

## Training
The model was trained with the parameters:

**DataLoader**:

`torch.utils.data.dataloader.DataLoader` of length 15600 with parameters:
```
{'batch_size': 32, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}
```

The model uses BitFit, weighted-mean pooling & GradCache, for details see: https://arxiv.org/abs/2202.08904

**Loss**:

`sentence_transformers.losses.MultipleNegativesRankingLoss.MNRLGradCache` 

Parameters of the fit()-Method:
```
{
    "epochs": 10,
    "evaluation_steps": 0,
    "evaluator": "NoneType",
    "max_grad_norm": 1,
    "optimizer_class": "<class 'transformers.optimization.AdamW'>",
    "optimizer_params": {
        "lr": 0.0004
    },
    "scheduler": "WarmupLinear",
    "steps_per_epoch": null,
    "warmup_steps": 1000,
    "weight_decay": 0.01
}
```


## Full Model Architecture
```
SentenceTransformer(
  (0): Transformer({'max_seq_length': 300, 'do_lower_case': False}) with Transformer model: BloomModel 
  (1): Pooling({'word_embedding_dimension': 4096, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': True, 'pooling_mode_lasttoken': False})
)
```

## Citing & Authors

```bibtex
@article{muennighoff2022sgpt,
  title={SGPT: GPT Sentence Embeddings for Semantic Search},
  author={Muennighoff, Niklas},
  journal={arXiv preprint arXiv:2202.08904},
  year={2022}
}
```
{"tags": ["sentence-transformers", "feature-extraction", "sentence-similarity", "mteb"], "pipeline_tag": "sentence-similarity", "model-index": [{"name": "sgpt-bloom-7b1-msmarco", "results": [{"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonCounterfactualClassification (en)", "type": "mteb/amazon_counterfactual", "config": "en", "split": "test", "revision": "2d8a100785abf0ae21420d2a55b0c56e3e1ea996"}, "metrics": [{"type": "accuracy", "value": 68.05970149253731}, {"type": "ap", "value": 31.640363460776193}, {"type": "f1", "value": 62.50025574145796}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonCounterfactualClassification (de)", "type": "mteb/amazon_counterfactual", "config": "de", "split": "test", "revision": "2d8a100785abf0ae21420d2a55b0c56e3e1ea996"}, "metrics": [{"type": "accuracy", "value": 61.34903640256959}, {"type": "ap", "value": 75.18797161500426}, {"type": "f1", "value": 59.04772570730417}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonCounterfactualClassification (en-ext)", "type": "mteb/amazon_counterfactual", "config": "en-ext", "split": "test", "revision": "2d8a100785abf0ae21420d2a55b0c56e3e1ea996"}, "metrics": [{"type": "accuracy", "value": 67.78110944527737}, {"type": "ap", "value": 19.218916023322706}, {"type": "f1", "value": 56.24477391445512}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonCounterfactualClassification (ja)", "type": "mteb/amazon_counterfactual", "config": "ja", "split": "test", "revision": "2d8a100785abf0ae21420d2a55b0c56e3e1ea996"}, "metrics": [{"type": "accuracy", "value": 58.23340471092078}, {"type": "ap", "value": 13.20222967424681}, {"type": "f1", "value": 47.511718095460296}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonPolarityClassification", "type": "mteb/amazon_polarity", "config": "default", "split": "test", "revision": "80714f8dcf8cefc218ef4f8c5a966dd83f75a0e1"}, "metrics": [{"type": "accuracy", "value": 68.97232499999998}, {"type": "ap", "value": 63.53632885535693}, {"type": "f1", "value": 68.62038513152868}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonReviewsClassification (en)", "type": "mteb/amazon_reviews_multi", "config": "en", "split": "test", "revision": "c379a6705fec24a2493fa68e011692605f44e119"}, "metrics": [{"type": "accuracy", "value": 33.855999999999995}, {"type": "f1", "value": 33.43468222830134}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonReviewsClassification (de)", "type": "mteb/amazon_reviews_multi", "config": "de", "split": "test", "revision": "c379a6705fec24a2493fa68e011692605f44e119"}, "metrics": [{"type": "accuracy", "value": 29.697999999999997}, {"type": "f1", "value": 29.39935388885501}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonReviewsClassification (es)", "type": "mteb/amazon_reviews_multi", "config": "es", "split": "test", "revision": "c379a6705fec24a2493fa68e011692605f44e119"}, "metrics": [{"type": "accuracy", "value": 35.974000000000004}, {"type": "f1", "value": 35.25910820714383}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonReviewsClassification (fr)", "type": "mteb/amazon_reviews_multi", "config": "fr", "split": "test", "revision": "c379a6705fec24a2493fa68e011692605f44e119"}, "metrics": [{"type": "accuracy", "value": 35.922}, {"type": "f1", "value": 35.38637028933444}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonReviewsClassification (ja)", "type": "mteb/amazon_reviews_multi", "config": "ja", "split": "test", "revision": "c379a6705fec24a2493fa68e011692605f44e119"}, "metrics": [{"type": "accuracy", "value": 27.636}, {"type": "f1", "value": 27.178349955978266}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonReviewsClassification (zh)", "type": "mteb/amazon_reviews_multi", "config": "zh", "split": "test", "revision": "c379a6705fec24a2493fa68e011692605f44e119"}, "metrics": [{"type": "accuracy", "value": 32.632}, {"type": "f1", "value": 32.08014766494587}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB ArguAna", "type": "arguana", "config": "default", "split": "test", "revision": "5b3e3697907184a9b77a3c99ee9ea1a9cbb1e4e3"}, "metrics": [{"type": "map_at_1", "value": 23.684}, {"type": "map_at_10", "value": 38.507999999999996}, {"type": "map_at_100", "value": 39.677}, {"type": "map_at_1000", "value": 39.690999999999995}, {"type": "map_at_3", "value": 33.369}, {"type": "map_at_5", "value": 36.15}, {"type": "mrr_at_1", "value": 24.04}, {"type": "mrr_at_10", "value": 38.664}, {"type": "mrr_at_100", "value": 39.833}, {"type": "mrr_at_1000", "value": 39.847}, {"type": "mrr_at_3", "value": 33.476}, {"type": "mrr_at_5", "value": 36.306}, {"type": "ndcg_at_1", "value": 23.684}, {"type": "ndcg_at_10", "value": 47.282000000000004}, {"type": "ndcg_at_100", "value": 52.215}, {"type": "ndcg_at_1000", "value": 52.551}, {"type": "ndcg_at_3", "value": 36.628}, {"type": "ndcg_at_5", "value": 41.653}, {"type": "precision_at_1", "value": 23.684}, {"type": "precision_at_10", "value": 7.553}, {"type": "precision_at_100", "value": 0.97}, {"type": "precision_at_1000", "value": 0.1}, {"type": "precision_at_3", "value": 15.363}, {"type": "precision_at_5", "value": 11.664}, {"type": "recall_at_1", "value": 23.684}, {"type": "recall_at_10", "value": 75.533}, {"type": "recall_at_100", "value": 97.013}, {"type": "recall_at_1000", "value": 99.57300000000001}, {"type": "recall_at_3", "value": 46.088}, {"type": "recall_at_5", "value": 58.321}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB ArxivClusteringP2P", "type": "mteb/arxiv-clustering-p2p", "config": "default", "split": "test", "revision": "0bbdb47bcbe3a90093699aefeed338a0f28a7ee8"}, "metrics": [{"type": "v_measure", "value": 44.59375023881131}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB ArxivClusteringS2S", "type": "mteb/arxiv-clustering-s2s", "config": "default", "split": "test", "revision": "b73bd54100e5abfa6e3a23dcafb46fe4d2438dc3"}, "metrics": [{"type": "v_measure", "value": 38.02921907752556}]}, {"task": {"type": "Reranking"}, "dataset": {"name": "MTEB AskUbuntuDupQuestions", "type": "mteb/askubuntudupquestions-reranking", "config": "default", "split": "test", "revision": "4d853f94cd57d85ec13805aeeac3ae3e5eb4c49c"}, "metrics": [{"type": "map", "value": 59.97321570342109}, {"type": "mrr", "value": 73.18284746955106}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB BIOSSES", "type": "mteb/biosses-sts", "config": "default", "split": "test", "revision": "9ee918f184421b6bd48b78f6c714d86546106103"}, "metrics": [{"type": "cos_sim_pearson", "value": 89.09091435741429}, {"type": "cos_sim_spearman", "value": 85.31459455332202}, {"type": "euclidean_pearson", "value": 79.3587681410798}, {"type": "euclidean_spearman", "value": 76.8174129874685}, {"type": "manhattan_pearson", "value": 79.57051762121769}, {"type": "manhattan_spearman", "value": 76.75837549768094}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB BUCC (de-en)", "type": "mteb/bucc-bitext-mining", "config": "de-en", "split": "test", "revision": "d51519689f32196a32af33b075a01d0e7c51e252"}, "metrics": [{"type": "accuracy", "value": 54.27974947807933}, {"type": "f1", "value": 54.00144411132214}, {"type": "precision", "value": 53.87119374071357}, {"type": "recall", "value": 54.27974947807933}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB BUCC (fr-en)", "type": "mteb/bucc-bitext-mining", "config": "fr-en", "split": "test", "revision": "d51519689f32196a32af33b075a01d0e7c51e252"}, "metrics": [{"type": "accuracy", "value": 97.3365617433414}, {"type": "f1", "value": 97.06141316310809}, {"type": "precision", "value": 96.92567319685965}, {"type": "recall", "value": 97.3365617433414}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB BUCC (ru-en)", "type": "mteb/bucc-bitext-mining", "config": "ru-en", "split": "test", "revision": "d51519689f32196a32af33b075a01d0e7c51e252"}, "metrics": [{"type": "accuracy", "value": 46.05472809144441}, {"type": "f1", "value": 45.30319274690595}, {"type": "precision", "value": 45.00015469655234}, {"type": "recall", "value": 46.05472809144441}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB BUCC (zh-en)", "type": "mteb/bucc-bitext-mining", "config": "zh-en", "split": "test", "revision": "d51519689f32196a32af33b075a01d0e7c51e252"}, "metrics": [{"type": "accuracy", "value": 98.10426540284361}, {"type": "f1", "value": 97.96384061786905}, {"type": "precision", "value": 97.89362822538178}, {"type": "recall", "value": 98.10426540284361}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB Banking77Classification", "type": "mteb/banking77", "config": "default", "split": "test", "revision": "44fa15921b4c889113cc5df03dd4901b49161ab7"}, "metrics": [{"type": "accuracy", "value": 84.33441558441558}, {"type": "f1", "value": 84.31653077470322}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB BiorxivClusteringP2P", "type": "mteb/biorxiv-clustering-p2p", "config": "default", "split": "test", "revision": "11d0121201d1f1f280e8cc8f3d98fb9c4d9f9c55"}, "metrics": [{"type": "v_measure", "value": 36.025318694698086}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB BiorxivClusteringS2S", "type": "mteb/biorxiv-clustering-s2s", "config": "default", "split": "test", "revision": "c0fab014e1bcb8d3a5e31b2088972a1e01547dc1"}, "metrics": [{"type": "v_measure", "value": 32.484889034590346}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB CQADupstackAndroidRetrieval", "type": "BeIR/cqadupstack", "config": "default", "split": "test", "revision": "2b9f5791698b5be7bc5e10535c8690f20043c3db"}, "metrics": [{"type": "map_at_1", "value": 30.203999999999997}, {"type": "map_at_10", "value": 41.314}, {"type": "map_at_100", "value": 42.66}, {"type": "map_at_1000", "value": 42.775999999999996}, {"type": "map_at_3", "value": 37.614999999999995}, {"type": "map_at_5", "value": 39.643}, {"type": "mrr_at_1", "value": 37.482}, {"type": "mrr_at_10", "value": 47.075}, {"type": "mrr_at_100", "value": 47.845}, {"type": "mrr_at_1000", "value": 47.887}, {"type": "mrr_at_3", "value": 44.635000000000005}, {"type": "mrr_at_5", "value": 45.966}, {"type": "ndcg_at_1", "value": 37.482}, {"type": "ndcg_at_10", "value": 47.676}, {"type": "ndcg_at_100", "value": 52.915}, {"type": "ndcg_at_1000", "value": 54.82900000000001}, {"type": "ndcg_at_3", "value": 42.562}, {"type": "ndcg_at_5", "value": 44.852}, {"type": "precision_at_1", "value": 37.482}, {"type": "precision_at_10", "value": 9.142}, {"type": "precision_at_100", "value": 1.436}, {"type": "precision_at_1000", "value": 0.189}, {"type": "precision_at_3", "value": 20.458000000000002}, {"type": "precision_at_5", "value": 14.821000000000002}, {"type": "recall_at_1", "value": 30.203999999999997}, {"type": "recall_at_10", "value": 60.343}, {"type": "recall_at_100", "value": 82.58}, {"type": "recall_at_1000", "value": 94.813}, {"type": "recall_at_3", "value": 45.389}, {"type": "recall_at_5", "value": 51.800999999999995}, {"type": "map_at_1", "value": 30.889}, {"type": "map_at_10", "value": 40.949999999999996}, {"type": "map_at_100", "value": 42.131}, {"type": "map_at_1000", "value": 42.253}, {"type": "map_at_3", "value": 38.346999999999994}, {"type": "map_at_5", "value": 39.782000000000004}, {"type": "mrr_at_1", "value": 38.79}, {"type": "mrr_at_10", "value": 46.944}, {"type": "mrr_at_100", "value": 47.61}, {"type": "mrr_at_1000", "value": 47.650999999999996}, {"type": "mrr_at_3", "value": 45.053}, {"type": "mrr_at_5", "value": 46.101}, {"type": "ndcg_at_1", "value": 38.79}, {"type": "ndcg_at_10", "value": 46.286}, {"type": "ndcg_at_100", "value": 50.637}, {"type": "ndcg_at_1000", "value": 52.649}, {"type": "ndcg_at_3", "value": 42.851}, {"type": "ndcg_at_5", "value": 44.311}, {"type": "precision_at_1", "value": 38.79}, {"type": "precision_at_10", "value": 8.516}, {"type": "precision_at_100", "value": 1.3679999999999999}, {"type": "precision_at_1000", "value": 0.183}, {"type": "precision_at_3", "value": 20.637}, {"type": "precision_at_5", "value": 14.318}, {"type": "recall_at_1", "value": 30.889}, {"type": "recall_at_10", "value": 55.327000000000005}, {"type": "recall_at_100", "value": 74.091}, {"type": "recall_at_1000", "value": 86.75500000000001}, {"type": "recall_at_3", "value": 44.557}, {"type": "recall_at_5", "value": 49.064}, {"type": "map_at_1", "value": 39.105000000000004}, {"type": "map_at_10", "value": 50.928}, {"type": "map_at_100", "value": 51.958000000000006}, {"type": "map_at_1000", "value": 52.017}, {"type": "map_at_3", "value": 47.638999999999996}, {"type": "map_at_5", "value": 49.624}, {"type": "mrr_at_1", "value": 44.639}, {"type": "mrr_at_10", "value": 54.261}, {"type": "mrr_at_100", "value": 54.913999999999994}, {"type": "mrr_at_1000", "value": 54.945}, {"type": "mrr_at_3", "value": 51.681999999999995}, {"type": "mrr_at_5", "value": 53.290000000000006}, {"type": "ndcg_at_1", "value": 44.639}, {"type": "ndcg_at_10", "value": 56.678}, {"type": "ndcg_at_100", "value": 60.649}, {"type": "ndcg_at_1000", "value": 61.855000000000004}, {"type": "ndcg_at_3", "value": 51.092999999999996}, {"type": "ndcg_at_5", "value": 54.096999999999994}, {"type": "precision_at_1", "value": 44.639}, {"type": "precision_at_10", "value": 9.028}, {"type": "precision_at_100", "value": 1.194}, {"type": "precision_at_1000", "value": 0.135}, {"type": "precision_at_3", "value": 22.508}, {"type": "precision_at_5", "value": 15.661}, {"type": "recall_at_1", "value": 39.105000000000004}, {"type": "recall_at_10", "value": 70.367}, {"type": "recall_at_100", "value": 87.359}, {"type": "recall_at_1000", "value": 95.88}, {"type": "recall_at_3", "value": 55.581}, {"type": "recall_at_5", "value": 62.821000000000005}, {"type": "map_at_1", "value": 23.777}, {"type": "map_at_10", "value": 32.297}, {"type": "map_at_100", "value": 33.516}, {"type": "map_at_1000", "value": 33.592}, {"type": "map_at_3", "value": 30.001}, {"type": "map_at_5", "value": 31.209999999999997}, {"type": "mrr_at_1", "value": 25.989}, {"type": "mrr_at_10", "value": 34.472}, {"type": "mrr_at_100", "value": 35.518}, {"type": "mrr_at_1000", "value": 35.577}, {"type": "mrr_at_3", "value": 32.185}, {"type": "mrr_at_5", "value": 33.399}, {"type": "ndcg_at_1", "value": 25.989}, {"type": "ndcg_at_10", "value": 37.037}, {"type": "ndcg_at_100", "value": 42.699}, {"type": "ndcg_at_1000", "value": 44.725}, {"type": "ndcg_at_3", "value": 32.485}, {"type": "ndcg_at_5", "value": 34.549}, {"type": "precision_at_1", "value": 25.989}, {"type": "precision_at_10", "value": 5.718}, {"type": "precision_at_100", "value": 0.89}, {"type": "precision_at_1000", "value": 0.11}, {"type": "precision_at_3", "value": 14.049}, {"type": "precision_at_5", "value": 9.672}, {"type": "recall_at_1", "value": 23.777}, {"type": "recall_at_10", "value": 49.472}, {"type": "recall_at_100", "value": 74.857}, {"type": "recall_at_1000", "value": 90.289}, {"type": "recall_at_3", "value": 37.086000000000006}, {"type": "recall_at_5", "value": 42.065999999999995}, {"type": "map_at_1", "value": 13.377}, {"type": "map_at_10", "value": 21.444}, {"type": "map_at_100", "value": 22.663}, {"type": "map_at_1000", "value": 22.8}, {"type": "map_at_3", "value": 18.857}, {"type": "map_at_5", "value": 20.426}, {"type": "mrr_at_1", "value": 16.542}, {"type": "mrr_at_10", "value": 25.326999999999998}, {"type": "mrr_at_100", "value": 26.323}, {"type": "mrr_at_1000", "value": 26.406000000000002}, {"type": "mrr_at_3", "value": 22.823}, {"type": "mrr_at_5", "value": 24.340999999999998}, {"type": "ndcg_at_1", "value": 16.542}, {"type": "ndcg_at_10", "value": 26.479000000000003}, {"type": "ndcg_at_100", "value": 32.29}, {"type": "ndcg_at_1000", "value": 35.504999999999995}, {"type": "ndcg_at_3", "value": 21.619}, {"type": "ndcg_at_5", "value": 24.19}, {"type": "precision_at_1", "value": 16.542}, {"type": "precision_at_10", "value": 5.075}, {"type": "precision_at_100", "value": 0.9339999999999999}, {"type": "precision_at_1000", "value": 0.135}, {"type": "precision_at_3", "value": 10.697}, {"type": "precision_at_5", "value": 8.134}, {"type": "recall_at_1", "value": 13.377}, {"type": "recall_at_10", "value": 38.027}, {"type": "recall_at_100", "value": 63.439}, {"type": "recall_at_1000", "value": 86.354}, {"type": "recall_at_3", "value": 25.0}, {"type": "recall_at_5", "value": 31.306}, {"type": "map_at_1", "value": 28.368}, {"type": "map_at_10", "value": 39.305}, {"type": "map_at_100", "value": 40.637}, {"type": "map_at_1000", "value": 40.753}, {"type": "map_at_3", "value": 36.077999999999996}, {"type": "map_at_5", "value": 37.829}, {"type": "mrr_at_1", "value": 34.937000000000005}, {"type": "mrr_at_10", "value": 45.03}, {"type": "mrr_at_100", "value": 45.78}, {"type": "mrr_at_1000", "value": 45.827}, {"type": "mrr_at_3", "value": 42.348}, {"type": "mrr_at_5", "value": 43.807}, {"type": "ndcg_at_1", "value": 34.937000000000005}, {"type": "ndcg_at_10", "value": 45.605000000000004}, {"type": "ndcg_at_100", "value": 50.941}, {"type": "ndcg_at_1000", "value": 52.983000000000004}, {"type": "ndcg_at_3", "value": 40.366}, {"type": "ndcg_at_5", "value": 42.759}, {"type": "precision_at_1", "value": 34.937000000000005}, {"type": "precision_at_10", "value": 8.402}, {"type": "precision_at_100", "value": 1.2959999999999998}, {"type": "precision_at_1000", "value": 0.164}, {"type": "precision_at_3", "value": 19.217000000000002}, {"type": "precision_at_5", "value": 13.725000000000001}, {"type": "recall_at_1", "value": 28.368}, {"type": "recall_at_10", "value": 58.5}, {"type": "recall_at_100", "value": 80.67999999999999}, {"type": "recall_at_1000", "value": 93.925}, {"type": "recall_at_3", "value": 43.956}, {"type": "recall_at_5", "value": 50.065000000000005}, {"type": "map_at_1", "value": 24.851}, {"type": "map_at_10", "value": 34.758}, {"type": "map_at_100", "value": 36.081}, {"type": "map_at_1000", "value": 36.205999999999996}, {"type": "map_at_3", "value": 31.678}, {"type": "map_at_5", "value": 33.398}, {"type": "mrr_at_1", "value": 31.279}, {"type": "mrr_at_10", "value": 40.138}, {"type": "mrr_at_100", "value": 41.005}, {"type": "mrr_at_1000", "value": 41.065000000000005}, {"type": "mrr_at_3", "value": 37.519000000000005}, {"type": "mrr_at_5", "value": 38.986}, {"type": "ndcg_at_1", "value": 31.279}, {"type": "ndcg_at_10", "value": 40.534}, {"type": "ndcg_at_100", "value": 46.093}, {"type": "ndcg_at_1000", "value": 48.59}, {"type": "ndcg_at_3", "value": 35.473}, {"type": "ndcg_at_5", "value": 37.801}, {"type": "precision_at_1", "value": 31.279}, {"type": "precision_at_10", "value": 7.477}, {"type": "precision_at_100", "value": 1.2}, {"type": "precision_at_1000", "value": 0.159}, {"type": "precision_at_3", "value": 17.047}, {"type": "precision_at_5", "value": 12.306000000000001}, {"type": "recall_at_1", "value": 24.851}, {"type": "recall_at_10", "value": 52.528}, {"type": "recall_at_100", "value": 76.198}, {"type": "recall_at_1000", "value": 93.12}, {"type": "recall_at_3", "value": 38.257999999999996}, {"type": "recall_at_5", "value": 44.440000000000005}, {"type": "map_at_1", "value": 25.289833333333334}, {"type": "map_at_10", "value": 34.379333333333335}, {"type": "map_at_100", "value": 35.56916666666666}, {"type": "map_at_1000", "value": 35.68633333333333}, {"type": "map_at_3", "value": 31.63916666666666}, {"type": "map_at_5", "value": 33.18383333333334}, {"type": "mrr_at_1", "value": 30.081749999999996}, {"type": "mrr_at_10", "value": 38.53658333333333}, {"type": "mrr_at_100", "value": 39.37825}, {"type": "mrr_at_1000", "value": 39.43866666666666}, {"type": "mrr_at_3", "value": 36.19025}, {"type": "mrr_at_5", "value": 37.519749999999995}, {"type": "ndcg_at_1", "value": 30.081749999999996}, {"type": "ndcg_at_10", "value": 39.62041666666667}, {"type": "ndcg_at_100", "value": 44.74825}, {"type": "ndcg_at_1000", "value": 47.11366666666667}, {"type": "ndcg_at_3", "value": 35.000499999999995}, {"type": "ndcg_at_5", "value": 37.19283333333333}, {"type": "precision_at_1", "value": 30.081749999999996}, {"type": "precision_at_10", "value": 6.940249999999999}, {"type": "precision_at_100", "value": 1.1164166666666668}, {"type": "precision_at_1000", "value": 0.15025000000000002}, {"type": "precision_at_3", "value": 16.110416666666666}, {"type": "precision_at_5", "value": 11.474416666666668}, {"type": "recall_at_1", "value": 25.289833333333334}, {"type": "recall_at_10", "value": 51.01591666666667}, {"type": "recall_at_100", "value": 73.55275000000002}, {"type": "recall_at_1000", "value": 90.02666666666667}, {"type": "recall_at_3", "value": 38.15208333333334}, {"type": "recall_at_5", "value": 43.78458333333334}, {"type": "map_at_1", "value": 23.479}, {"type": "map_at_10", "value": 31.2}, {"type": "map_at_100", "value": 32.11}, {"type": "map_at_1000", "value": 32.214}, {"type": "map_at_3", "value": 29.093999999999998}, {"type": "map_at_5", "value": 30.415}, {"type": "mrr_at_1", "value": 26.840000000000003}, {"type": "mrr_at_10", "value": 34.153}, {"type": "mrr_at_100", "value": 34.971000000000004}, {"type": "mrr_at_1000", "value": 35.047}, {"type": "mrr_at_3", "value": 32.285000000000004}, {"type": "mrr_at_5", "value": 33.443}, {"type": "ndcg_at_1", "value": 26.840000000000003}, {"type": "ndcg_at_10", "value": 35.441}, {"type": "ndcg_at_100", "value": 40.150000000000006}, {"type": "ndcg_at_1000", "value": 42.74}, {"type": "ndcg_at_3", "value": 31.723000000000003}, {"type": "ndcg_at_5", "value": 33.71}, {"type": "precision_at_1", "value": 26.840000000000003}, {"type": "precision_at_10", "value": 5.552}, {"type": "precision_at_100", "value": 0.859}, {"type": "precision_at_1000", "value": 0.11499999999999999}, {"type": "precision_at_3", "value": 13.804}, {"type": "precision_at_5", "value": 9.600999999999999}, {"type": "recall_at_1", "value": 23.479}, {"type": "recall_at_10", "value": 45.442}, {"type": "recall_at_100", "value": 67.465}, {"type": "recall_at_1000", "value": 86.53}, {"type": "recall_at_3", "value": 35.315999999999995}, {"type": "recall_at_5", "value": 40.253}, {"type": "map_at_1", "value": 16.887}, {"type": "map_at_10", "value": 23.805}, {"type": "map_at_100", "value": 24.804000000000002}, {"type": "map_at_1000", "value": 24.932000000000002}, {"type": "map_at_3", "value": 21.632}, {"type": "map_at_5", "value": 22.845}, {"type": "mrr_at_1", "value": 20.75}, {"type": "mrr_at_10", "value": 27.686}, {"type": "mrr_at_100", "value": 28.522}, {"type": "mrr_at_1000", "value": 28.605000000000004}, {"type": "mrr_at_3", "value": 25.618999999999996}, {"type": "mrr_at_5", "value": 26.723999999999997}, {"type": "ndcg_at_1", "value": 20.75}, {"type": "ndcg_at_10", "value": 28.233000000000004}, {"type": "ndcg_at_100", "value": 33.065}, {"type": "ndcg_at_1000", "value": 36.138999999999996}, {"type": "ndcg_at_3", "value": 24.361}, {"type": "ndcg_at_5", "value": 26.111}, {"type": "precision_at_1", "value": 20.75}, {"type": "precision_at_10", "value": 5.124}, {"type": "precision_at_100", "value": 0.8750000000000001}, {"type": "precision_at_1000", "value": 0.131}, {"type": "precision_at_3", "value": 11.539000000000001}, {"type": "precision_at_5", "value": 8.273}, {"type": "recall_at_1", "value": 16.887}, {"type": "recall_at_10", "value": 37.774}, {"type": "recall_at_100", "value": 59.587}, {"type": "recall_at_1000", "value": 81.523}, {"type": "recall_at_3", "value": 26.837}, {"type": "recall_at_5", "value": 31.456}, {"type": "map_at_1", "value": 25.534000000000002}, {"type": "map_at_10", "value": 33.495999999999995}, {"type": "map_at_100", "value": 34.697}, {"type": "map_at_1000", "value": 34.805}, {"type": "map_at_3", "value": 31.22}, {"type": "map_at_5", "value": 32.277}, {"type": "mrr_at_1", "value": 29.944}, {"type": "mrr_at_10", "value": 37.723}, {"type": "mrr_at_100", "value": 38.645}, {"type": "mrr_at_1000", "value": 38.712999999999994}, {"type": "mrr_at_3", "value": 35.665}, {"type": "mrr_at_5", "value": 36.681999999999995}, {"type": "ndcg_at_1", "value": 29.944}, {"type": "ndcg_at_10", "value": 38.407000000000004}, {"type": "ndcg_at_100", "value": 43.877}, {"type": "ndcg_at_1000", "value": 46.312}, {"type": "ndcg_at_3", "value": 34.211000000000006}, {"type": "ndcg_at_5", "value": 35.760999999999996}, {"type": "precision_at_1", "value": 29.944}, {"type": "precision_at_10", "value": 6.343}, {"type": "precision_at_100", "value": 1.023}, {"type": "precision_at_1000", "value": 0.133}, {"type": "precision_at_3", "value": 15.360999999999999}, {"type": "precision_at_5", "value": 10.428999999999998}, {"type": "recall_at_1", "value": 25.534000000000002}, {"type": "recall_at_10", "value": 49.204}, {"type": "recall_at_100", "value": 72.878}, {"type": "recall_at_1000", "value": 89.95}, {"type": "recall_at_3", "value": 37.533}, {"type": "recall_at_5", "value": 41.611}, {"type": "map_at_1", "value": 26.291999999999998}, {"type": "map_at_10", "value": 35.245}, {"type": "map_at_100", "value": 36.762}, {"type": "map_at_1000", "value": 36.983}, {"type": "map_at_3", "value": 32.439}, {"type": "map_at_5", "value": 33.964}, {"type": "mrr_at_1", "value": 31.423000000000002}, {"type": "mrr_at_10", "value": 39.98}, {"type": "mrr_at_100", "value": 40.791}, {"type": "mrr_at_1000", "value": 40.854}, {"type": "mrr_at_3", "value": 37.451}, {"type": "mrr_at_5", "value": 38.854}, {"type": "ndcg_at_1", "value": 31.423000000000002}, {"type": "ndcg_at_10", "value": 40.848}, {"type": "ndcg_at_100", "value": 46.35}, {"type": "ndcg_at_1000", "value": 49.166}, {"type": "ndcg_at_3", "value": 36.344}, {"type": "ndcg_at_5", "value": 38.36}, {"type": "precision_at_1", "value": 31.423000000000002}, {"type": "precision_at_10", "value": 7.767}, {"type": "precision_at_100", "value": 1.498}, {"type": "precision_at_1000", "value": 0.23700000000000002}, {"type": "precision_at_3", "value": 16.733}, {"type": "precision_at_5", "value": 12.213000000000001}, {"type": "recall_at_1", "value": 26.291999999999998}, {"type": "recall_at_10", "value": 51.184}, {"type": "recall_at_100", "value": 76.041}, {"type": "recall_at_1000", "value": 94.11500000000001}, {"type": "recall_at_3", "value": 38.257000000000005}, {"type": "recall_at_5", "value": 43.68}, {"type": "map_at_1", "value": 20.715}, {"type": "map_at_10", "value": 27.810000000000002}, {"type": "map_at_100", "value": 28.810999999999996}, {"type": "map_at_1000", "value": 28.904999999999998}, {"type": "map_at_3", "value": 25.069999999999997}, {"type": "map_at_5", "value": 26.793}, {"type": "mrr_at_1", "value": 22.366}, {"type": "mrr_at_10", "value": 29.65}, {"type": "mrr_at_100", "value": 30.615}, {"type": "mrr_at_1000", "value": 30.686999999999998}, {"type": "mrr_at_3", "value": 27.017999999999997}, {"type": "mrr_at_5", "value": 28.644}, {"type": "ndcg_at_1", "value": 22.366}, {"type": "ndcg_at_10", "value": 32.221}, {"type": "ndcg_at_100", "value": 37.313}, {"type": "ndcg_at_1000", "value": 39.871}, {"type": "ndcg_at_3", "value": 26.918}, {"type": "ndcg_at_5", "value": 29.813000000000002}, {"type": "precision_at_1", "value": 22.366}, {"type": "precision_at_10", "value": 5.139}, {"type": "precision_at_100", "value": 0.8240000000000001}, {"type": "precision_at_1000", "value": 0.11199999999999999}, {"type": "precision_at_3", "value": 11.275}, {"type": "precision_at_5", "value": 8.540000000000001}, {"type": "recall_at_1", "value": 20.715}, {"type": "recall_at_10", "value": 44.023}, {"type": "recall_at_100", "value": 67.458}, {"type": "recall_at_1000", "value": 87.066}, {"type": "recall_at_3", "value": 30.055}, {"type": "recall_at_5", "value": 36.852000000000004}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB ClimateFEVER", "type": "climate-fever", "config": "default", "split": "test", "revision": "392b78eb68c07badcd7c2cd8f39af108375dfcce"}, "metrics": [{"type": "map_at_1", "value": 11.859}, {"type": "map_at_10", "value": 20.625}, {"type": "map_at_100", "value": 22.5}, {"type": "map_at_1000", "value": 22.689}, {"type": "map_at_3", "value": 16.991}, {"type": "map_at_5", "value": 18.781}, {"type": "mrr_at_1", "value": 26.906000000000002}, {"type": "mrr_at_10", "value": 39.083}, {"type": "mrr_at_100", "value": 39.978}, {"type": "mrr_at_1000", "value": 40.014}, {"type": "mrr_at_3", "value": 35.44}, {"type": "mrr_at_5", "value": 37.619}, {"type": "ndcg_at_1", "value": 26.906000000000002}, {"type": "ndcg_at_10", "value": 29.386000000000003}, {"type": "ndcg_at_100", "value": 36.510999999999996}, {"type": "ndcg_at_1000", "value": 39.814}, {"type": "ndcg_at_3", "value": 23.558}, {"type": "ndcg_at_5", "value": 25.557999999999996}, {"type": "precision_at_1", "value": 26.906000000000002}, {"type": "precision_at_10", "value": 9.342}, {"type": "precision_at_100", "value": 1.6969999999999998}, {"type": "precision_at_1000", "value": 0.231}, {"type": "precision_at_3", "value": 17.503}, {"type": "precision_at_5", "value": 13.655000000000001}, {"type": "recall_at_1", "value": 11.859}, {"type": "recall_at_10", "value": 35.929}, {"type": "recall_at_100", "value": 60.21300000000001}, {"type": "recall_at_1000", "value": 78.606}, {"type": "recall_at_3", "value": 21.727}, {"type": "recall_at_5", "value": 27.349}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB DBPedia", "type": "dbpedia-entity", "config": "default", "split": "test", "revision": "f097057d03ed98220bc7309ddb10b71a54d667d6"}, "metrics": [{"type": "map_at_1", "value": 8.627}, {"type": "map_at_10", "value": 18.248}, {"type": "map_at_100", "value": 25.19}, {"type": "map_at_1000", "value": 26.741}, {"type": "map_at_3", "value": 13.286000000000001}, {"type": "map_at_5", "value": 15.126000000000001}, {"type": "mrr_at_1", "value": 64.75}, {"type": "mrr_at_10", "value": 71.865}, {"type": "mrr_at_100", "value": 72.247}, {"type": "mrr_at_1000", "value": 72.255}, {"type": "mrr_at_3", "value": 69.958}, {"type": "mrr_at_5", "value": 71.108}, {"type": "ndcg_at_1", "value": 53.25}, {"type": "ndcg_at_10", "value": 39.035}, {"type": "ndcg_at_100", "value": 42.735}, {"type": "ndcg_at_1000", "value": 50.166}, {"type": "ndcg_at_3", "value": 43.857}, {"type": "ndcg_at_5", "value": 40.579}, {"type": "precision_at_1", "value": 64.75}, {"type": "precision_at_10", "value": 30.75}, {"type": "precision_at_100", "value": 9.54}, {"type": "precision_at_1000", "value": 2.035}, {"type": "precision_at_3", "value": 47.333}, {"type": "precision_at_5", "value": 39.0}, {"type": "recall_at_1", "value": 8.627}, {"type": "recall_at_10", "value": 23.413}, {"type": "recall_at_100", "value": 48.037}, {"type": "recall_at_1000", "value": 71.428}, {"type": "recall_at_3", "value": 14.158999999999999}, {"type": "recall_at_5", "value": 17.002}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB EmotionClassification", "type": "mteb/emotion", "config": "default", "split": "test", "revision": "829147f8f75a25f005913200eb5ed41fae320aa1"}, "metrics": [{"type": "accuracy", "value": 44.865}, {"type": "f1", "value": 41.56625743266997}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB FEVER", "type": "fever", "config": "default", "split": "test", "revision": "1429cf27e393599b8b359b9b72c666f96b2525f9"}, "metrics": [{"type": "map_at_1", "value": 57.335}, {"type": "map_at_10", "value": 68.29499999999999}, {"type": "map_at_100", "value": 68.69800000000001}, {"type": "map_at_1000", "value": 68.714}, {"type": "map_at_3", "value": 66.149}, {"type": "map_at_5", "value": 67.539}, {"type": "mrr_at_1", "value": 61.656}, {"type": "mrr_at_10", "value": 72.609}, {"type": "mrr_at_100", "value": 72.923}, {"type": "mrr_at_1000", "value": 72.928}, {"type": "mrr_at_3", "value": 70.645}, {"type": "mrr_at_5", "value": 71.938}, {"type": "ndcg_at_1", "value": 61.656}, {"type": "ndcg_at_10", "value": 73.966}, {"type": "ndcg_at_100", "value": 75.663}, {"type": "ndcg_at_1000", "value": 75.986}, {"type": "ndcg_at_3", "value": 69.959}, {"type": "ndcg_at_5", "value": 72.269}, {"type": "precision_at_1", "value": 61.656}, {"type": "precision_at_10", "value": 9.581000000000001}, {"type": "precision_at_100", "value": 1.054}, {"type": "precision_at_1000", "value": 0.11}, {"type": "precision_at_3", "value": 27.743000000000002}, {"type": "precision_at_5", "value": 17.939}, {"type": "recall_at_1", "value": 57.335}, {"type": "recall_at_10", "value": 87.24300000000001}, {"type": "recall_at_100", "value": 94.575}, {"type": "recall_at_1000", "value": 96.75399999999999}, {"type": "recall_at_3", "value": 76.44800000000001}, {"type": "recall_at_5", "value": 82.122}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB FiQA2018", "type": "fiqa", "config": "default", "split": "test", "revision": "41b686a7f28c59bcaaa5791efd47c67c8ebe28be"}, "metrics": [{"type": "map_at_1", "value": 17.014000000000003}, {"type": "map_at_10", "value": 28.469}, {"type": "map_at_100", "value": 30.178}, {"type": "map_at_1000", "value": 30.369}, {"type": "map_at_3", "value": 24.63}, {"type": "map_at_5", "value": 26.891}, {"type": "mrr_at_1", "value": 34.259}, {"type": "mrr_at_10", "value": 43.042}, {"type": "mrr_at_100", "value": 43.91}, {"type": "mrr_at_1000", "value": 43.963}, {"type": "mrr_at_3", "value": 40.483999999999995}, {"type": "mrr_at_5", "value": 42.135}, {"type": "ndcg_at_1", "value": 34.259}, {"type": "ndcg_at_10", "value": 35.836}, {"type": "ndcg_at_100", "value": 42.488}, {"type": "ndcg_at_1000", "value": 45.902}, {"type": "ndcg_at_3", "value": 32.131}, {"type": "ndcg_at_5", "value": 33.697}, {"type": "precision_at_1", "value": 34.259}, {"type": "precision_at_10", "value": 10.0}, {"type": "precision_at_100", "value": 1.699}, {"type": "precision_at_1000", "value": 0.22999999999999998}, {"type": "precision_at_3", "value": 21.502}, {"type": "precision_at_5", "value": 16.296}, {"type": "recall_at_1", "value": 17.014000000000003}, {"type": "recall_at_10", "value": 42.832}, {"type": "recall_at_100", "value": 67.619}, {"type": "recall_at_1000", "value": 88.453}, {"type": "recall_at_3", "value": 29.537000000000003}, {"type": "recall_at_5", "value": 35.886}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB HotpotQA", "type": "hotpotqa", "config": "default", "split": "test", "revision": "766870b35a1b9ca65e67a0d1913899973551fc6c"}, "metrics": [{"type": "map_at_1", "value": 34.558}, {"type": "map_at_10", "value": 48.039}, {"type": "map_at_100", "value": 48.867}, {"type": "map_at_1000", "value": 48.941}, {"type": "map_at_3", "value": 45.403}, {"type": "map_at_5", "value": 46.983999999999995}, {"type": "mrr_at_1", "value": 69.11500000000001}, {"type": "mrr_at_10", "value": 75.551}, {"type": "mrr_at_100", "value": 75.872}, {"type": "mrr_at_1000", "value": 75.887}, {"type": "mrr_at_3", "value": 74.447}, {"type": "mrr_at_5", "value": 75.113}, {"type": "ndcg_at_1", "value": 69.11500000000001}, {"type": "ndcg_at_10", "value": 57.25599999999999}, {"type": "ndcg_at_100", "value": 60.417}, {"type": "ndcg_at_1000", "value": 61.976}, {"type": "ndcg_at_3", "value": 53.258}, {"type": "ndcg_at_5", "value": 55.374}, {"type": "precision_at_1", "value": 69.11500000000001}, {"type": "precision_at_10", "value": 11.689}, {"type": "precision_at_100", "value": 1.418}, {"type": "precision_at_1000", "value": 0.163}, {"type": "precision_at_3", "value": 33.018}, {"type": "precision_at_5", "value": 21.488}, {"type": "recall_at_1", "value": 34.558}, {"type": "recall_at_10", "value": 58.447}, {"type": "recall_at_100", "value": 70.91199999999999}, {"type": "recall_at_1000", "value": 81.31}, {"type": "recall_at_3", "value": 49.527}, {"type": "recall_at_5", "value": 53.72}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB ImdbClassification", "type": "mteb/imdb", "config": "default", "split": "test", "revision": "8d743909f834c38949e8323a8a6ce8721ea6c7f4"}, "metrics": [{"type": "accuracy", "value": 61.772000000000006}, {"type": "ap", "value": 57.48217702943605}, {"type": "f1", "value": 61.20495351356274}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB MSMARCO", "type": "msmarco", "config": "default", "split": "validation", "revision": "e6838a846e2408f22cf5cc337ebc83e0bcf77849"}, "metrics": [{"type": "map_at_1", "value": 22.044}, {"type": "map_at_10", "value": 34.211000000000006}, {"type": "map_at_100", "value": 35.394}, {"type": "map_at_1000", "value": 35.443000000000005}, {"type": "map_at_3", "value": 30.318}, {"type": "map_at_5", "value": 32.535}, {"type": "mrr_at_1", "value": 22.722}, {"type": "mrr_at_10", "value": 34.842}, {"type": "mrr_at_100", "value": 35.954}, {"type": "mrr_at_1000", "value": 35.997}, {"type": "mrr_at_3", "value": 30.991000000000003}, {"type": "mrr_at_5", "value": 33.2}, {"type": "ndcg_at_1", "value": 22.722}, {"type": "ndcg_at_10", "value": 41.121}, {"type": "ndcg_at_100", "value": 46.841}, {"type": "ndcg_at_1000", "value": 48.049}, {"type": "ndcg_at_3", "value": 33.173}, {"type": "ndcg_at_5", "value": 37.145}, {"type": "precision_at_1", "value": 22.722}, {"type": "precision_at_10", "value": 6.516}, {"type": "precision_at_100", "value": 0.9400000000000001}, {"type": "precision_at_1000", "value": 0.104}, {"type": "precision_at_3", "value": 14.093}, {"type": "precision_at_5", "value": 10.473}, {"type": "recall_at_1", "value": 22.044}, {"type": "recall_at_10", "value": 62.382000000000005}, {"type": "recall_at_100", "value": 88.914}, {"type": "recall_at_1000", "value": 98.099}, {"type": "recall_at_3", "value": 40.782000000000004}, {"type": "recall_at_5", "value": 50.322}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPDomainClassification (en)", "type": "mteb/mtop_domain", "config": "en", "split": "test", "revision": "a7e2a951126a26fc8c6a69f835f33a346ba259e3"}, "metrics": [{"type": "accuracy", "value": 93.68217054263563}, {"type": "f1", "value": 93.25810075739523}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPDomainClassification (de)", "type": "mteb/mtop_domain", "config": "de", "split": "test", "revision": "a7e2a951126a26fc8c6a69f835f33a346ba259e3"}, "metrics": [{"type": "accuracy", "value": 82.05409974640745}, {"type": "f1", "value": 80.42814140324903}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPDomainClassification (es)", "type": "mteb/mtop_domain", "config": "es", "split": "test", "revision": "a7e2a951126a26fc8c6a69f835f33a346ba259e3"}, "metrics": [{"type": "accuracy", "value": 93.54903268845896}, {"type": "f1", "value": 92.8909878077932}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPDomainClassification (fr)", "type": "mteb/mtop_domain", "config": "fr", "split": "test", "revision": "a7e2a951126a26fc8c6a69f835f33a346ba259e3"}, "metrics": [{"type": "accuracy", "value": 90.98340119010334}, {"type": "f1", "value": 90.51522537281313}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPDomainClassification (hi)", "type": "mteb/mtop_domain", "config": "hi", "split": "test", "revision": "a7e2a951126a26fc8c6a69f835f33a346ba259e3"}, "metrics": [{"type": "accuracy", "value": 89.33309429903191}, {"type": "f1", "value": 88.60371305209185}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPDomainClassification (th)", "type": "mteb/mtop_domain", "config": "th", "split": "test", "revision": "a7e2a951126a26fc8c6a69f835f33a346ba259e3"}, "metrics": [{"type": "accuracy", "value": 60.4882459312839}, {"type": "f1", "value": 59.02590456131682}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPIntentClassification (en)", "type": "mteb/mtop_intent", "config": "en", "split": "test", "revision": "6299947a7777084cc2d4b64235bf7190381ce755"}, "metrics": [{"type": "accuracy", "value": 71.34290925672595}, {"type": "f1", "value": 54.44803151449109}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPIntentClassification (de)", "type": "mteb/mtop_intent", "config": "de", "split": "test", "revision": "6299947a7777084cc2d4b64235bf7190381ce755"}, "metrics": [{"type": "accuracy", "value": 61.92448577063963}, {"type": "f1", "value": 43.125939975781854}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPIntentClassification (es)", "type": "mteb/mtop_intent", "config": "es", "split": "test", "revision": "6299947a7777084cc2d4b64235bf7190381ce755"}, "metrics": [{"type": "accuracy", "value": 74.48965977318213}, {"type": "f1", "value": 51.855353687466696}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPIntentClassification (fr)", "type": "mteb/mtop_intent", "config": "fr", "split": "test", "revision": "6299947a7777084cc2d4b64235bf7190381ce755"}, "metrics": [{"type": "accuracy", "value": 69.11994989038521}, {"type": "f1", "value": 50.57872704171278}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPIntentClassification (hi)", "type": "mteb/mtop_intent", "config": "hi", "split": "test", "revision": "6299947a7777084cc2d4b64235bf7190381ce755"}, "metrics": [{"type": "accuracy", "value": 64.84761563284331}, {"type": "f1", "value": 43.61322970761394}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPIntentClassification (th)", "type": "mteb/mtop_intent", "config": "th", "split": "test", "revision": "6299947a7777084cc2d4b64235bf7190381ce755"}, "metrics": [{"type": "accuracy", "value": 49.35623869801085}, {"type": "f1", "value": 33.48547326952042}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (af)", "type": "mteb/amazon_massive_intent", "config": "af", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 47.85474108944183}, {"type": "f1", "value": 46.50175016795915}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (am)", "type": "mteb/amazon_massive_intent", "config": "am", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 33.29858776059179}, {"type": "f1", "value": 31.803027601259082}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (ar)", "type": "mteb/amazon_massive_intent", "config": "ar", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 59.24680564895763}, {"type": "f1", "value": 57.037691806846865}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (az)", "type": "mteb/amazon_massive_intent", "config": "az", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 45.23537323470073}, {"type": "f1", "value": 44.81126398428613}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (bn)", "type": "mteb/amazon_massive_intent", "config": "bn", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 61.590450571620714}, {"type": "f1", "value": 59.247442149977104}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (cy)", "type": "mteb/amazon_massive_intent", "config": "cy", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 44.9226630800269}, {"type": "f1", "value": 44.076183379991654}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (da)", "type": "mteb/amazon_massive_intent", "config": "da", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 51.23066577000672}, {"type": "f1", "value": 50.20719330417618}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (de)", "type": "mteb/amazon_massive_intent", "config": "de", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 56.0995292535306}, {"type": "f1", "value": 53.29421532133969}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (el)", "type": "mteb/amazon_massive_intent", "config": "el", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 46.12642905178211}, {"type": "f1", "value": 44.441530267639635}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (en)", "type": "mteb/amazon_massive_intent", "config": "en", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 69.67047747141896}, {"type": "f1", "value": 68.38493366054783}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (es)", "type": "mteb/amazon_massive_intent", "config": "es", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 66.3483523873571}, {"type": "f1", "value": 65.13046416817832}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (fa)", "type": "mteb/amazon_massive_intent", "config": "fa", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 51.20040349697378}, {"type": "f1", "value": 49.02889836601541}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (fi)", "type": "mteb/amazon_massive_intent", "config": "fi", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 45.33288500336248}, {"type": "f1", "value": 42.91893101970983}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (fr)", "type": "mteb/amazon_massive_intent", "config": "fr", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 66.95359784801613}, {"type": "f1", "value": 64.98788914810562}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (he)", "type": "mteb/amazon_massive_intent", "config": "he", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 43.18090114324143}, {"type": "f1", "value": 41.31250407417542}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (hi)", "type": "mteb/amazon_massive_intent", "config": "hi", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 63.54068594485541}, {"type": "f1", "value": 61.94829361488948}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (hu)", "type": "mteb/amazon_massive_intent", "config": "hu", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 44.7343644922663}, {"type": "f1", "value": 43.23001702247849}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (hy)", "type": "mteb/amazon_massive_intent", "config": "hy", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 38.1271015467384}, {"type": "f1", "value": 36.94700198241727}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (id)", "type": "mteb/amazon_massive_intent", "config": "id", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 64.05514458641561}, {"type": "f1", "value": 62.35033731674541}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (is)", "type": "mteb/amazon_massive_intent", "config": "is", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 44.351042367182245}, {"type": "f1", "value": 43.13370397574502}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (it)", "type": "mteb/amazon_massive_intent", "config": "it", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 60.77000672494955}, {"type": "f1", "value": 59.71546868957779}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (ja)", "type": "mteb/amazon_massive_intent", "config": "ja", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 61.22057834566241}, {"type": "f1", "value": 59.447639306287044}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (jv)", "type": "mteb/amazon_massive_intent", "config": "jv", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 50.9448554135844}, {"type": "f1", "value": 48.524338247875214}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (ka)", "type": "mteb/amazon_massive_intent", "config": "ka", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 33.8399462004035}, {"type": "f1", "value": 33.518999997305535}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (km)", "type": "mteb/amazon_massive_intent", "config": "km", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 37.34028244788165}, {"type": "f1", "value": 35.6156599064704}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (kn)", "type": "mteb/amazon_massive_intent", "config": "kn", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 53.544048419636844}, {"type": "f1", "value": 51.29299915455352}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (ko)", "type": "mteb/amazon_massive_intent", "config": "ko", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 53.35574983187625}, {"type": "f1", "value": 51.463936565192945}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (lv)", "type": "mteb/amazon_massive_intent", "config": "lv", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 46.503026227303295}, {"type": "f1", "value": 46.049497734375514}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (ml)", "type": "mteb/amazon_massive_intent", "config": "ml", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 58.268325487558826}, {"type": "f1", "value": 56.10849656896158}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (mn)", "type": "mteb/amazon_massive_intent", "config": "mn", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 40.27572293207801}, {"type": "f1", "value": 40.20097238549224}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (ms)", "type": "mteb/amazon_massive_intent", "config": "ms", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 59.64694014794889}, {"type": "f1", "value": 58.39584148789066}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (my)", "type": "mteb/amazon_massive_intent", "config": "my", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 37.41761936785474}, {"type": "f1", "value": 35.04551731363685}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (nb)", "type": "mteb/amazon_massive_intent", "config": "nb", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 49.408204438466704}, {"type": "f1", "value": 48.39369057638714}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (nl)", "type": "mteb/amazon_massive_intent", "config": "nl", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 52.09482178883659}, {"type": "f1", "value": 49.91518031712698}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (pl)", "type": "mteb/amazon_massive_intent", "config": "pl", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 50.477471418964356}, {"type": "f1", "value": 48.429495257184705}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (pt)", "type": "mteb/amazon_massive_intent", "config": "pt", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 66.69468728984532}, {"type": "f1", "value": 65.40306868707009}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (ro)", "type": "mteb/amazon_massive_intent", "config": "ro", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 50.52790854068594}, {"type": "f1", "value": 49.780400354514}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (ru)", "type": "mteb/amazon_massive_intent", "config": "ru", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 58.31540013449899}, {"type": "f1", "value": 56.144142926685134}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (sl)", "type": "mteb/amazon_massive_intent", "config": "sl", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 47.74041694687289}, {"type": "f1", "value": 46.16767322761359}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (sq)", "type": "mteb/amazon_massive_intent", "config": "sq", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 48.94418291862811}, {"type": "f1", "value": 48.445352284756325}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (sv)", "type": "mteb/amazon_massive_intent", "config": "sv", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 50.78681909885676}, {"type": "f1", "value": 49.64882295494536}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (sw)", "type": "mteb/amazon_massive_intent", "config": "sw", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 49.811701412239415}, {"type": "f1", "value": 48.213234514449375}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (ta)", "type": "mteb/amazon_massive_intent", "config": "ta", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 56.39542703429725}, {"type": "f1", "value": 54.031981085233795}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (te)", "type": "mteb/amazon_massive_intent", "config": "te", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 54.71082716879623}, {"type": "f1", "value": 52.513144113474596}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (th)", "type": "mteb/amazon_massive_intent", "config": "th", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 44.425016812373904}, {"type": "f1", "value": 43.96016300057656}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (tl)", "type": "mteb/amazon_massive_intent", "config": "tl", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 50.205110961667785}, {"type": "f1", "value": 48.86669996798709}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (tr)", "type": "mteb/amazon_massive_intent", "config": "tr", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 46.56355077336921}, {"type": "f1", "value": 45.18252022585022}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (ur)", "type": "mteb/amazon_massive_intent", "config": "ur", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 56.748486886348346}, {"type": "f1", "value": 54.29884570375382}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (vi)", "type": "mteb/amazon_massive_intent", "config": "vi", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 64.52589105581708}, {"type": "f1", "value": 62.97947342861603}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (zh-CN)", "type": "mteb/amazon_massive_intent", "config": "zh-CN", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 67.06792199058508}, {"type": "f1", "value": 65.36025601634017}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (zh-TW)", "type": "mteb/amazon_massive_intent", "config": "zh-TW", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 62.89172831203766}, {"type": "f1", "value": 62.69803707054342}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (af)", "type": "mteb/amazon_massive_scenario", "config": "af", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 51.47276395427035}, {"type": "f1", "value": 49.37463208130799}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (am)", "type": "mteb/amazon_massive_scenario", "config": "am", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 34.86886348352387}, {"type": "f1", "value": 33.74178074349636}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (ar)", "type": "mteb/amazon_massive_scenario", "config": "ar", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 65.20511096166778}, {"type": "f1", "value": 65.85812500602437}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (az)", "type": "mteb/amazon_massive_scenario", "config": "az", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 45.578345662407536}, {"type": "f1", "value": 44.44514917028003}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (bn)", "type": "mteb/amazon_massive_scenario", "config": "bn", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 67.29657027572293}, {"type": "f1", "value": 67.24477523937466}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (cy)", "type": "mteb/amazon_massive_scenario", "config": "cy", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 46.29455279085407}, {"type": "f1", "value": 43.8563839951935}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (da)", "type": "mteb/amazon_massive_scenario", "config": "da", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 53.52387357094821}, {"type": "f1", "value": 51.70977848027552}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (de)", "type": "mteb/amazon_massive_scenario", "config": "de", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 61.741761936785466}, {"type": "f1", "value": 60.219169644792295}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (el)", "type": "mteb/amazon_massive_scenario", "config": "el", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 48.957632817753876}, {"type": "f1", "value": 46.878428264460034}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (en)", "type": "mteb/amazon_massive_scenario", "config": "en", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 75.33624747814393}, {"type": "f1", "value": 75.9143846211171}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (es)", "type": "mteb/amazon_massive_scenario", "config": "es", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 73.34229993275049}, {"type": "f1", "value": 73.78165397558983}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (fa)", "type": "mteb/amazon_massive_scenario", "config": "fa", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 53.174176193678555}, {"type": "f1", "value": 51.709679227778985}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (fi)", "type": "mteb/amazon_massive_scenario", "config": "fi", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 44.6906523201076}, {"type": "f1", "value": 41.54881682785664}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (fr)", "type": "mteb/amazon_massive_scenario", "config": "fr", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 72.9119031607263}, {"type": "f1", "value": 73.2742013056326}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (he)", "type": "mteb/amazon_massive_scenario", "config": "he", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 43.10356422326832}, {"type": "f1", "value": 40.8859122581252}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (hi)", "type": "mteb/amazon_massive_scenario", "config": "hi", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 69.27370544720914}, {"type": "f1", "value": 69.39544506405082}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (hu)", "type": "mteb/amazon_massive_scenario", "config": "hu", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 45.16476126429052}, {"type": "f1", "value": 42.74022531579054}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (hy)", "type": "mteb/amazon_massive_scenario", "config": "hy", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 38.73234700739744}, {"type": "f1", "value": 37.40546754951026}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (id)", "type": "mteb/amazon_massive_scenario", "config": "id", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 70.12777404169468}, {"type": "f1", "value": 70.27219152812738}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (is)", "type": "mteb/amazon_massive_scenario", "config": "is", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 44.21318090114325}, {"type": "f1", "value": 41.934593213829366}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (it)", "type": "mteb/amazon_massive_scenario", "config": "it", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 65.57162071284466}, {"type": "f1", "value": 64.83341759045335}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (ja)", "type": "mteb/amazon_massive_scenario", "config": "ja", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 65.75991930060525}, {"type": "f1", "value": 65.16549875504951}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (jv)", "type": "mteb/amazon_massive_scenario", "config": "jv", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 54.79488903833223}, {"type": "f1", "value": 54.03616401426859}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (ka)", "type": "mteb/amazon_massive_scenario", "config": "ka", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 32.992602555480836}, {"type": "f1", "value": 31.820068470018846}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (km)", "type": "mteb/amazon_massive_scenario", "config": "km", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 39.34431741761937}, {"type": "f1", "value": 36.436221665290105}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (kn)", "type": "mteb/amazon_massive_scenario", "config": "kn", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 60.501008742434436}, {"type": "f1", "value": 60.051013712579085}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (ko)", "type": "mteb/amazon_massive_scenario", "config": "ko", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 55.689307330195035}, {"type": "f1", "value": 53.94058032286942}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (lv)", "type": "mteb/amazon_massive_scenario", "config": "lv", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 44.351042367182245}, {"type": "f1", "value": 42.05421666771541}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (ml)", "type": "mteb/amazon_massive_scenario", "config": "ml", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 65.53127101546738}, {"type": "f1", "value": 65.98462024333497}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (mn)", "type": "mteb/amazon_massive_scenario", "config": "mn", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 38.71553463349025}, {"type": "f1", "value": 37.44327037149584}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (ms)", "type": "mteb/amazon_massive_scenario", "config": "ms", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 64.98991257565567}, {"type": "f1", "value": 63.87720198978004}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (my)", "type": "mteb/amazon_massive_scenario", "config": "my", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 36.839273705447205}, {"type": "f1", "value": 35.233967279698376}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (nb)", "type": "mteb/amazon_massive_scenario", "config": "nb", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 51.79892400806993}, {"type": "f1", "value": 49.66926632125972}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (nl)", "type": "mteb/amazon_massive_scenario", "config": "nl", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 56.31809011432415}, {"type": "f1", "value": 53.832185336179826}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (pl)", "type": "mteb/amazon_massive_scenario", "config": "pl", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 49.979825151311374}, {"type": "f1", "value": 48.83013175441888}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (pt)", "type": "mteb/amazon_massive_scenario", "config": "pt", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 71.45595158036315}, {"type": "f1", "value": 72.08708814699702}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (ro)", "type": "mteb/amazon_massive_scenario", "config": "ro", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 53.68527236045729}, {"type": "f1", "value": 52.23278593929981}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (ru)", "type": "mteb/amazon_massive_scenario", "config": "ru", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 61.60390047074647}, {"type": "f1", "value": 60.50391482195116}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (sl)", "type": "mteb/amazon_massive_scenario", "config": "sl", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 48.036314727639535}, {"type": "f1", "value": 46.43480413383716}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (sq)", "type": "mteb/amazon_massive_scenario", "config": "sq", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 50.05716207128445}, {"type": "f1", "value": 48.85821859948888}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (sv)", "type": "mteb/amazon_massive_scenario", "config": "sv", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 51.728312037659705}, {"type": "f1", "value": 49.89292996950847}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (sw)", "type": "mteb/amazon_massive_scenario", "config": "sw", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 54.21990585070613}, {"type": "f1", "value": 52.8711542984193}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (ta)", "type": "mteb/amazon_massive_scenario", "config": "ta", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 62.770679219905844}, {"type": "f1", "value": 63.09441501491594}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (te)", "type": "mteb/amazon_massive_scenario", "config": "te", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 62.58574310692671}, {"type": "f1", "value": 61.61370697612978}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (th)", "type": "mteb/amazon_massive_scenario", "config": "th", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 45.17821116341628}, {"type": "f1", "value": 43.85143229183324}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (tl)", "type": "mteb/amazon_massive_scenario", "config": "tl", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 52.064559515803644}, {"type": "f1", "value": 50.94356892049626}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (tr)", "type": "mteb/amazon_massive_scenario", "config": "tr", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 47.205783456624076}, {"type": "f1", "value": 47.04223644120489}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (ur)", "type": "mteb/amazon_massive_scenario", "config": "ur", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 64.25689307330195}, {"type": "f1", "value": 63.89944944984115}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (vi)", "type": "mteb/amazon_massive_scenario", "config": "vi", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 70.60524546065905}, {"type": "f1", "value": 71.5634157334358}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (zh-CN)", "type": "mteb/amazon_massive_scenario", "config": "zh-CN", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 73.95427034297242}, {"type": "f1", "value": 74.39706882311063}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (zh-TW)", "type": "mteb/amazon_massive_scenario", "config": "zh-TW", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 70.29926025554808}, {"type": "f1", "value": 71.32045932560297}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB MedrxivClusteringP2P", "type": "mteb/medrxiv-clustering-p2p", "config": "default", "split": "test", "revision": "dcefc037ef84348e49b0d29109e891c01067226b"}, "metrics": [{"type": "v_measure", "value": 31.054474964883806}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB MedrxivClusteringS2S", "type": "mteb/medrxiv-clustering-s2s", "config": "default", "split": "test", "revision": "3cd0e71dfbe09d4de0f9e5ecba43e7ce280959dc"}, "metrics": [{"type": "v_measure", "value": 29.259725940477523}]}, {"task": {"type": "Reranking"}, "dataset": {"name": "MTEB MindSmallReranking", "type": "mteb/mind_small", "config": "default", "split": "test", "revision": "3bdac13927fdc888b903db93b2ffdbd90b295a69"}, "metrics": [{"type": "map", "value": 31.785007883256572}, {"type": "mrr", "value": 32.983556622438456}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB NFCorpus", "type": "nfcorpus", "config": "default", "split": "test", "revision": "7eb63cc0c1eb59324d709ebed25fcab851fa7610"}, "metrics": [{"type": "map_at_1", "value": 5.742}, {"type": "map_at_10", "value": 13.074}, {"type": "map_at_100", "value": 16.716}, {"type": "map_at_1000", "value": 18.238}, {"type": "map_at_3", "value": 9.600999999999999}, {"type": "map_at_5", "value": 11.129999999999999}, {"type": "mrr_at_1", "value": 47.988}, {"type": "mrr_at_10", "value": 55.958}, {"type": "mrr_at_100", "value": 56.58800000000001}, {"type": "mrr_at_1000", "value": 56.620000000000005}, {"type": "mrr_at_3", "value": 54.025}, {"type": "mrr_at_5", "value": 55.31}, {"type": "ndcg_at_1", "value": 46.44}, {"type": "ndcg_at_10", "value": 35.776}, {"type": "ndcg_at_100", "value": 32.891999999999996}, {"type": "ndcg_at_1000", "value": 41.835}, {"type": "ndcg_at_3", "value": 41.812}, {"type": "ndcg_at_5", "value": 39.249}, {"type": "precision_at_1", "value": 48.297000000000004}, {"type": "precision_at_10", "value": 26.687}, {"type": "precision_at_100", "value": 8.511000000000001}, {"type": "precision_at_1000", "value": 2.128}, {"type": "precision_at_3", "value": 39.009}, {"type": "precision_at_5", "value": 33.994}, {"type": "recall_at_1", "value": 5.742}, {"type": "recall_at_10", "value": 16.993}, {"type": "recall_at_100", "value": 33.69}, {"type": "recall_at_1000", "value": 66.75}, {"type": "recall_at_3", "value": 10.817}, {"type": "recall_at_5", "value": 13.256}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB NQ", "type": "nq", "config": "default", "split": "test", "revision": "6062aefc120bfe8ece5897809fb2e53bfe0d128c"}, "metrics": [{"type": "map_at_1", "value": 30.789}, {"type": "map_at_10", "value": 45.751999999999995}, {"type": "map_at_100", "value": 46.766000000000005}, {"type": "map_at_1000", "value": 46.798}, {"type": "map_at_3", "value": 41.746}, {"type": "map_at_5", "value": 44.046}, {"type": "mrr_at_1", "value": 34.618}, {"type": "mrr_at_10", "value": 48.288}, {"type": "mrr_at_100", "value": 49.071999999999996}, {"type": "mrr_at_1000", "value": 49.094}, {"type": "mrr_at_3", "value": 44.979}, {"type": "mrr_at_5", "value": 46.953}, {"type": "ndcg_at_1", "value": 34.589}, {"type": "ndcg_at_10", "value": 53.151}, {"type": "ndcg_at_100", "value": 57.537000000000006}, {"type": "ndcg_at_1000", "value": 58.321999999999996}, {"type": "ndcg_at_3", "value": 45.628}, {"type": "ndcg_at_5", "value": 49.474000000000004}, {"type": "precision_at_1", "value": 34.589}, {"type": "precision_at_10", "value": 8.731}, {"type": "precision_at_100", "value": 1.119}, {"type": "precision_at_1000", "value": 0.11900000000000001}, {"type": "precision_at_3", "value": 20.819}, {"type": "precision_at_5", "value": 14.728}, {"type": "recall_at_1", "value": 30.789}, {"type": "recall_at_10", "value": 73.066}, {"type": "recall_at_100", "value": 92.27}, {"type": "recall_at_1000", "value": 98.18}, {"type": "recall_at_3", "value": 53.632999999999996}, {"type": "recall_at_5", "value": 62.476}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB QuoraRetrieval", "type": "quora", "config": "default", "split": "test", "revision": "6205996560df11e3a3da9ab4f926788fc30a7db4"}, "metrics": [{"type": "map_at_1", "value": 54.993}, {"type": "map_at_10", "value": 69.07600000000001}, {"type": "map_at_100", "value": 70.05799999999999}, {"type": "map_at_1000", "value": 70.09}, {"type": "map_at_3", "value": 65.456}, {"type": "map_at_5", "value": 67.622}, {"type": "mrr_at_1", "value": 63.07000000000001}, {"type": "mrr_at_10", "value": 72.637}, {"type": "mrr_at_100", "value": 73.029}, {"type": "mrr_at_1000", "value": 73.033}, {"type": "mrr_at_3", "value": 70.572}, {"type": "mrr_at_5", "value": 71.86399999999999}, {"type": "ndcg_at_1", "value": 63.07000000000001}, {"type": "ndcg_at_10", "value": 74.708}, {"type": "ndcg_at_100", "value": 77.579}, {"type": "ndcg_at_1000", "value": 77.897}, {"type": "ndcg_at_3", "value": 69.69999999999999}, {"type": "ndcg_at_5", "value": 72.321}, {"type": "precision_at_1", "value": 63.07000000000001}, {"type": "precision_at_10", "value": 11.851}, {"type": "precision_at_100", "value": 1.481}, {"type": "precision_at_1000", "value": 0.156}, {"type": "precision_at_3", "value": 30.747000000000003}, {"type": "precision_at_5", "value": 20.830000000000002}, {"type": "recall_at_1", "value": 54.993}, {"type": "recall_at_10", "value": 87.18900000000001}, {"type": "recall_at_100", "value": 98.137}, {"type": "recall_at_1000", "value": 99.833}, {"type": "recall_at_3", "value": 73.654}, {"type": "recall_at_5", "value": 80.36}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB RedditClustering", "type": "mteb/reddit-clustering", "config": "default", "split": "test", "revision": "b2805658ae38990172679479369a78b86de8c390"}, "metrics": [{"type": "v_measure", "value": 35.53178375429036}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB RedditClusteringP2P", "type": "mteb/reddit-clustering-p2p", "config": "default", "split": "test", "revision": "385e3cb46b4cfa89021f56c4380204149d0efe33"}, "metrics": [{"type": "v_measure", "value": 54.520782970558265}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB SCIDOCS", "type": "scidocs", "config": "default", "split": "test", "revision": "5c59ef3e437a0a9651c8fe6fde943e7dce59fba5"}, "metrics": [{"type": "map_at_1", "value": 4.3229999999999995}, {"type": "map_at_10", "value": 10.979999999999999}, {"type": "map_at_100", "value": 12.867}, {"type": "map_at_1000", "value": 13.147}, {"type": "map_at_3", "value": 7.973}, {"type": "map_at_5", "value": 9.513}, {"type": "mrr_at_1", "value": 21.3}, {"type": "mrr_at_10", "value": 32.34}, {"type": "mrr_at_100", "value": 33.428999999999995}, {"type": "mrr_at_1000", "value": 33.489999999999995}, {"type": "mrr_at_3", "value": 28.999999999999996}, {"type": "mrr_at_5", "value": 31.019999999999996}, {"type": "ndcg_at_1", "value": 21.3}, {"type": "ndcg_at_10", "value": 18.619}, {"type": "ndcg_at_100", "value": 26.108999999999998}, {"type": "ndcg_at_1000", "value": 31.253999999999998}, {"type": "ndcg_at_3", "value": 17.842}, {"type": "ndcg_at_5", "value": 15.673}, {"type": "precision_at_1", "value": 21.3}, {"type": "precision_at_10", "value": 9.55}, {"type": "precision_at_100", "value": 2.0340000000000003}, {"type": "precision_at_1000", "value": 0.327}, {"type": "precision_at_3", "value": 16.667}, {"type": "precision_at_5", "value": 13.76}, {"type": "recall_at_1", "value": 4.3229999999999995}, {"type": "recall_at_10", "value": 19.387}, {"type": "recall_at_100", "value": 41.307}, {"type": "recall_at_1000", "value": 66.475}, {"type": "recall_at_3", "value": 10.143}, {"type": "recall_at_5", "value": 14.007}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB SICK-R", "type": "mteb/sickr-sts", "config": "default", "split": "test", "revision": "20a6d6f312dd54037fe07a32d58e5e168867909d"}, "metrics": [{"type": "cos_sim_pearson", "value": 78.77975189382573}, {"type": "cos_sim_spearman", "value": 69.81522686267631}, {"type": "euclidean_pearson", "value": 71.37617936889518}, {"type": "euclidean_spearman", "value": 65.71738481148611}, {"type": "manhattan_pearson", "value": 71.58222165832424}, {"type": "manhattan_spearman", "value": 65.86851365286654}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS12", "type": "mteb/sts12-sts", "config": "default", "split": "test", "revision": "fdf84275bb8ce4b49c971d02e84dd1abc677a50f"}, "metrics": [{"type": "cos_sim_pearson", "value": 77.75509450443367}, {"type": "cos_sim_spearman", "value": 69.66180222442091}, {"type": "euclidean_pearson", "value": 74.98512779786111}, {"type": "euclidean_spearman", "value": 69.5997451409469}, {"type": "manhattan_pearson", "value": 75.50135090962459}, {"type": "manhattan_spearman", "value": 69.94984748475302}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS13", "type": "mteb/sts13-sts", "config": "default", "split": "test", "revision": "1591bfcbe8c69d4bf7fe2a16e2451017832cafb9"}, "metrics": [{"type": "cos_sim_pearson", "value": 79.42363892383264}, {"type": "cos_sim_spearman", "value": 79.66529244176742}, {"type": "euclidean_pearson", "value": 79.50429208135942}, {"type": "euclidean_spearman", "value": 80.44767586416276}, {"type": "manhattan_pearson", "value": 79.58563944997708}, {"type": "manhattan_spearman", "value": 80.51452267103}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS14", "type": "mteb/sts14-sts", "config": "default", "split": "test", "revision": "e2125984e7df8b7871f6ae9949cf6b6795e7c54b"}, "metrics": [{"type": "cos_sim_pearson", "value": 79.2749401478149}, {"type": "cos_sim_spearman", "value": 74.6076920702392}, {"type": "euclidean_pearson", "value": 73.3302002952881}, {"type": "euclidean_spearman", "value": 70.67029803077013}, {"type": "manhattan_pearson", "value": 73.52699344010296}, {"type": "manhattan_spearman", "value": 70.8517556194297}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS15", "type": "mteb/sts15-sts", "config": "default", "split": "test", "revision": "1cd7298cac12a96a373b6a2f18738bb3e739a9b6"}, "metrics": [{"type": "cos_sim_pearson", "value": 83.20884740785921}, {"type": "cos_sim_spearman", "value": 83.80600789090722}, {"type": "euclidean_pearson", "value": 74.9154089816344}, {"type": "euclidean_spearman", "value": 75.69243899592276}, {"type": "manhattan_pearson", "value": 75.0312832634451}, {"type": "manhattan_spearman", "value": 75.78324960357642}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS16", "type": "mteb/sts16-sts", "config": "default", "split": "test", "revision": "360a0b2dff98700d09e634a01e1cc1624d3e42cd"}, "metrics": [{"type": "cos_sim_pearson", "value": 79.63194141000497}, {"type": "cos_sim_spearman", "value": 80.40118418350866}, {"type": "euclidean_pearson", "value": 72.07354384551088}, {"type": "euclidean_spearman", "value": 72.28819150373845}, {"type": "manhattan_pearson", "value": 72.08736119834145}, {"type": "manhattan_spearman", "value": 72.28347083261288}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (ko-ko)", "type": "mteb/sts17-crosslingual-sts", "config": "ko-ko", "split": "test", "revision": "9fc37e8c632af1c87a3d23e685d49552a02582a0"}, "metrics": [{"type": "cos_sim_pearson", "value": 66.78512789499386}, {"type": "cos_sim_spearman", "value": 66.89125587193288}, {"type": "euclidean_pearson", "value": 58.74535708627959}, {"type": "euclidean_spearman", "value": 59.62103716794647}, {"type": "manhattan_pearson", "value": 59.00494529143961}, {"type": "manhattan_spearman", "value": 59.832257846799806}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (ar-ar)", "type": "mteb/sts17-crosslingual-sts", "config": "ar-ar", "split": "test", "revision": "9fc37e8c632af1c87a3d23e685d49552a02582a0"}, "metrics": [{"type": "cos_sim_pearson", "value": 75.48960503523992}, {"type": "cos_sim_spearman", "value": 76.4223037534204}, {"type": "euclidean_pearson", "value": 64.93966381820944}, {"type": "euclidean_spearman", "value": 62.39697395373789}, {"type": "manhattan_pearson", "value": 65.54480770061505}, {"type": "manhattan_spearman", "value": 62.944204863043105}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (en-ar)", "type": "mteb/sts17-crosslingual-sts", "config": "en-ar", "split": "test", "revision": "9fc37e8c632af1c87a3d23e685d49552a02582a0"}, "metrics": [{"type": "cos_sim_pearson", "value": 77.7331440643619}, {"type": "cos_sim_spearman", "value": 78.0748413292835}, {"type": "euclidean_pearson", "value": 38.533108233460304}, {"type": "euclidean_spearman", "value": 35.37638615280026}, {"type": "manhattan_pearson", "value": 41.0639726746513}, {"type": "manhattan_spearman", "value": 37.688161243671765}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (en-de)", "type": "mteb/sts17-crosslingual-sts", "config": "en-de", "split": "test", "revision": "9fc37e8c632af1c87a3d23e685d49552a02582a0"}, "metrics": [{"type": "cos_sim_pearson", "value": 58.4628923720782}, {"type": "cos_sim_spearman", "value": 59.10093128795948}, {"type": "euclidean_pearson", "value": 30.422902393436836}, {"type": "euclidean_spearman", "value": 27.837806030497457}, {"type": "manhattan_pearson", "value": 32.51576984630963}, {"type": "manhattan_spearman", "value": 29.181887010982514}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (en-en)", "type": "mteb/sts17-crosslingual-sts", "config": "en-en", "split": "test", "revision": "9fc37e8c632af1c87a3d23e685d49552a02582a0"}, "metrics": [{"type": "cos_sim_pearson", "value": 86.87447904613737}, {"type": "cos_sim_spearman", "value": 87.06554974065622}, {"type": "euclidean_pearson", "value": 76.82669047851108}, {"type": "euclidean_spearman", "value": 75.45711985511991}, {"type": "manhattan_pearson", "value": 77.46644556452847}, {"type": "manhattan_spearman", "value": 76.0249120007112}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (en-tr)", "type": "mteb/sts17-crosslingual-sts", "config": "en-tr", "split": "test", "revision": "9fc37e8c632af1c87a3d23e685d49552a02582a0"}, "metrics": [{"type": "cos_sim_pearson", "value": 17.784495723497468}, {"type": "cos_sim_spearman", "value": 11.79629537128697}, {"type": "euclidean_pearson", "value": -4.354328445994008}, {"type": "euclidean_spearman", "value": -6.984566116230058}, {"type": "manhattan_pearson", "value": -4.166751901507852}, {"type": "manhattan_spearman", "value": -6.984143198323786}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (es-en)", "type": "mteb/sts17-crosslingual-sts", "config": "es-en", "split": "test", "revision": "9fc37e8c632af1c87a3d23e685d49552a02582a0"}, "metrics": [{"type": "cos_sim_pearson", "value": 76.9009642643449}, {"type": "cos_sim_spearman", "value": 78.21764726338341}, {"type": "euclidean_pearson", "value": 50.578959144342925}, {"type": "euclidean_spearman", "value": 51.664379260719606}, {"type": "manhattan_pearson", "value": 53.95690880393329}, {"type": "manhattan_spearman", "value": 54.910058464050785}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (es-es)", "type": "mteb/sts17-crosslingual-sts", "config": "es-es", "split": "test", "revision": "9fc37e8c632af1c87a3d23e685d49552a02582a0"}, "metrics": [{"type": "cos_sim_pearson", "value": 86.41638022270219}, {"type": "cos_sim_spearman", "value": 86.00477030366811}, {"type": "euclidean_pearson", "value": 79.7224037788285}, {"type": "euclidean_spearman", "value": 79.21417626867616}, {"type": "manhattan_pearson", "value": 80.29412412756984}, {"type": "manhattan_spearman", "value": 79.49460867616206}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (fr-en)", "type": "mteb/sts17-crosslingual-sts", "config": "fr-en", "split": "test", "revision": "9fc37e8c632af1c87a3d23e685d49552a02582a0"}, "metrics": [{"type": "cos_sim_pearson", "value": 79.90432664091082}, {"type": "cos_sim_spearman", "value": 80.46007940700204}, {"type": "euclidean_pearson", "value": 49.25348015214428}, {"type": "euclidean_spearman", "value": 47.13113020475859}, {"type": "manhattan_pearson", "value": 54.57291204043908}, {"type": "manhattan_spearman", "value": 51.98559736896087}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (it-en)", "type": "mteb/sts17-crosslingual-sts", "config": "it-en", "split": "test", "revision": "9fc37e8c632af1c87a3d23e685d49552a02582a0"}, "metrics": [{"type": "cos_sim_pearson", "value": 52.55164822309034}, {"type": "cos_sim_spearman", "value": 51.57629192137736}, {"type": "euclidean_pearson", "value": 16.63360593235354}, {"type": "euclidean_spearman", "value": 14.479679923782912}, {"type": "manhattan_pearson", "value": 18.524867185117472}, {"type": "manhattan_spearman", "value": 16.65940056664755}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (nl-en)", "type": "mteb/sts17-crosslingual-sts", "config": "nl-en", "split": "test", "revision": "9fc37e8c632af1c87a3d23e685d49552a02582a0"}, "metrics": [{"type": "cos_sim_pearson", "value": 46.83690919715875}, {"type": "cos_sim_spearman", "value": 45.84993650002922}, {"type": "euclidean_pearson", "value": 6.173128686815117}, {"type": "euclidean_spearman", "value": 6.260781946306191}, {"type": "manhattan_pearson", "value": 7.328440452367316}, {"type": "manhattan_spearman", "value": 7.370842306497447}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (en)", "type": "mteb/sts22-crosslingual-sts", "config": "en", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 64.97916914277232}, {"type": "cos_sim_spearman", "value": 66.13392188807865}, {"type": "euclidean_pearson", "value": 65.3921146908468}, {"type": "euclidean_spearman", "value": 65.8381588635056}, {"type": "manhattan_pearson", "value": 65.8866165769975}, {"type": "manhattan_spearman", "value": 66.27774050472219}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (de)", "type": "mteb/sts22-crosslingual-sts", "config": "de", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 25.605130445111545}, {"type": "cos_sim_spearman", "value": 30.054844562369254}, {"type": "euclidean_pearson", "value": 23.890611005408196}, {"type": "euclidean_spearman", "value": 29.07902600726761}, {"type": "manhattan_pearson", "value": 24.239478426621833}, {"type": "manhattan_spearman", "value": 29.48547576782375}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (es)", "type": "mteb/sts22-crosslingual-sts", "config": "es", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 61.6665616159781}, {"type": "cos_sim_spearman", "value": 65.41310206289988}, {"type": "euclidean_pearson", "value": 68.38805493215008}, {"type": "euclidean_spearman", "value": 65.22777377603435}, {"type": "manhattan_pearson", "value": 69.37445390454346}, {"type": "manhattan_spearman", "value": 66.02437701858754}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (pl)", "type": "mteb/sts22-crosslingual-sts", "config": "pl", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 15.302891825626372}, {"type": "cos_sim_spearman", "value": 31.134517255070097}, {"type": "euclidean_pearson", "value": 12.672592658843143}, {"type": "euclidean_spearman", "value": 29.14881036784207}, {"type": "manhattan_pearson", "value": 13.528545327757735}, {"type": "manhattan_spearman", "value": 29.56217928148797}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (tr)", "type": "mteb/sts22-crosslingual-sts", "config": "tr", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 28.79299114515319}, {"type": "cos_sim_spearman", "value": 47.135864983626206}, {"type": "euclidean_pearson", "value": 40.66410787594309}, {"type": "euclidean_spearman", "value": 45.09585593138228}, {"type": "manhattan_pearson", "value": 42.02561630700308}, {"type": "manhattan_spearman", "value": 45.43979983670554}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (ar)", "type": "mteb/sts22-crosslingual-sts", "config": "ar", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 46.00096625052943}, {"type": "cos_sim_spearman", "value": 58.67147426715496}, {"type": "euclidean_pearson", "value": 54.7154367422438}, {"type": "euclidean_spearman", "value": 59.003235142442634}, {"type": "manhattan_pearson", "value": 56.3116235357115}, {"type": "manhattan_spearman", "value": 60.12956331404423}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (ru)", "type": "mteb/sts22-crosslingual-sts", "config": "ru", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 29.3396354650316}, {"type": "cos_sim_spearman", "value": 43.3632935734809}, {"type": "euclidean_pearson", "value": 31.18506539466593}, {"type": "euclidean_spearman", "value": 37.531745324803815}, {"type": "manhattan_pearson", "value": 32.829038232529015}, {"type": "manhattan_spearman", "value": 38.04574361589953}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (zh)", "type": "mteb/sts22-crosslingual-sts", "config": "zh", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 62.9596148375188}, {"type": "cos_sim_spearman", "value": 66.77653412402461}, {"type": "euclidean_pearson", "value": 64.53156585980886}, {"type": "euclidean_spearman", "value": 66.2884373036083}, {"type": "manhattan_pearson", "value": 65.2831035495143}, {"type": "manhattan_spearman", "value": 66.83641945244322}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (fr)", "type": "mteb/sts22-crosslingual-sts", "config": "fr", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 79.9138821493919}, {"type": "cos_sim_spearman", "value": 80.38097535004677}, {"type": "euclidean_pearson", "value": 76.2401499094322}, {"type": "euclidean_spearman", "value": 77.00897050735907}, {"type": "manhattan_pearson", "value": 76.69531453728563}, {"type": "manhattan_spearman", "value": 77.83189696428695}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (de-en)", "type": "mteb/sts22-crosslingual-sts", "config": "de-en", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 51.27009640779202}, {"type": "cos_sim_spearman", "value": 51.16120562029285}, {"type": "euclidean_pearson", "value": 52.20594985566323}, {"type": "euclidean_spearman", "value": 52.75331049709882}, {"type": "manhattan_pearson", "value": 52.2725118792549}, {"type": "manhattan_spearman", "value": 53.614847968995115}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (es-en)", "type": "mteb/sts22-crosslingual-sts", "config": "es-en", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 70.46044814118835}, {"type": "cos_sim_spearman", "value": 75.05760236668672}, {"type": "euclidean_pearson", "value": 72.80128921879461}, {"type": "euclidean_spearman", "value": 73.81164755219257}, {"type": "manhattan_pearson", "value": 72.7863795809044}, {"type": "manhattan_spearman", "value": 73.65932033818906}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (it)", "type": "mteb/sts22-crosslingual-sts", "config": "it", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 61.89276840435938}, {"type": "cos_sim_spearman", "value": 65.65042955732055}, {"type": "euclidean_pearson", "value": 61.22969491863841}, {"type": "euclidean_spearman", "value": 63.451215637904724}, {"type": "manhattan_pearson", "value": 61.16138956945465}, {"type": "manhattan_spearman", "value": 63.34966179331079}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (pl-en)", "type": "mteb/sts22-crosslingual-sts", "config": "pl-en", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 56.377577221753626}, {"type": "cos_sim_spearman", "value": 53.31223653270353}, {"type": "euclidean_pearson", "value": 26.488793041564307}, {"type": "euclidean_spearman", "value": 19.524551741701472}, {"type": "manhattan_pearson", "value": 24.322868054606474}, {"type": "manhattan_spearman", "value": 19.50371443994939}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (zh-en)", "type": "mteb/sts22-crosslingual-sts", "config": "zh-en", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 69.3634693673425}, {"type": "cos_sim_spearman", "value": 68.45051245419702}, {"type": "euclidean_pearson", "value": 56.1417414374769}, {"type": "euclidean_spearman", "value": 55.89891749631458}, {"type": "manhattan_pearson", "value": 57.266417430882925}, {"type": "manhattan_spearman", "value": 56.57927102744128}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (es-it)", "type": "mteb/sts22-crosslingual-sts", "config": "es-it", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 60.04169437653179}, {"type": "cos_sim_spearman", "value": 65.49531007553446}, {"type": "euclidean_pearson", "value": 58.583860732586324}, {"type": "euclidean_spearman", "value": 58.80034792537441}, {"type": "manhattan_pearson", "value": 59.02513161664622}, {"type": "manhattan_spearman", "value": 58.42942047904558}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (de-fr)", "type": "mteb/sts22-crosslingual-sts", "config": "de-fr", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 48.81035211493999}, {"type": "cos_sim_spearman", "value": 53.27599246786967}, {"type": "euclidean_pearson", "value": 52.25710699032889}, {"type": "euclidean_spearman", "value": 55.22995695529873}, {"type": "manhattan_pearson", "value": 51.894901893217884}, {"type": "manhattan_spearman", "value": 54.95919975149795}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (de-pl)", "type": "mteb/sts22-crosslingual-sts", "config": "de-pl", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 36.75993101477816}, {"type": "cos_sim_spearman", "value": 43.050156692479355}, {"type": "euclidean_pearson", "value": 51.49021084746248}, {"type": "euclidean_spearman", "value": 49.54771253090078}, {"type": "manhattan_pearson", "value": 54.68410760796417}, {"type": "manhattan_spearman", "value": 48.19277197691717}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (fr-pl)", "type": "mteb/sts22-crosslingual-sts", "config": "fr-pl", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 48.553763306386486}, {"type": "cos_sim_spearman", "value": 28.17180849095055}, {"type": "euclidean_pearson", "value": 17.50739087826514}, {"type": "euclidean_spearman", "value": 16.903085094570333}, {"type": "manhattan_pearson", "value": 20.750046512534112}, {"type": "manhattan_spearman", "value": 5.634361698190111}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STSBenchmark", "type": "mteb/stsbenchmark-sts", "config": "default", "split": "test", "revision": "8913289635987208e6e7c72789e4be2fe94b6abd"}, "metrics": [{"type": "cos_sim_pearson", "value": 82.17107190594417}, {"type": "cos_sim_spearman", "value": 80.89611873505183}, {"type": "euclidean_pearson", "value": 71.82491561814403}, {"type": "euclidean_spearman", "value": 70.33608835403274}, {"type": "manhattan_pearson", "value": 71.89538332420133}, {"type": "manhattan_spearman", "value": 70.36082395775944}]}, {"task": {"type": "Reranking"}, "dataset": {"name": "MTEB SciDocsRR", "type": "mteb/scidocs-reranking", "config": "default", "split": "test", "revision": "56a6d0140cf6356659e2a7c1413286a774468d44"}, "metrics": [{"type": "map", "value": 79.77047154974562}, {"type": "mrr", "value": 94.25887021475256}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB SciFact", "type": "scifact", "config": "default", "split": "test", "revision": "a75ae049398addde9b70f6b268875f5cbce99089"}, "metrics": [{"type": "map_at_1", "value": 56.328}, {"type": "map_at_10", "value": 67.167}, {"type": "map_at_100", "value": 67.721}, {"type": "map_at_1000", "value": 67.735}, {"type": "map_at_3", "value": 64.20400000000001}, {"type": "map_at_5", "value": 65.904}, {"type": "mrr_at_1", "value": 59.667}, {"type": "mrr_at_10", "value": 68.553}, {"type": "mrr_at_100", "value": 68.992}, {"type": "mrr_at_1000", "value": 69.004}, {"type": "mrr_at_3", "value": 66.22200000000001}, {"type": "mrr_at_5", "value": 67.739}, {"type": "ndcg_at_1", "value": 59.667}, {"type": "ndcg_at_10", "value": 72.111}, {"type": "ndcg_at_100", "value": 74.441}, {"type": "ndcg_at_1000", "value": 74.90599999999999}, {"type": "ndcg_at_3", "value": 67.11399999999999}, {"type": "ndcg_at_5", "value": 69.687}, {"type": "precision_at_1", "value": 59.667}, {"type": "precision_at_10", "value": 9.733}, {"type": "precision_at_100", "value": 1.09}, {"type": "precision_at_1000", "value": 0.11299999999999999}, {"type": "precision_at_3", "value": 26.444000000000003}, {"type": "precision_at_5", "value": 17.599999999999998}, {"type": "recall_at_1", "value": 56.328}, {"type": "recall_at_10", "value": 85.8}, {"type": "recall_at_100", "value": 96.167}, {"type": "recall_at_1000", "value": 100.0}, {"type": "recall_at_3", "value": 72.433}, {"type": "recall_at_5", "value": 78.972}]}, {"task": {"type": "PairClassification"}, "dataset": {"name": "MTEB SprintDuplicateQuestions", "type": "mteb/sprintduplicatequestions-pairclassification", "config": "default", "split": "test", "revision": "5a8256d0dff9c4bd3be3ba3e67e4e70173f802ea"}, "metrics": [{"type": "cos_sim_accuracy", "value": 99.8019801980198}, {"type": "cos_sim_ap", "value": 94.92527097094644}, {"type": "cos_sim_f1", "value": 89.91935483870968}, {"type": "cos_sim_precision", "value": 90.65040650406505}, {"type": "cos_sim_recall", "value": 89.2}, {"type": "dot_accuracy", "value": 99.51782178217822}, {"type": "dot_ap", "value": 81.30756869559929}, {"type": "dot_f1", "value": 75.88235294117648}, {"type": "dot_precision", "value": 74.42307692307692}, {"type": "dot_recall", "value": 77.4}, {"type": "euclidean_accuracy", "value": 99.73069306930694}, {"type": "euclidean_ap", "value": 91.05040371796932}, {"type": "euclidean_f1", "value": 85.7889237199582}, {"type": "euclidean_precision", "value": 89.82494529540482}, {"type": "euclidean_recall", "value": 82.1}, {"type": "manhattan_accuracy", "value": 99.73762376237623}, {"type": "manhattan_ap", "value": 91.4823412839869}, {"type": "manhattan_f1", "value": 86.39836984207845}, {"type": "manhattan_precision", "value": 88.05815160955348}, {"type": "manhattan_recall", "value": 84.8}, {"type": "max_accuracy", "value": 99.8019801980198}, {"type": "max_ap", "value": 94.92527097094644}, {"type": "max_f1", "value": 89.91935483870968}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB StackExchangeClustering", "type": "mteb/stackexchange-clustering", "config": "default", "split": "test", "revision": "70a89468f6dccacc6aa2b12a6eac54e74328f235"}, "metrics": [{"type": "v_measure", "value": 55.13046832022158}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB StackExchangeClusteringP2P", "type": "mteb/stackexchange-clustering-p2p", "config": "default", "split": "test", "revision": "d88009ab563dd0b16cfaf4436abaf97fa3550cf0"}, "metrics": [{"type": "v_measure", "value": 34.31252463546675}]}, {"task": {"type": "Reranking"}, "dataset": {"name": "MTEB StackOverflowDupQuestions", "type": "mteb/stackoverflowdupquestions-reranking", "config": "default", "split": "test", "revision": "ef807ea29a75ec4f91b50fd4191cb4ee4589a9f9"}, "metrics": [{"type": "map", "value": 51.06639688231414}, {"type": "mrr", "value": 51.80205415499534}]}, {"task": {"type": "Summarization"}, "dataset": {"name": "MTEB SummEval", "type": "mteb/summeval", "config": "default", "split": "test", "revision": "8753c2788d36c01fc6f05d03fe3f7268d63f9122"}, "metrics": [{"type": "cos_sim_pearson", "value": 31.963331462886956}, {"type": "cos_sim_spearman", "value": 33.59510652629926}, {"type": "dot_pearson", "value": 29.033733540882125}, {"type": "dot_spearman", "value": 31.550290638315506}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB TRECCOVID", "type": "trec-covid", "config": "default", "split": "test", "revision": "2c8041b2c07a79b6f7ba8fe6acc72e5d9f92d217"}, "metrics": [{"type": "map_at_1", "value": 0.23600000000000002}, {"type": "map_at_10", "value": 2.09}, {"type": "map_at_100", "value": 12.466000000000001}, {"type": "map_at_1000", "value": 29.852}, {"type": "map_at_3", "value": 0.6859999999999999}, {"type": "map_at_5", "value": 1.099}, {"type": "mrr_at_1", "value": 88.0}, {"type": "mrr_at_10", "value": 94.0}, {"type": "mrr_at_100", "value": 94.0}, {"type": "mrr_at_1000", "value": 94.0}, {"type": "mrr_at_3", "value": 94.0}, {"type": "mrr_at_5", "value": 94.0}, {"type": "ndcg_at_1", "value": 86.0}, {"type": "ndcg_at_10", "value": 81.368}, {"type": "ndcg_at_100", "value": 61.879}, {"type": "ndcg_at_1000", "value": 55.282}, {"type": "ndcg_at_3", "value": 84.816}, {"type": "ndcg_at_5", "value": 82.503}, {"type": "precision_at_1", "value": 88.0}, {"type": "precision_at_10", "value": 85.6}, {"type": "precision_at_100", "value": 63.85999999999999}, {"type": "precision_at_1000", "value": 24.682000000000002}, {"type": "precision_at_3", "value": 88.667}, {"type": "precision_at_5", "value": 86.0}, {"type": "recall_at_1", "value": 0.23600000000000002}, {"type": "recall_at_10", "value": 2.25}, {"type": "recall_at_100", "value": 15.488}, {"type": "recall_at_1000", "value": 52.196}, {"type": "recall_at_3", "value": 0.721}, {"type": "recall_at_5", "value": 1.159}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (sqi-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "sqi-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 12.7}, {"type": "f1", "value": 10.384182044950325}, {"type": "precision", "value": 9.805277385275312}, {"type": "recall", "value": 12.7}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (fry-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "fry-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 30.63583815028902}, {"type": "f1", "value": 24.623726947426373}, {"type": "precision", "value": 22.987809919828013}, {"type": "recall", "value": 30.63583815028902}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (kur-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "kur-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 10.487804878048781}, {"type": "f1", "value": 8.255945048627975}, {"type": "precision", "value": 7.649047253615001}, {"type": "recall", "value": 10.487804878048781}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (tur-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "tur-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 8.5}, {"type": "f1", "value": 6.154428783776609}, {"type": "precision", "value": 5.680727638128585}, {"type": "recall", "value": 8.5}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (deu-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "deu-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 73.0}, {"type": "f1", "value": 70.10046605876393}, {"type": "precision", "value": 69.0018253968254}, {"type": "recall", "value": 73.0}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (nld-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "nld-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 32.7}, {"type": "f1", "value": 29.7428583868239}, {"type": "precision", "value": 28.81671359506905}, {"type": "recall", "value": 32.7}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (ron-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "ron-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 31.5}, {"type": "f1", "value": 27.228675552174003}, {"type": "precision", "value": 25.950062299847747}, {"type": "recall", "value": 31.5}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (ang-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "ang-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 35.82089552238806}, {"type": "f1", "value": 28.75836980510979}, {"type": "precision", "value": 26.971643613434658}, {"type": "recall", "value": 35.82089552238806}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (ido-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "ido-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 49.8}, {"type": "f1", "value": 43.909237401451776}, {"type": "precision", "value": 41.944763440988936}, {"type": "recall", "value": 49.8}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (jav-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "jav-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 18.536585365853657}, {"type": "f1", "value": 15.020182570246751}, {"type": "precision", "value": 14.231108073213337}, {"type": "recall", "value": 18.536585365853657}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (isl-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "isl-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 8.7}, {"type": "f1", "value": 6.2934784902885355}, {"type": "precision", "value": 5.685926293425392}, {"type": "recall", "value": 8.7}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (slv-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "slv-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 12.879708383961116}, {"type": "f1", "value": 10.136118341751114}, {"type": "precision", "value": 9.571444036679436}, {"type": "recall", "value": 12.879708383961116}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (cym-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "cym-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 9.217391304347826}, {"type": "f1", "value": 6.965003297761793}, {"type": "precision", "value": 6.476093529199119}, {"type": "recall", "value": 9.217391304347826}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (kaz-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "kaz-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 4.3478260869565215}, {"type": "f1", "value": 3.3186971707677397}, {"type": "precision", "value": 3.198658632552104}, {"type": "recall", "value": 4.3478260869565215}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (est-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "est-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 6.9}, {"type": "f1", "value": 4.760708297894056}, {"type": "precision", "value": 4.28409511756074}, {"type": "recall", "value": 6.9}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (heb-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "heb-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 2.1999999999999997}, {"type": "f1", "value": 1.6862703878117107}, {"type": "precision", "value": 1.6048118233915603}, {"type": "recall", "value": 2.1999999999999997}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (gla-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "gla-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 3.0156815440289506}, {"type": "f1", "value": 2.0913257250659134}, {"type": "precision", "value": 1.9072775486461648}, {"type": "recall", "value": 3.0156815440289506}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (mar-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "mar-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 49.0}, {"type": "f1", "value": 45.5254456536713}, {"type": "precision", "value": 44.134609250398725}, {"type": "recall", "value": 49.0}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (lat-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "lat-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 33.5}, {"type": "f1", "value": 28.759893973182564}, {"type": "precision", "value": 27.401259116024836}, {"type": "recall", "value": 33.5}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (bel-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "bel-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 10.2}, {"type": "f1", "value": 8.030039981676275}, {"type": "precision", "value": 7.548748077210127}, {"type": "recall", "value": 10.2}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (pms-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "pms-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 38.095238095238095}, {"type": "f1", "value": 31.944999250262406}, {"type": "precision", "value": 30.04452690166976}, {"type": "recall", "value": 38.095238095238095}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (gle-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "gle-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 4.8}, {"type": "f1", "value": 3.2638960786708067}, {"type": "precision", "value": 3.0495382950729644}, {"type": "recall", "value": 4.8}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (pes-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "pes-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 15.8}, {"type": "f1", "value": 12.131087470371275}, {"type": "precision", "value": 11.141304011547815}, {"type": "recall", "value": 15.8}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (nob-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "nob-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 23.3}, {"type": "f1", "value": 21.073044636921384}, {"type": "precision", "value": 20.374220568287285}, {"type": "recall", "value": 23.3}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (bul-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "bul-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 24.9}, {"type": "f1", "value": 20.091060685364987}, {"type": "precision", "value": 18.899700591081224}, {"type": "recall", "value": 24.9}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (cbk-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "cbk-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 70.1}, {"type": "f1", "value": 64.62940836940835}, {"type": "precision", "value": 62.46559523809524}, {"type": "recall", "value": 70.1}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (hun-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "hun-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 7.199999999999999}, {"type": "f1", "value": 5.06613460576115}, {"type": "precision", "value": 4.625224463391809}, {"type": "recall", "value": 7.199999999999999}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (uig-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "uig-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 1.7999999999999998}, {"type": "f1", "value": 1.2716249514772895}, {"type": "precision", "value": 1.2107445914723798}, {"type": "recall", "value": 1.7999999999999998}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (rus-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "rus-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 65.5}, {"type": "f1", "value": 59.84399711399712}, {"type": "precision", "value": 57.86349567099567}, {"type": "recall", "value": 65.5}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (spa-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "spa-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 95.7}, {"type": "f1", "value": 94.48333333333333}, {"type": "precision", "value": 93.89999999999999}, {"type": "recall", "value": 95.7}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (hye-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "hye-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 0.8086253369272237}, {"type": "f1", "value": 0.4962046191492002}, {"type": "precision", "value": 0.47272438578554393}, {"type": "recall", "value": 0.8086253369272237}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (tel-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "tel-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 69.23076923076923}, {"type": "f1", "value": 64.6227941099736}, {"type": "precision", "value": 63.03795877325289}, {"type": "recall", "value": 69.23076923076923}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (afr-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "afr-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 20.599999999999998}, {"type": "f1", "value": 16.62410040660465}, {"type": "precision", "value": 15.598352437967069}, {"type": "recall", "value": 20.599999999999998}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (mon-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "mon-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 4.318181818181818}, {"type": "f1", "value": 2.846721192535661}, {"type": "precision", "value": 2.6787861417537147}, {"type": "recall", "value": 4.318181818181818}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (arz-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "arz-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 74.84276729559748}, {"type": "f1", "value": 70.6638714185884}, {"type": "precision", "value": 68.86792452830188}, {"type": "recall", "value": 74.84276729559748}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (hrv-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "hrv-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 15.9}, {"type": "f1", "value": 12.793698974586706}, {"type": "precision", "value": 12.088118017657736}, {"type": "recall", "value": 15.9}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (nov-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "nov-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 59.92217898832685}, {"type": "f1", "value": 52.23086900129701}, {"type": "precision", "value": 49.25853869433636}, {"type": "recall", "value": 59.92217898832685}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (gsw-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "gsw-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 27.350427350427353}, {"type": "f1", "value": 21.033781033781032}, {"type": "precision", "value": 19.337955491801644}, {"type": "recall", "value": 27.350427350427353}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (nds-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "nds-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 29.299999999999997}, {"type": "f1", "value": 23.91597452425777}, {"type": "precision", "value": 22.36696598364942}, {"type": "recall", "value": 29.299999999999997}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (ukr-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "ukr-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 27.3}, {"type": "f1", "value": 22.059393517688886}, {"type": "precision", "value": 20.503235534170887}, {"type": "recall", "value": 27.3}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (uzb-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "uzb-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 8.177570093457943}, {"type": "f1", "value": 4.714367017906037}, {"type": "precision", "value": 4.163882933965758}, {"type": "recall", "value": 8.177570093457943}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (lit-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "lit-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 5.800000000000001}, {"type": "f1", "value": 4.4859357432293825}, {"type": "precision", "value": 4.247814465614043}, {"type": "recall", "value": 5.800000000000001}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (ina-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "ina-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 78.4}, {"type": "f1", "value": 73.67166666666667}, {"type": "precision", "value": 71.83285714285714}, {"type": "recall", "value": 78.4}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (lfn-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "lfn-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 50.3}, {"type": "f1", "value": 44.85221545883311}, {"type": "precision", "value": 43.04913026243909}, {"type": "recall", "value": 50.3}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (zsm-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "zsm-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 83.5}, {"type": "f1", "value": 79.95151515151515}, {"type": "precision", "value": 78.53611111111111}, {"type": "recall", "value": 83.5}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (ita-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "ita-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 69.89999999999999}, {"type": "f1", "value": 65.03756269256269}, {"type": "precision", "value": 63.233519536019536}, {"type": "recall", "value": 69.89999999999999}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (cmn-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "cmn-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 93.2}, {"type": "f1", "value": 91.44666666666666}, {"type": "precision", "value": 90.63333333333333}, {"type": "recall", "value": 93.2}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (lvs-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "lvs-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 8.3}, {"type": "f1", "value": 6.553388144729963}, {"type": "precision", "value": 6.313497782829976}, {"type": "recall", "value": 8.3}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (glg-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "glg-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 83.6}, {"type": "f1", "value": 79.86243107769424}, {"type": "precision", "value": 78.32555555555555}, {"type": "recall", "value": 83.6}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (ceb-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "ceb-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 9.166666666666666}, {"type": "f1", "value": 6.637753604420271}, {"type": "precision", "value": 6.10568253585495}, {"type": "recall", "value": 9.166666666666666}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (bre-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "bre-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 7.3999999999999995}, {"type": "f1", "value": 4.6729483612322165}, {"type": "precision", "value": 4.103844520292658}, {"type": "recall", "value": 7.3999999999999995}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (ben-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "ben-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 80.30000000000001}, {"type": "f1", "value": 75.97666666666667}, {"type": "precision", "value": 74.16}, {"type": "recall", "value": 80.30000000000001}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (swg-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "swg-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 23.214285714285715}, {"type": "f1", "value": 16.88988095238095}, {"type": "precision", "value": 15.364937641723353}, {"type": "recall", "value": 23.214285714285715}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (arq-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "arq-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 33.15038419319429}, {"type": "f1", "value": 27.747873024072415}, {"type": "precision", "value": 25.99320572578704}, {"type": "recall", "value": 33.15038419319429}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (kab-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "kab-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 2.6}, {"type": "f1", "value": 1.687059048752127}, {"type": "precision", "value": 1.5384884521299}, {"type": "recall", "value": 2.6}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (fra-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "fra-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 93.30000000000001}, {"type": "f1", "value": 91.44000000000001}, {"type": "precision", "value": 90.59166666666667}, {"type": "recall", "value": 93.30000000000001}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (por-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "por-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 94.1}, {"type": "f1", "value": 92.61666666666667}, {"type": "precision", "value": 91.88333333333333}, {"type": "recall", "value": 94.1}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (tat-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "tat-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 5.0}, {"type": "f1", "value": 3.589591971281927}, {"type": "precision", "value": 3.3046491614532854}, {"type": "recall", "value": 5.0}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (oci-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "oci-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 45.9}, {"type": "f1", "value": 40.171969141969136}, {"type": "precision", "value": 38.30764368870302}, {"type": "recall", "value": 45.9}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (pol-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "pol-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 16.900000000000002}, {"type": "f1", "value": 14.094365204207351}, {"type": "precision", "value": 13.276519841269844}, {"type": "recall", "value": 16.900000000000002}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (war-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "war-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 12.8}, {"type": "f1", "value": 10.376574912567156}, {"type": "precision", "value": 9.758423963284509}, {"type": "recall", "value": 12.8}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (aze-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "aze-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 8.1}, {"type": "f1", "value": 6.319455355175778}, {"type": "precision", "value": 5.849948830628881}, {"type": "recall", "value": 8.1}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (vie-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "vie-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 95.5}, {"type": "f1", "value": 94.19666666666667}, {"type": "precision", "value": 93.60000000000001}, {"type": "recall", "value": 95.5}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (nno-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "nno-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 19.1}, {"type": "f1", "value": 16.280080686081906}, {"type": "precision", "value": 15.451573089395668}, {"type": "recall", "value": 19.1}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (cha-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "cha-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 30.656934306569344}, {"type": "f1", "value": 23.2568647897115}, {"type": "precision", "value": 21.260309034031664}, {"type": "recall", "value": 30.656934306569344}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (mhr-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "mhr-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 2.1999999999999997}, {"type": "f1", "value": 1.556861047295521}, {"type": "precision", "value": 1.4555993437238521}, {"type": "recall", "value": 2.1999999999999997}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (dan-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "dan-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 27.500000000000004}, {"type": "f1", "value": 23.521682636223492}, {"type": "precision", "value": 22.345341306967683}, {"type": "recall", "value": 27.500000000000004}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (ell-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "ell-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 7.3999999999999995}, {"type": "f1", "value": 5.344253880846173}, {"type": "precision", "value": 4.999794279068863}, {"type": "recall", "value": 7.3999999999999995}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (amh-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "amh-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 0.5952380952380952}, {"type": "f1", "value": 0.026455026455026457}, {"type": "precision", "value": 0.013528138528138528}, {"type": "recall", "value": 0.5952380952380952}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (pam-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "pam-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 7.3}, {"type": "f1", "value": 5.853140211779251}, {"type": "precision", "value": 5.505563080945322}, {"type": "recall", "value": 7.3}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (hsb-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "hsb-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 13.250517598343686}, {"type": "f1", "value": 9.676349506190704}, {"type": "precision", "value": 8.930392053553216}, {"type": "recall", "value": 13.250517598343686}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (srp-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "srp-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 14.499999999999998}, {"type": "f1", "value": 11.68912588067557}, {"type": "precision", "value": 11.024716513105519}, {"type": "recall", "value": 14.499999999999998}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (epo-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "epo-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 30.099999999999998}, {"type": "f1", "value": 26.196880936315146}, {"type": "precision", "value": 25.271714086169478}, {"type": "recall", "value": 30.099999999999998}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (kzj-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "kzj-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 6.4}, {"type": "f1", "value": 5.1749445942023335}, {"type": "precision", "value": 4.975338142029625}, {"type": "recall", "value": 6.4}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (awa-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "awa-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 39.39393939393939}, {"type": "f1", "value": 35.005707393767096}, {"type": "precision", "value": 33.64342032053631}, {"type": "recall", "value": 39.39393939393939}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (fao-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "fao-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 18.3206106870229}, {"type": "f1", "value": 12.610893447220345}, {"type": "precision", "value": 11.079228765297467}, {"type": "recall", "value": 18.3206106870229}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (mal-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "mal-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 85.58951965065502}, {"type": "f1", "value": 83.30363944928548}, {"type": "precision", "value": 82.40026591554977}, {"type": "recall", "value": 85.58951965065502}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (ile-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "ile-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 65.7}, {"type": "f1", "value": 59.589642857142856}, {"type": "precision", "value": 57.392826797385624}, {"type": "recall", "value": 65.7}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (bos-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "bos-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 18.07909604519774}, {"type": "f1", "value": 13.65194306689995}, {"type": "precision", "value": 12.567953943826327}, {"type": "recall", "value": 18.07909604519774}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (cor-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "cor-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 4.6}, {"type": "f1", "value": 2.8335386392505013}, {"type": "precision", "value": 2.558444143575722}, {"type": "recall", "value": 4.6}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (cat-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "cat-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 90.7}, {"type": "f1", "value": 88.30666666666666}, {"type": "precision", "value": 87.195}, {"type": "recall", "value": 90.7}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (eus-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "eus-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 57.699999999999996}, {"type": "f1", "value": 53.38433067253876}, {"type": "precision", "value": 51.815451335350346}, {"type": "recall", "value": 57.699999999999996}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (yue-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "yue-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 80.60000000000001}, {"type": "f1", "value": 77.0290354090354}, {"type": "precision", "value": 75.61685897435898}, {"type": "recall", "value": 80.60000000000001}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (swe-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "swe-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 24.6}, {"type": "f1", "value": 19.52814960069739}, {"type": "precision", "value": 18.169084599880502}, {"type": "recall", "value": 24.6}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (dtp-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "dtp-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 5.0}, {"type": "f1", "value": 3.4078491753102376}, {"type": "precision", "value": 3.1757682319102387}, {"type": "recall", "value": 5.0}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (kat-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "kat-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 1.2064343163538873}, {"type": "f1", "value": 0.4224313053283095}, {"type": "precision", "value": 0.3360484946842894}, {"type": "recall", "value": 1.2064343163538873}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (jpn-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "jpn-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 76.1}, {"type": "f1", "value": 71.36246031746032}, {"type": "precision", "value": 69.5086544011544}, {"type": "recall", "value": 76.1}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (csb-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "csb-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 14.229249011857709}, {"type": "f1", "value": 10.026578603653704}, {"type": "precision", "value": 9.09171178352764}, {"type": "recall", "value": 14.229249011857709}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (xho-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "xho-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 8.450704225352112}, {"type": "f1", "value": 5.51214407186151}, {"type": "precision", "value": 4.928281812084629}, {"type": "recall", "value": 8.450704225352112}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (orv-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "orv-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 7.664670658682635}, {"type": "f1", "value": 5.786190079917295}, {"type": "precision", "value": 5.3643643579244}, {"type": "recall", "value": 7.664670658682635}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (ind-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "ind-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 90.5}, {"type": "f1", "value": 88.03999999999999}, {"type": "precision", "value": 86.94833333333334}, {"type": "recall", "value": 90.5}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (tuk-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "tuk-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 7.389162561576355}, {"type": "f1", "value": 5.482366349556517}, {"type": "precision", "value": 5.156814449917898}, {"type": "recall", "value": 7.389162561576355}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (max-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "max-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 41.54929577464789}, {"type": "f1", "value": 36.13520282534367}, {"type": "precision", "value": 34.818226488560995}, {"type": "recall", "value": 41.54929577464789}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (swh-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "swh-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 20.76923076923077}, {"type": "f1", "value": 16.742497560177643}, {"type": "precision", "value": 15.965759712090138}, {"type": "recall", "value": 20.76923076923077}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (hin-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "hin-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 88.1}, {"type": "f1", "value": 85.23176470588236}, {"type": "precision", "value": 84.04458333333334}, {"type": "recall", "value": 88.1}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (dsb-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "dsb-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 11.899791231732777}, {"type": "f1", "value": 8.776706659565102}, {"type": "precision", "value": 8.167815946521582}, {"type": "recall", "value": 11.899791231732777}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (ber-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "ber-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 6.1}, {"type": "f1", "value": 4.916589537178435}, {"type": "precision", "value": 4.72523017415345}, {"type": "recall", "value": 6.1}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (tam-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "tam-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 76.54723127035831}, {"type": "f1", "value": 72.75787187839306}, {"type": "precision", "value": 71.43338442869005}, {"type": "recall", "value": 76.54723127035831}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (slk-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "slk-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 11.700000000000001}, {"type": "f1", "value": 9.975679190026007}, {"type": "precision", "value": 9.569927715653522}, {"type": "recall", "value": 11.700000000000001}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (tgl-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "tgl-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 13.100000000000001}, {"type": "f1", "value": 10.697335850115408}, {"type": "precision", "value": 10.113816082086341}, {"type": "recall", "value": 13.100000000000001}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (ast-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "ast-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 76.37795275590551}, {"type": "f1", "value": 71.12860892388451}, {"type": "precision", "value": 68.89763779527559}, {"type": "recall", "value": 76.37795275590551}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (mkd-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "mkd-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 13.700000000000001}, {"type": "f1", "value": 10.471861684067568}, {"type": "precision", "value": 9.602902567641697}, {"type": "recall", "value": 13.700000000000001}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (khm-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "khm-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 0.554016620498615}, {"type": "f1", "value": 0.37034084643642423}, {"type": "precision", "value": 0.34676040281208437}, {"type": "recall", "value": 0.554016620498615}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (ces-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "ces-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 12.4}, {"type": "f1", "value": 9.552607451092534}, {"type": "precision", "value": 8.985175505050504}, {"type": "recall", "value": 12.4}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (tzl-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "tzl-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 33.65384615384615}, {"type": "f1", "value": 27.820512820512818}, {"type": "precision", "value": 26.09432234432234}, {"type": "recall", "value": 33.65384615384615}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (urd-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "urd-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 74.5}, {"type": "f1", "value": 70.09686507936507}, {"type": "precision", "value": 68.3117857142857}, {"type": "recall", "value": 74.5}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (ara-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "ara-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 88.3}, {"type": "f1", "value": 85.37333333333333}, {"type": "precision", "value": 84.05833333333334}, {"type": "recall", "value": 88.3}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (kor-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "kor-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 25.0}, {"type": "f1", "value": 22.393124632031995}, {"type": "precision", "value": 21.58347686592367}, {"type": "recall", "value": 25.0}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (yid-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "yid-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 0.589622641509434}, {"type": "f1", "value": 0.15804980033762941}, {"type": "precision", "value": 0.1393275384872965}, {"type": "recall", "value": 0.589622641509434}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (fin-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "fin-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 4.1000000000000005}, {"type": "f1", "value": 3.4069011332551775}, {"type": "precision", "value": 3.1784507042253516}, {"type": "recall", "value": 4.1000000000000005}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (tha-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "tha-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 3.102189781021898}, {"type": "f1", "value": 2.223851811694751}, {"type": "precision", "value": 2.103465682299194}, {"type": "recall", "value": 3.102189781021898}]}, {"task": {"type": "BitextMining"}, "dataset": {"name": "MTEB Tatoeba (wuu-eng)", "type": "mteb/tatoeba-bitext-mining", "config": "wuu-eng", "split": "test", "revision": "ed9e4a974f867fd9736efcf222fc3a26487387a5"}, "metrics": [{"type": "accuracy", "value": 83.1}, {"type": "f1", "value": 79.58255835667599}, {"type": "precision", "value": 78.09708333333333}, {"type": "recall", "value": 83.1}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB Touche2020", "type": "webis-touche2020", "config": "default", "split": "test", "revision": "527b7d77e16e343303e68cb6af11d6e18b9f7b3b"}, "metrics": [{"type": "map_at_1", "value": 2.322}, {"type": "map_at_10", "value": 8.959999999999999}, {"type": "map_at_100", "value": 15.136}, {"type": "map_at_1000", "value": 16.694}, {"type": "map_at_3", "value": 4.837000000000001}, {"type": "map_at_5", "value": 6.196}, {"type": "mrr_at_1", "value": 28.571}, {"type": "mrr_at_10", "value": 47.589999999999996}, {"type": "mrr_at_100", "value": 48.166}, {"type": "mrr_at_1000", "value": 48.169000000000004}, {"type": "mrr_at_3", "value": 43.197}, {"type": "mrr_at_5", "value": 45.646}, {"type": "ndcg_at_1", "value": 26.531}, {"type": "ndcg_at_10", "value": 23.982}, {"type": "ndcg_at_100", "value": 35.519}, {"type": "ndcg_at_1000", "value": 46.878}, {"type": "ndcg_at_3", "value": 26.801000000000002}, {"type": "ndcg_at_5", "value": 24.879}, {"type": "precision_at_1", "value": 28.571}, {"type": "precision_at_10", "value": 22.041}, {"type": "precision_at_100", "value": 7.4079999999999995}, {"type": "precision_at_1000", "value": 1.492}, {"type": "precision_at_3", "value": 28.571}, {"type": "precision_at_5", "value": 25.306}, {"type": "recall_at_1", "value": 2.322}, {"type": "recall_at_10", "value": 15.443999999999999}, {"type": "recall_at_100", "value": 45.918}, {"type": "recall_at_1000", "value": 79.952}, {"type": "recall_at_3", "value": 6.143}, {"type": "recall_at_5", "value": 8.737}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB ToxicConversationsClassification", "type": "mteb/toxic_conversations_50k", "config": "default", "split": "test", "revision": "edfaf9da55d3dd50d43143d90c1ac476895ae6de"}, "metrics": [{"type": "accuracy", "value": 66.5452}, {"type": "ap", "value": 12.99191723223892}, {"type": "f1", "value": 51.667665096195734}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB TweetSentimentExtractionClassification", "type": "mteb/tweet_sentiment_extraction", "config": "default", "split": "test", "revision": "62146448f05be9e52a36b8ee9936447ea787eede"}, "metrics": [{"type": "accuracy", "value": 55.854555744199196}, {"type": "f1", "value": 56.131766302254185}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB TwentyNewsgroupsClustering", "type": "mteb/twentynewsgroups-clustering", "config": "default", "split": "test", "revision": "091a54f9a36281ce7d6590ec8c75dd485e7e01d4"}, "metrics": [{"type": "v_measure", "value": 37.27891385518074}]}, {"task": {"type": "PairClassification"}, "dataset": {"name": "MTEB TwitterSemEval2015", "type": "mteb/twittersemeval2015-pairclassification", "config": "default", "split": "test", "revision": "70970daeab8776df92f5ea462b6173c0b46fd2d1"}, "metrics": [{"type": "cos_sim_accuracy", "value": 83.53102461703523}, {"type": "cos_sim_ap", "value": 65.30753664579191}, {"type": "cos_sim_f1", "value": 61.739943872778305}, {"type": "cos_sim_precision", "value": 55.438891222175556}, {"type": "cos_sim_recall", "value": 69.65699208443272}, {"type": "dot_accuracy", "value": 80.38981939560112}, {"type": "dot_ap", "value": 53.52081118421347}, {"type": "dot_f1", "value": 54.232957844617346}, {"type": "dot_precision", "value": 48.43393486828459}, {"type": "dot_recall", "value": 61.60949868073878}, {"type": "euclidean_accuracy", "value": 82.23758717291531}, {"type": "euclidean_ap", "value": 60.361102792772535}, {"type": "euclidean_f1", "value": 57.50518791791561}, {"type": "euclidean_precision", "value": 51.06470106470107}, {"type": "euclidean_recall", "value": 65.8047493403694}, {"type": "manhattan_accuracy", "value": 82.14221851344102}, {"type": "manhattan_ap", "value": 60.341937223793366}, {"type": "manhattan_f1", "value": 57.53803596127247}, {"type": "manhattan_precision", "value": 51.08473188702415}, {"type": "manhattan_recall", "value": 65.85751978891821}, {"type": "max_accuracy", "value": 83.53102461703523}, {"type": "max_ap", "value": 65.30753664579191}, {"type": "max_f1", "value": 61.739943872778305}]}, {"task": {"type": "PairClassification"}, "dataset": {"name": "MTEB TwitterURLCorpus", "type": "mteb/twitterurlcorpus-pairclassification", "config": "default", "split": "test", "revision": "8b6510b0b1fa4e4c4f879467980e9be563ec1cdf"}, "metrics": [{"type": "cos_sim_accuracy", "value": 88.75305623471883}, {"type": "cos_sim_ap", "value": 85.46387153880272}, {"type": "cos_sim_f1", "value": 77.91527673159008}, {"type": "cos_sim_precision", "value": 72.93667315828353}, {"type": "cos_sim_recall", "value": 83.62334462580844}, {"type": "dot_accuracy", "value": 85.08169363915086}, {"type": "dot_ap", "value": 74.96808060965559}, {"type": "dot_f1", "value": 71.39685033990366}, {"type": "dot_precision", "value": 64.16948111759288}, {"type": "dot_recall", "value": 80.45888512473051}, {"type": "euclidean_accuracy", "value": 85.84235650250321}, {"type": "euclidean_ap", "value": 78.42045145247211}, {"type": "euclidean_f1", "value": 70.32669630775179}, {"type": "euclidean_precision", "value": 70.6298050788227}, {"type": "euclidean_recall", "value": 70.02617801047121}, {"type": "manhattan_accuracy", "value": 85.86176116738464}, {"type": "manhattan_ap", "value": 78.54012451558276}, {"type": "manhattan_f1", "value": 70.56508080693389}, {"type": "manhattan_precision", "value": 69.39626293456413}, {"type": "manhattan_recall", "value": 71.77394518016631}, {"type": "max_accuracy", "value": 88.75305623471883}, {"type": "max_ap", "value": 85.46387153880272}, {"type": "max_f1", "value": 77.91527673159008}]}]}]}
sentence-transformers pytorch bloom feature-extraction sentence-similarity mteb arxiv:2202.08904 model-index endpoints_compatible has_space region:us
TAGS
#sentence-transformers #pytorch #bloom #feature-extraction #sentence-similarity #mteb #arxiv-2202.08904 #model-index #endpoints_compatible #has_space #region-us 


## Usage

For usage instructions, refer to: URL

The model was trained with the command


## Evaluation Results


'{"ndcgs": {"sgpt-bloom-7b1-msmarco": {"scifact": {"NDCG@10": 0.71824}, "nfcorpus": {"NDCG@10": 0.35748}, "arguana": {"NDCG@10": 0.47281}, "scidocs": {"NDCG@10": 0.18435}, "fiqa": {"NDCG@10": 0.35736}, "cqadupstack": {"NDCG@10": 0.3708525}, "quora": {"NDCG@10": 0.74655}, "trec-covid": {"NDCG@10": 0.82731}, "webis-touche2020": {"NDCG@10": 0.2365}}}'

See the evaluation folder or MTEB for more results.

## Training
The model was trained with the parameters:

DataLoader:

'URL.dataloader.DataLoader' of length 15600 with parameters:


The model uses BitFit, weighted-mean pooling & GradCache, for details see: URL

Loss:

'sentence_transformers.losses.MultipleNegativesRankingLoss.MNRLGradCache' 

Parameters of the fit()-Method:



## Full Model Architecture


## Citing & Authors


## Usage

For usage instructions, refer to: URL

The model was trained with the command ## Evaluation Results


'{"ndcgs": {"sgpt-bloom-7b1-msmarco": {"scifact": {"NDCG@10": 0.71824}, "nfcorpus": {"NDCG@10": 0.35748}, "arguana": {"NDCG@10": 0.47281}, "scidocs": {"NDCG@10": 0.18435}, "fiqa": {"NDCG@10": 0.35736}, "cqadupstack": {"NDCG@10": 0.3708525}, "quora": {"NDCG@10": 0.74655}, "trec-covid": {"NDCG@10": 0.82731}, "webis-touche2020": {"NDCG@10": 0.2365}}}'

See the evaluation folder or MTEB for more results. ## Training
The model was trained with the parameters:

DataLoader:

'URL.dataloader.DataLoader' of length 15600 with parameters:


The model uses BitFit, weighted-mean pooling & GradCache, for details see: URL

Loss:

'sentence_transformers.losses.MultipleNegativesRankingLoss.MNRLGradCache' 

Parameters of the fit()-Method: ## Full Model Architecture ## Citing & Authors
TAGS
#sentence-transformers #pytorch #bloom #feature-extraction #sentence-similarity #mteb #arxiv-2202.08904 #model-index #endpoints_compatible #has_space #region-us 
 ## Usage

For usage instructions, refer to: URL

The model was trained with the command ## Evaluation Results


'{"ndcgs": {"sgpt-bloom-7b1-msmarco": {"scifact": {"NDCG@10": 0.71824}, "nfcorpus": {"NDCG@10": 0.35748}, "arguana": {"NDCG@10": 0.47281}, "scidocs": {"NDCG@10": 0.18435}, "fiqa": {"NDCG@10": 0.35736}, "cqadupstack": {"NDCG@10": 0.3708525}, "quora": {"NDCG@10": 0.74655}, "trec-covid": {"NDCG@10": 0.82731}, "webis-touche2020": {"NDCG@10": 0.2365}}}'

See the evaluation folder or MTEB for more results. ## Training
The model was trained with the parameters:

DataLoader:

'URL.dataloader.DataLoader' of length 15600 with parameters:


The model uses BitFit, weighted-mean pooling & GradCache, for details see: URL

Loss:

'sentence_transformers.losses.MultipleNegativesRankingLoss.MNRLGradCache' 

Parameters of the fit()-Method: ## Full Model Architecture ## Citing & Authors
passage: TAGS
#sentence-transformers #pytorch #bloom #feature-extraction #sentence-similarity #mteb #arxiv-2202.08904 #model-index #endpoints_compatible #has_space #region-us 
## Usage

For usage instructions, refer to: URL

The model was trained with the command## Evaluation Results


'{"ndcgs": {"sgpt-bloom-7b1-msmarco": {"scifact": {"NDCG@10": 0.71824}, "nfcorpus": {"NDCG@10": 0.35748}, "arguana": {"NDCG@10": 0.47281}, "scidocs": {"NDCG@10": 0.18435}, "fiqa": {"NDCG@10": 0.35736}, "cqadupstack": {"NDCG@10": 0.3708525}, "quora": {"NDCG@10": 0.74655}, "trec-covid": {"NDCG@10": 0.82731}, "webis-touche2020": {"NDCG@10": 0.2365}}}'

See the evaluation folder or MTEB for more results.## Training
The model was trained with the parameters:

DataLoader:

'URL.dataloader.DataLoader' of length 15600 with parameters:


The model uses BitFit, weighted-mean pooling & GradCache, for details see: URL

Loss:

'sentence_transformers.losses.MultipleNegativesRankingLoss.MNRLGradCache' 

Parameters of the fit()-Method:## Full Model Architecture## Citing & Authors