
# SGPT-125M-weightedmean-msmarco-specb-bitfit

## Usage

For usage instructions, refer to our codebase: https://github.com/Muennighoff/sgpt 

## Evaluation Results

For eval results, refer to the eval folder or our paper: https://arxiv.org/abs/2202.08904


## Training
The model was trained with the parameters:

**DataLoader**:

`torch.utils.data.dataloader.DataLoader` of length 15600 with parameters:
```
{'batch_size': 32, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}
```

**Loss**:

`sentence_transformers.losses.MultipleNegativesRankingLoss.MultipleNegativesRankingLoss` with parameters:
  ```
  {'scale': 20.0, 'similarity_fct': 'cos_sim'}
  ```

Parameters of the fit()-Method:
```
{
    "epochs": 10,
    "evaluation_steps": 0,
    "evaluator": "NoneType",
    "max_grad_norm": 1,
    "optimizer_class": "<class 'transformers.optimization.AdamW'>",
    "optimizer_params": {
        "lr": 0.0002
    },
    "scheduler": "WarmupLinear",
    "steps_per_epoch": null,
    "warmup_steps": 1000,
    "weight_decay": 0.01
}
```


## Full Model Architecture
```
SentenceTransformer(
  (0): Transformer({'max_seq_length': 300, 'do_lower_case': False}) with Transformer model: GPTNeoModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': True, 'pooling_mode_lasttoken': False})
)
```

## Citing & Authors

```bibtex
@article{muennighoff2022sgpt,
  title={SGPT: GPT Sentence Embeddings for Semantic Search},
  author={Muennighoff, Niklas},
  journal={arXiv preprint arXiv:2202.08904},
  year={2022}
}
```

{"tags": ["sentence-transformers", "feature-extraction", "sentence-similarity", "mteb"], "pipeline_tag": "sentence-similarity", "model-index": [{"name": "SGPT-125M-weightedmean-msmarco-specb-bitfit", "results": [{"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonCounterfactualClassification (en)", "type": "mteb/amazon_counterfactual", "config": "en", "split": "test", "revision": "2d8a100785abf0ae21420d2a55b0c56e3e1ea996"}, "metrics": [{"type": "accuracy", "value": 61.23880597014926}, {"type": "ap", "value": 25.854431650388644}, {"type": "f1", "value": 55.751862762818604}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonCounterfactualClassification (de)", "type": "mteb/amazon_counterfactual", "config": "de", "split": "test", "revision": "2d8a100785abf0ae21420d2a55b0c56e3e1ea996"}, "metrics": [{"type": "accuracy", "value": 56.88436830835117}, {"type": "ap", "value": 72.67279104379772}, {"type": "f1", "value": 54.449840243786404}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonCounterfactualClassification (en-ext)", "type": "mteb/amazon_counterfactual", "config": "en-ext", "split": "test", "revision": "2d8a100785abf0ae21420d2a55b0c56e3e1ea996"}, "metrics": [{"type": "accuracy", "value": 58.27586206896551}, {"type": "ap", "value": 14.067357642500387}, {"type": "f1", "value": 48.172318518691334}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonCounterfactualClassification (ja)", "type": "mteb/amazon_counterfactual", "config": "ja", "split": "test", "revision": "2d8a100785abf0ae21420d2a55b0c56e3e1ea996"}, "metrics": [{"type": "accuracy", "value": 54.64668094218415}, {"type": "ap", "value": 11.776694555054965}, {"type": "f1", "value": 44.526622834078765}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonPolarityClassification", "type": "mteb/amazon_polarity", "config": "default", "split": "test", "revision": "80714f8dcf8cefc218ef4f8c5a966dd83f75a0e1"}, "metrics": [{"type": "accuracy", "value": 65.401225}, {"type": "ap", "value": 60.22809958678552}, {"type": "f1", "value": 65.0251824898292}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonReviewsClassification (en)", "type": "mteb/amazon_reviews_multi", "config": "en", "split": "test", "revision": "c379a6705fec24a2493fa68e011692605f44e119"}, "metrics": [{"type": "accuracy", "value": 31.165999999999993}, {"type": "f1", "value": 30.908870050167437}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonReviewsClassification (de)", "type": "mteb/amazon_reviews_multi", "config": "de", "split": "test", "revision": "c379a6705fec24a2493fa68e011692605f44e119"}, "metrics": [{"type": "accuracy", "value": 24.79}, {"type": "f1", "value": 24.5833598854121}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonReviewsClassification (es)", "type": "mteb/amazon_reviews_multi", "config": "es", "split": "test", "revision": "c379a6705fec24a2493fa68e011692605f44e119"}, "metrics": [{"type": "accuracy", "value": 26.643999999999995}, {"type": "f1", "value": 26.39012792213563}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonReviewsClassification (fr)", "type": "mteb/amazon_reviews_multi", "config": "fr", "split": "test", "revision": "c379a6705fec24a2493fa68e011692605f44e119"}, "metrics": [{"type": "accuracy", "value": 26.386000000000003}, {"type": "f1", "value": 26.276867791454873}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonReviewsClassification (ja)", "type": "mteb/amazon_reviews_multi", "config": "ja", "split": "test", "revision": "c379a6705fec24a2493fa68e011692605f44e119"}, "metrics": [{"type": "accuracy", "value": 22.078000000000003}, {"type": "f1", "value": 21.797960290226843}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB AmazonReviewsClassification (zh)", "type": "mteb/amazon_reviews_multi", "config": "zh", "split": "test", "revision": "c379a6705fec24a2493fa68e011692605f44e119"}, "metrics": [{"type": "accuracy", "value": 24.274}, {"type": "f1", "value": 23.887054434822627}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB ArguAna", "type": "arguana", "config": "default", "split": "test", "revision": "5b3e3697907184a9b77a3c99ee9ea1a9cbb1e4e3"}, "metrics": [{"type": "map_at_1", "value": 22.404}, {"type": "map_at_10", "value": 36.845}, {"type": "map_at_100", "value": 37.945}, {"type": "map_at_1000", "value": 37.966}, {"type": "map_at_3", "value": 31.78}, {"type": "map_at_5", "value": 34.608}, {"type": "mrr_at_1", "value": 22.902}, {"type": "mrr_at_10", "value": 37.034}, {"type": "mrr_at_100", "value": 38.134}, {"type": "mrr_at_1000", "value": 38.155}, {"type": "mrr_at_3", "value": 31.935000000000002}, {"type": "mrr_at_5", "value": 34.812}, {"type": "ndcg_at_1", "value": 22.404}, {"type": "ndcg_at_10", "value": 45.425}, {"type": "ndcg_at_100", "value": 50.354}, {"type": "ndcg_at_1000", "value": 50.873999999999995}, {"type": "ndcg_at_3", "value": 34.97}, {"type": "ndcg_at_5", "value": 40.081}, {"type": "precision_at_1", "value": 22.404}, {"type": "precision_at_10", "value": 7.303999999999999}, {"type": "precision_at_100", "value": 0.951}, {"type": "precision_at_1000", "value": 0.099}, {"type": "precision_at_3", "value": 14.746}, {"type": "precision_at_5", "value": 11.337}, {"type": "recall_at_1", "value": 22.404}, {"type": "recall_at_10", "value": 73.044}, {"type": "recall_at_100", "value": 95.092}, {"type": "recall_at_1000", "value": 99.075}, {"type": "recall_at_3", "value": 44.239}, {"type": "recall_at_5", "value": 56.686}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB ArxivClusteringP2P", "type": "mteb/arxiv-clustering-p2p", "config": "default", "split": "test", "revision": "0bbdb47bcbe3a90093699aefeed338a0f28a7ee8"}, "metrics": [{"type": "v_measure", "value": 39.70858340673288}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB ArxivClusteringS2S", "type": "mteb/arxiv-clustering-s2s", "config": "default", "split": "test", "revision": "b73bd54100e5abfa6e3a23dcafb46fe4d2438dc3"}, "metrics": [{"type": "v_measure", "value": 28.242847713721048}]}, {"task": {"type": "Reranking"}, "dataset": {"name": "MTEB AskUbuntuDupQuestions", "type": "mteb/askubuntudupquestions-reranking", "config": "default", "split": "test", "revision": "4d853f94cd57d85ec13805aeeac3ae3e5eb4c49c"}, "metrics": [{"type": "map", "value": 55.83700395192393}, {"type": "mrr", "value": 70.3891307215407}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB BIOSSES", "type": "mteb/biosses-sts", "config": "default", "split": "test", "revision": "9ee918f184421b6bd48b78f6c714d86546106103"}, "metrics": [{"type": "cos_sim_pearson", "value": 79.25366801756223}, {"type": "cos_sim_spearman", "value": 75.20954502580506}, {"type": "euclidean_pearson", "value": 78.79900722991617}, {"type": "euclidean_spearman", "value": 77.79996549607588}, {"type": "manhattan_pearson", "value": 78.18408109480399}, {"type": "manhattan_spearman", "value": 76.85958262303106}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB Banking77Classification", "type": "mteb/banking77", "config": "default", "split": "test", "revision": "44fa15921b4c889113cc5df03dd4901b49161ab7"}, "metrics": [{"type": "accuracy", "value": 77.70454545454545}, {"type": "f1", "value": 77.6929000113803}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB BiorxivClusteringP2P", "type": "mteb/biorxiv-clustering-p2p", "config": "default", "split": "test", "revision": "11d0121201d1f1f280e8cc8f3d98fb9c4d9f9c55"}, "metrics": [{"type": "v_measure", "value": 33.63260395543984}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB BiorxivClusteringS2S", "type": "mteb/biorxiv-clustering-s2s", "config": "default", "split": "test", "revision": "c0fab014e1bcb8d3a5e31b2088972a1e01547dc1"}, "metrics": [{"type": "v_measure", "value": 27.038042665369925}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB CQADupstackAndroidRetrieval", "type": "BeIR/cqadupstack", "config": "default", "split": "test", "revision": "2b9f5791698b5be7bc5e10535c8690f20043c3db"}, "metrics": [{"type": "map_at_1", "value": 22.139}, {"type": "map_at_10", "value": 28.839}, {"type": "map_at_100", "value": 30.023}, {"type": "map_at_1000", "value": 30.153000000000002}, {"type": "map_at_3", "value": 26.521}, {"type": "map_at_5", "value": 27.775}, {"type": "mrr_at_1", "value": 26.466}, {"type": "mrr_at_10", "value": 33.495000000000005}, {"type": "mrr_at_100", "value": 34.416999999999994}, {"type": "mrr_at_1000", "value": 34.485}, {"type": "mrr_at_3", "value": 31.402}, {"type": "mrr_at_5", "value": 32.496}, {"type": "ndcg_at_1", "value": 26.466}, {"type": "ndcg_at_10", "value": 33.372}, {"type": "ndcg_at_100", "value": 38.7}, {"type": "ndcg_at_1000", "value": 41.696}, {"type": "ndcg_at_3", "value": 29.443}, {"type": "ndcg_at_5", "value": 31.121}, {"type": "precision_at_1", "value": 26.466}, {"type": "precision_at_10", "value": 6.037}, {"type": "precision_at_100", "value": 1.0670000000000002}, {"type": "precision_at_1000", "value": 0.16199999999999998}, {"type": "precision_at_3", "value": 13.782}, {"type": "precision_at_5", "value": 9.757}, {"type": "recall_at_1", "value": 22.139}, {"type": "recall_at_10", "value": 42.39}, {"type": "recall_at_100", "value": 65.427}, {"type": "recall_at_1000", "value": 86.04899999999999}, {"type": "recall_at_3", "value": 31.127}, {"type": "recall_at_5", "value": 35.717999999999996}, {"type": "map_at_1", "value": 20.652}, {"type": "map_at_10", "value": 27.558}, {"type": "map_at_100", "value": 28.473}, {"type": "map_at_1000", "value": 28.577}, {"type": "map_at_3", "value": 25.402}, {"type": "map_at_5", "value": 26.68}, {"type": "mrr_at_1", "value": 25.223000000000003}, {"type": "mrr_at_10", "value": 31.966}, {"type": "mrr_at_100", "value": 32.664}, {"type": "mrr_at_1000", "value": 32.724}, {"type": "mrr_at_3", "value": 30.074}, {"type": "mrr_at_5", "value": 31.249}, {"type": "ndcg_at_1", "value": 25.223000000000003}, {"type": "ndcg_at_10", "value": 31.694}, {"type": "ndcg_at_100", "value": 35.662}, {"type": "ndcg_at_1000", "value": 38.092}, {"type": "ndcg_at_3", "value": 28.294000000000004}, {"type": "ndcg_at_5", "value": 30.049}, {"type": "precision_at_1", "value": 25.223000000000003}, {"type": "precision_at_10", "value": 5.777}, {"type": "precision_at_100", "value": 0.9730000000000001}, {"type": "precision_at_1000", "value": 0.13999999999999999}, {"type": "precision_at_3", "value": 13.397}, {"type": "precision_at_5", "value": 9.605}, {"type": "recall_at_1", "value": 20.652}, {"type": "recall_at_10", "value": 39.367999999999995}, {"type": "recall_at_100", "value": 56.485}, {"type": "recall_at_1000", "value": 73.292}, {"type": "recall_at_3", "value": 29.830000000000002}, {"type": "recall_at_5", "value": 34.43}, {"type": "map_at_1", "value": 25.180000000000003}, {"type": "map_at_10", "value": 34.579}, {"type": "map_at_100", "value": 35.589999999999996}, {"type": "map_at_1000", "value": 35.68}, {"type": "map_at_3", "value": 31.735999999999997}, {"type": "map_at_5", "value": 33.479}, {"type": "mrr_at_1", "value": 29.467}, {"type": "mrr_at_10", "value": 37.967}, {"type": "mrr_at_100", "value": 38.800000000000004}, {"type": "mrr_at_1000", "value": 38.858}, {"type": "mrr_at_3", "value": 35.465}, {"type": "mrr_at_5", "value": 37.057}, {"type": "ndcg_at_1", "value": 29.467}, {"type": "ndcg_at_10", "value": 39.796}, {"type": "ndcg_at_100", "value": 44.531}, {"type": "ndcg_at_1000", "value": 46.666000000000004}, {"type": "ndcg_at_3", "value": 34.676}, {"type": "ndcg_at_5", "value": 37.468}, {"type": "precision_at_1", "value": 29.467}, {"type": "precision_at_10", "value": 6.601999999999999}, {"type": "precision_at_100", "value": 0.9900000000000001}, {"type": "precision_at_1000", "value": 0.124}, {"type": "precision_at_3", "value": 15.568999999999999}, {"type": "precision_at_5", "value": 11.172}, {"type": "recall_at_1", "value": 25.180000000000003}, {"type": "recall_at_10", "value": 52.269}, {"type": "recall_at_100", "value": 73.574}, {"type": "recall_at_1000", "value": 89.141}, {"type": "recall_at_3", "value": 38.522}, {"type": "recall_at_5", "value": 45.323}, {"type": "map_at_1", "value": 16.303}, {"type": "map_at_10", "value": 21.629}, {"type": "map_at_100", "value": 22.387999999999998}, {"type": "map_at_1000", "value": 22.489}, {"type": "map_at_3", "value": 19.608}, {"type": "map_at_5", "value": 20.774}, {"type": "mrr_at_1", "value": 17.740000000000002}, {"type": "mrr_at_10", "value": 23.214000000000002}, {"type": "mrr_at_100", "value": 23.97}, {"type": "mrr_at_1000", "value": 24.054000000000002}, {"type": "mrr_at_3", "value": 21.243000000000002}, {"type": "mrr_at_5", "value": 22.322}, {"type": "ndcg_at_1", "value": 17.740000000000002}, {"type": "ndcg_at_10", "value": 25.113000000000003}, {"type": "ndcg_at_100", "value": 29.287999999999997}, {"type": "ndcg_at_1000", "value": 32.204}, {"type": "ndcg_at_3", "value": 21.111}, {"type": "ndcg_at_5", "value": 23.061999999999998}, {"type": "precision_at_1", "value": 17.740000000000002}, {"type": "precision_at_10", "value": 3.955}, {"type": "precision_at_100", "value": 0.644}, {"type": "precision_at_1000", "value": 0.093}, {"type": "precision_at_3", "value": 8.851}, {"type": "precision_at_5", "value": 6.418}, {"type": "recall_at_1", "value": 16.303}, {"type": "recall_at_10", "value": 34.487}, {"type": "recall_at_100", "value": 54.413999999999994}, {"type": "recall_at_1000", "value": 77.158}, {"type": "recall_at_3", "value": 23.733}, {"type": "recall_at_5", "value": 28.381}, {"type": "map_at_1", "value": 10.133000000000001}, {"type": "map_at_10", "value": 15.665999999999999}, {"type": "map_at_100", "value": 16.592000000000002}, {"type": "map_at_1000", "value": 16.733999999999998}, {"type": "map_at_3", "value": 13.625000000000002}, {"type": "map_at_5", "value": 14.721}, {"type": "mrr_at_1", "value": 12.562000000000001}, {"type": "mrr_at_10", "value": 18.487000000000002}, {"type": "mrr_at_100", "value": 19.391}, {"type": "mrr_at_1000", "value": 19.487}, {"type": "mrr_at_3", "value": 16.418}, {"type": "mrr_at_5", "value": 17.599999999999998}, {"type": "ndcg_at_1", "value": 12.562000000000001}, {"type": "ndcg_at_10", "value": 19.43}, {"type": "ndcg_at_100", "value": 24.546}, {"type": "ndcg_at_1000", "value": 28.193}, {"type": "ndcg_at_3", "value": 15.509999999999998}, {"type": "ndcg_at_5", "value": 17.322000000000003}, {"type": "precision_at_1", "value": 12.562000000000001}, {"type": "precision_at_10", "value": 3.794}, {"type": "precision_at_100", "value": 0.74}, {"type": "precision_at_1000", "value": 0.122}, {"type": "precision_at_3", "value": 7.546}, {"type": "precision_at_5", "value": 5.721}, {"type": "recall_at_1", "value": 10.133000000000001}, {"type": "recall_at_10", "value": 28.261999999999997}, {"type": "recall_at_100", "value": 51.742999999999995}, {"type": "recall_at_1000", "value": 78.075}, {"type": "recall_at_3", "value": 17.634}, {"type": "recall_at_5", "value": 22.128999999999998}, {"type": "map_at_1", "value": 19.991999999999997}, {"type": "map_at_10", "value": 27.346999999999998}, {"type": "map_at_100", "value": 28.582}, {"type": "map_at_1000", "value": 28.716}, {"type": "map_at_3", "value": 24.907}, {"type": "map_at_5", "value": 26.1}, {"type": "mrr_at_1", "value": 23.773}, {"type": "mrr_at_10", "value": 31.647}, {"type": "mrr_at_100", "value": 32.639}, {"type": "mrr_at_1000", "value": 32.706}, {"type": "mrr_at_3", "value": 29.195}, {"type": "mrr_at_5", "value": 30.484}, {"type": "ndcg_at_1", "value": 23.773}, {"type": "ndcg_at_10", "value": 32.322}, {"type": "ndcg_at_100", "value": 37.996}, {"type": "ndcg_at_1000", "value": 40.819}, {"type": "ndcg_at_3", "value": 27.876}, {"type": "ndcg_at_5", "value": 29.664}, {"type": "precision_at_1", "value": 23.773}, {"type": "precision_at_10", "value": 5.976999999999999}, {"type": "precision_at_100", "value": 1.055}, {"type": "precision_at_1000", "value": 0.15}, {"type": "precision_at_3", "value": 13.122}, {"type": "precision_at_5", "value": 9.451}, {"type": "recall_at_1", "value": 19.991999999999997}, {"type": "recall_at_10", "value": 43.106}, {"type": "recall_at_100", "value": 67.264}, {"type": "recall_at_1000", "value": 86.386}, {"type": "recall_at_3", "value": 30.392000000000003}, {"type": "recall_at_5", "value": 34.910999999999994}, {"type": "map_at_1", "value": 17.896}, {"type": "map_at_10", "value": 24.644}, {"type": "map_at_100", "value": 25.790000000000003}, {"type": "map_at_1000", "value": 25.913999999999998}, {"type": "map_at_3", "value": 22.694}, {"type": "map_at_5", "value": 23.69}, {"type": "mrr_at_1", "value": 21.346999999999998}, {"type": "mrr_at_10", "value": 28.594}, {"type": "mrr_at_100", "value": 29.543999999999997}, {"type": "mrr_at_1000", "value": 29.621}, {"type": "mrr_at_3", "value": 26.807}, {"type": "mrr_at_5", "value": 27.669}, {"type": "ndcg_at_1", "value": 21.346999999999998}, {"type": "ndcg_at_10", "value": 28.833}, {"type": "ndcg_at_100", "value": 34.272000000000006}, {"type": "ndcg_at_1000", "value": 37.355}, {"type": "ndcg_at_3", "value": 25.373}, {"type": "ndcg_at_5", "value": 26.756}, {"type": "precision_at_1", "value": 21.346999999999998}, {"type": "precision_at_10", "value": 5.2170000000000005}, {"type": "precision_at_100", "value": 0.954}, {"type": "precision_at_1000", "value": 0.13899999999999998}, {"type": "precision_at_3", "value": 11.948}, {"type": "precision_at_5", "value": 8.425}, {"type": "recall_at_1", "value": 17.896}, {"type": "recall_at_10", "value": 37.291000000000004}, {"type": "recall_at_100", "value": 61.138000000000005}, {"type": "recall_at_1000", "value": 83.212}, {"type": "recall_at_3", "value": 27.705999999999996}, {"type": "recall_at_5", "value": 31.234}, {"type": "map_at_1", "value": 17.195166666666665}, {"type": "map_at_10", "value": 23.329083333333333}, {"type": "map_at_100", "value": 24.30308333333333}, {"type": "map_at_1000", "value": 24.422416666666667}, {"type": "map_at_3", "value": 21.327416666666664}, {"type": "map_at_5", "value": 22.419999999999998}, {"type": "mrr_at_1", "value": 19.999916666666667}, {"type": "mrr_at_10", "value": 26.390166666666666}, {"type": "mrr_at_100", "value": 27.230999999999998}, {"type": "mrr_at_1000", "value": 27.308333333333334}, {"type": "mrr_at_3", "value": 24.4675}, {"type": "mrr_at_5", "value": 25.541083333333336}, {"type": "ndcg_at_1", "value": 19.999916666666667}, {"type": "ndcg_at_10", "value": 27.248666666666665}, {"type": "ndcg_at_100", "value": 32.00258333333334}, {"type": "ndcg_at_1000", "value": 34.9465}, {"type": "ndcg_at_3", "value": 23.58566666666667}, {"type": "ndcg_at_5", "value": 25.26341666666666}, {"type": "precision_at_1", "value": 19.999916666666667}, {"type": "precision_at_10", "value": 4.772166666666666}, {"type": "precision_at_100", "value": 0.847}, {"type": "precision_at_1000", "value": 0.12741666666666668}, {"type": "precision_at_3", "value": 10.756166666666669}, {"type": "precision_at_5", "value": 7.725416666666667}, {"type": "recall_at_1", "value": 17.195166666666665}, {"type": "recall_at_10", "value": 35.99083333333334}, {"type": "recall_at_100", "value": 57.467999999999996}, {"type": "recall_at_1000", "value": 78.82366666666667}, {"type": "recall_at_3", "value": 25.898499999999995}, {"type": "recall_at_5", "value": 30.084333333333333}, {"type": "map_at_1", "value": 16.779}, {"type": "map_at_10", "value": 21.557000000000002}, {"type": "map_at_100", "value": 22.338}, {"type": "map_at_1000", "value": 22.421}, {"type": "map_at_3", "value": 19.939}, {"type": "map_at_5", "value": 20.903}, {"type": "mrr_at_1", "value": 18.404999999999998}, {"type": "mrr_at_10", "value": 23.435}, {"type": "mrr_at_100", "value": 24.179000000000002}, {"type": "mrr_at_1000", "value": 24.25}, {"type": "mrr_at_3", "value": 21.907}, {"type": "mrr_at_5", "value": 22.781000000000002}, {"type": "ndcg_at_1", "value": 18.404999999999998}, {"type": "ndcg_at_10", "value": 24.515}, {"type": "ndcg_at_100", "value": 28.721000000000004}, {"type": "ndcg_at_1000", "value": 31.259999999999998}, {"type": "ndcg_at_3", "value": 21.508}, {"type": "ndcg_at_5", "value": 23.01}, {"type": "precision_at_1", "value": 18.404999999999998}, {"type": "precision_at_10", "value": 3.834}, {"type": "precision_at_100", "value": 0.641}, {"type": "precision_at_1000", "value": 0.093}, {"type": "precision_at_3", "value": 9.151}, {"type": "precision_at_5", "value": 6.503}, {"type": "recall_at_1", "value": 16.779}, {"type": "recall_at_10", "value": 31.730000000000004}, {"type": "recall_at_100", "value": 51.673}, {"type": "recall_at_1000", "value": 71.17599999999999}, {"type": "recall_at_3", "value": 23.518}, {"type": "recall_at_5", "value": 27.230999999999998}, {"type": "map_at_1", "value": 9.279}, {"type": "map_at_10", "value": 13.822000000000001}, {"type": "map_at_100", "value": 14.533}, {"type": "map_at_1000", "value": 14.649999999999999}, {"type": "map_at_3", "value": 12.396}, {"type": "map_at_5", "value": 13.214}, {"type": "mrr_at_1", "value": 11.149000000000001}, {"type": "mrr_at_10", "value": 16.139}, {"type": "mrr_at_100", "value": 16.872}, {"type": "mrr_at_1000", "value": 16.964000000000002}, {"type": "mrr_at_3", "value": 14.613000000000001}, {"type": "mrr_at_5", "value": 15.486}, {"type": "ndcg_at_1", "value": 11.149000000000001}, {"type": "ndcg_at_10", "value": 16.82}, {"type": "ndcg_at_100", "value": 20.73}, {"type": "ndcg_at_1000", "value": 23.894000000000002}, {"type": "ndcg_at_3", "value": 14.11}, {"type": "ndcg_at_5", "value": 15.404000000000002}, {"type": "precision_at_1", "value": 11.149000000000001}, {"type": "precision_at_10", "value": 3.063}, {"type": "precision_at_100", "value": 0.587}, {"type": "precision_at_1000", "value": 0.1}, {"type": "precision_at_3", "value": 6.699}, {"type": "precision_at_5", "value": 4.928}, {"type": "recall_at_1", "value": 9.279}, {"type": "recall_at_10", "value": 23.745}, {"type": "recall_at_100", "value": 41.873}, {"type": "recall_at_1000", "value": 64.982}, {"type": "recall_at_3", "value": 16.152}, {"type": "recall_at_5", "value": 19.409000000000002}, {"type": "map_at_1", "value": 16.36}, {"type": "map_at_10", "value": 21.927}, {"type": "map_at_100", "value": 22.889}, {"type": "map_at_1000", "value": 22.994}, {"type": "map_at_3", "value": 20.433}, {"type": "map_at_5", "value": 21.337}, {"type": "mrr_at_1", "value": 18.75}, {"type": "mrr_at_10", "value": 24.859}, {"type": "mrr_at_100", "value": 25.746999999999996}, {"type": "mrr_at_1000", "value": 25.829}, {"type": "mrr_at_3", "value": 23.383000000000003}, {"type": "mrr_at_5", "value": 24.297}, {"type": "ndcg_at_1", "value": 18.75}, {"type": "ndcg_at_10", "value": 25.372}, {"type": "ndcg_at_100", "value": 30.342999999999996}, {"type": "ndcg_at_1000", "value": 33.286}, {"type": "ndcg_at_3", "value": 22.627}, {"type": "ndcg_at_5", "value": 24.04}, {"type": "precision_at_1", "value": 18.75}, {"type": "precision_at_10", "value": 4.1419999999999995}, {"type": "precision_at_100", "value": 0.738}, {"type": "precision_at_1000", "value": 0.11100000000000002}, {"type": "precision_at_3", "value": 10.261000000000001}, {"type": "precision_at_5", "value": 7.164}, {"type": "recall_at_1", "value": 16.36}, {"type": "recall_at_10", "value": 32.949}, {"type": "recall_at_100", "value": 55.552}, {"type": "recall_at_1000", "value": 77.09899999999999}, {"type": "recall_at_3", "value": 25.538}, {"type": "recall_at_5", "value": 29.008}, {"type": "map_at_1", "value": 17.39}, {"type": "map_at_10", "value": 23.058}, {"type": "map_at_100", "value": 24.445}, {"type": "map_at_1000", "value": 24.637999999999998}, {"type": "map_at_3", "value": 21.037}, {"type": "map_at_5", "value": 21.966}, {"type": "mrr_at_1", "value": 19.96}, {"type": "mrr_at_10", "value": 26.301000000000002}, {"type": "mrr_at_100", "value": 27.297}, {"type": "mrr_at_1000", "value": 27.375}, {"type": "mrr_at_3", "value": 24.340999999999998}, {"type": "mrr_at_5", "value": 25.339}, {"type": "ndcg_at_1", "value": 19.96}, {"type": "ndcg_at_10", "value": 27.249000000000002}, {"type": "ndcg_at_100", "value": 32.997}, {"type": "ndcg_at_1000", "value": 36.359}, {"type": "ndcg_at_3", "value": 23.519000000000002}, {"type": "ndcg_at_5", "value": 24.915000000000003}, {"type": "precision_at_1", "value": 19.96}, {"type": "precision_at_10", "value": 5.356000000000001}, {"type": "precision_at_100", "value": 1.198}, {"type": "precision_at_1000", "value": 0.20400000000000001}, {"type": "precision_at_3", "value": 10.738}, {"type": "precision_at_5", "value": 7.904999999999999}, {"type": "recall_at_1", "value": 17.39}, {"type": "recall_at_10", "value": 35.254999999999995}, {"type": "recall_at_100", "value": 61.351}, {"type": "recall_at_1000", "value": 84.395}, {"type": "recall_at_3", "value": 25.194}, {"type": "recall_at_5", "value": 28.546}, {"type": "map_at_1", "value": 14.238999999999999}, {"type": "map_at_10", "value": 19.323}, {"type": "map_at_100", "value": 19.994}, {"type": "map_at_1000", "value": 20.102999999999998}, {"type": "map_at_3", "value": 17.631}, {"type": "map_at_5", "value": 18.401}, {"type": "mrr_at_1", "value": 15.157000000000002}, {"type": "mrr_at_10", "value": 20.578}, {"type": "mrr_at_100", "value": 21.252}, {"type": "mrr_at_1000", "value": 21.346999999999998}, {"type": "mrr_at_3", "value": 18.762}, {"type": "mrr_at_5", "value": 19.713}, {"type": "ndcg_at_1", "value": 15.157000000000002}, {"type": "ndcg_at_10", "value": 22.468}, {"type": "ndcg_at_100", "value": 26.245}, {"type": "ndcg_at_1000", "value": 29.534}, {"type": "ndcg_at_3", "value": 18.981}, {"type": "ndcg_at_5", "value": 20.349999999999998}, {"type": "precision_at_1", "value": 15.157000000000002}, {"type": "precision_at_10", "value": 3.512}, {"type": "precision_at_100", "value": 0.577}, {"type": "precision_at_1000", "value": 0.091}, {"type": "precision_at_3", "value": 8.01}, {"type": "precision_at_5", "value": 5.656}, {"type": "recall_at_1", "value": 14.238999999999999}, {"type": "recall_at_10", "value": 31.038}, {"type": "recall_at_100", "value": 49.122}, {"type": "recall_at_1000", "value": 74.919}, {"type": "recall_at_3", "value": 21.436}, {"type": "recall_at_5", "value": 24.692}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB ClimateFEVER", "type": "climate-fever", "config": "default", "split": "test", "revision": "392b78eb68c07badcd7c2cd8f39af108375dfcce"}, "metrics": [{"type": "map_at_1", "value": 8.828}, {"type": "map_at_10", "value": 14.982000000000001}, {"type": "map_at_100", "value": 16.495}, {"type": "map_at_1000", "value": 16.658}, {"type": "map_at_3", "value": 12.366000000000001}, {"type": "map_at_5", "value": 13.655000000000001}, {"type": "mrr_at_1", "value": 19.088}, {"type": "mrr_at_10", "value": 29.29}, {"type": "mrr_at_100", "value": 30.291}, {"type": "mrr_at_1000", "value": 30.342000000000002}, {"type": "mrr_at_3", "value": 25.907000000000004}, {"type": "mrr_at_5", "value": 27.840999999999998}, {"type": "ndcg_at_1", "value": 19.088}, {"type": "ndcg_at_10", "value": 21.858}, {"type": "ndcg_at_100", "value": 28.323999999999998}, {"type": "ndcg_at_1000", "value": 31.561}, {"type": "ndcg_at_3", "value": 17.175}, {"type": "ndcg_at_5", "value": 18.869}, {"type": "precision_at_1", "value": 19.088}, {"type": "precision_at_10", "value": 6.9190000000000005}, {"type": "precision_at_100", "value": 1.376}, {"type": "precision_at_1000", "value": 0.197}, {"type": "precision_at_3", "value": 12.703999999999999}, {"type": "precision_at_5", "value": 9.993}, {"type": "recall_at_1", "value": 8.828}, {"type": "recall_at_10", "value": 27.381}, {"type": "recall_at_100", "value": 50.0}, {"type": "recall_at_1000", "value": 68.355}, {"type": "recall_at_3", "value": 16.118}, {"type": "recall_at_5", "value": 20.587}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB DBPedia", "type": "dbpedia-entity", "config": "default", "split": "test", "revision": "f097057d03ed98220bc7309ddb10b71a54d667d6"}, "metrics": [{"type": "map_at_1", "value": 5.586}, {"type": "map_at_10", "value": 10.040000000000001}, {"type": "map_at_100", "value": 12.55}, {"type": "map_at_1000", "value": 13.123999999999999}, {"type": "map_at_3", "value": 7.75}, {"type": "map_at_5", "value": 8.835999999999999}, {"type": "mrr_at_1", "value": 42.25}, {"type": "mrr_at_10", "value": 51.205999999999996}, {"type": "mrr_at_100", "value": 51.818}, {"type": "mrr_at_1000", "value": 51.855}, {"type": "mrr_at_3", "value": 48.875}, {"type": "mrr_at_5", "value": 50.488}, {"type": "ndcg_at_1", "value": 32.25}, {"type": "ndcg_at_10", "value": 22.718}, {"type": "ndcg_at_100", "value": 24.359}, {"type": "ndcg_at_1000", "value": 29.232000000000003}, {"type": "ndcg_at_3", "value": 25.974000000000004}, {"type": "ndcg_at_5", "value": 24.291999999999998}, {"type": "precision_at_1", "value": 42.25}, {"type": "precision_at_10", "value": 17.75}, {"type": "precision_at_100", "value": 5.032}, {"type": "precision_at_1000", "value": 1.117}, {"type": "precision_at_3", "value": 28.833}, {"type": "precision_at_5", "value": 24.25}, {"type": "recall_at_1", "value": 5.586}, {"type": "recall_at_10", "value": 14.16}, {"type": "recall_at_100", "value": 28.051}, {"type": "recall_at_1000", "value": 45.157000000000004}, {"type": "recall_at_3", "value": 8.758000000000001}, {"type": "recall_at_5", "value": 10.975999999999999}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB EmotionClassification", "type": "mteb/emotion", "config": "default", "split": "test", "revision": "829147f8f75a25f005913200eb5ed41fae320aa1"}, "metrics": [{"type": "accuracy", "value": 39.075}, {"type": "f1", "value": 35.01420354708222}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB FEVER", "type": "fever", "config": "default", "split": "test", "revision": "1429cf27e393599b8b359b9b72c666f96b2525f9"}, "metrics": [{"type": "map_at_1", "value": 43.519999999999996}, {"type": "map_at_10", "value": 54.368}, {"type": "map_at_100", "value": 54.918}, {"type": "map_at_1000", "value": 54.942}, {"type": "map_at_3", "value": 51.712}, {"type": "map_at_5", "value": 53.33599999999999}, {"type": "mrr_at_1", "value": 46.955000000000005}, {"type": "mrr_at_10", "value": 58.219}, {"type": "mrr_at_100", "value": 58.73500000000001}, {"type": "mrr_at_1000", "value": 58.753}, {"type": "mrr_at_3", "value": 55.518}, {"type": "mrr_at_5", "value": 57.191}, {"type": "ndcg_at_1", "value": 46.955000000000005}, {"type": "ndcg_at_10", "value": 60.45}, {"type": "ndcg_at_100", "value": 63.047}, {"type": "ndcg_at_1000", "value": 63.712999999999994}, {"type": "ndcg_at_3", "value": 55.233}, {"type": "ndcg_at_5", "value": 58.072}, {"type": "precision_at_1", "value": 46.955000000000005}, {"type": "precision_at_10", "value": 8.267}, {"type": "precision_at_100", "value": 0.962}, {"type": "precision_at_1000", "value": 0.10300000000000001}, {"type": "precision_at_3", "value": 22.326999999999998}, {"type": "precision_at_5", "value": 14.940999999999999}, {"type": "recall_at_1", "value": 43.519999999999996}, {"type": "recall_at_10", "value": 75.632}, {"type": "recall_at_100", "value": 87.41600000000001}, {"type": "recall_at_1000", "value": 92.557}, {"type": "recall_at_3", "value": 61.597}, {"type": "recall_at_5", "value": 68.518}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB FiQA2018", "type": "fiqa", "config": "default", "split": "test", "revision": "41b686a7f28c59bcaaa5791efd47c67c8ebe28be"}, "metrics": [{"type": "map_at_1", "value": 9.549000000000001}, {"type": "map_at_10", "value": 15.762}, {"type": "map_at_100", "value": 17.142}, {"type": "map_at_1000", "value": 17.329}, {"type": "map_at_3", "value": 13.575000000000001}, {"type": "map_at_5", "value": 14.754000000000001}, {"type": "mrr_at_1", "value": 19.753}, {"type": "mrr_at_10", "value": 26.568}, {"type": "mrr_at_100", "value": 27.606}, {"type": "mrr_at_1000", "value": 27.68}, {"type": "mrr_at_3", "value": 24.203}, {"type": "mrr_at_5", "value": 25.668999999999997}, {"type": "ndcg_at_1", "value": 19.753}, {"type": "ndcg_at_10", "value": 21.118000000000002}, {"type": "ndcg_at_100", "value": 27.308}, {"type": "ndcg_at_1000", "value": 31.304}, {"type": "ndcg_at_3", "value": 18.319}, {"type": "ndcg_at_5", "value": 19.414}, {"type": "precision_at_1", "value": 19.753}, {"type": "precision_at_10", "value": 6.08}, {"type": "precision_at_100", "value": 1.204}, {"type": "precision_at_1000", "value": 0.192}, {"type": "precision_at_3", "value": 12.191}, {"type": "precision_at_5", "value": 9.383}, {"type": "recall_at_1", "value": 9.549000000000001}, {"type": "recall_at_10", "value": 26.131}, {"type": "recall_at_100", "value": 50.544999999999995}, {"type": "recall_at_1000", "value": 74.968}, {"type": "recall_at_3", "value": 16.951}, {"type": "recall_at_5", "value": 20.95}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB HotpotQA", "type": "hotpotqa", "config": "default", "split": "test", "revision": "766870b35a1b9ca65e67a0d1913899973551fc6c"}, "metrics": [{"type": "map_at_1", "value": 25.544}, {"type": "map_at_10", "value": 32.62}, {"type": "map_at_100", "value": 33.275}, {"type": "map_at_1000", "value": 33.344}, {"type": "map_at_3", "value": 30.851}, {"type": "map_at_5", "value": 31.868999999999996}, {"type": "mrr_at_1", "value": 51.087}, {"type": "mrr_at_10", "value": 57.704}, {"type": "mrr_at_100", "value": 58.175}, {"type": "mrr_at_1000", "value": 58.207}, {"type": "mrr_at_3", "value": 56.106}, {"type": "mrr_at_5", "value": 57.074000000000005}, {"type": "ndcg_at_1", "value": 51.087}, {"type": "ndcg_at_10", "value": 40.876000000000005}, {"type": "ndcg_at_100", "value": 43.762}, {"type": "ndcg_at_1000", "value": 45.423}, {"type": "ndcg_at_3", "value": 37.65}, {"type": "ndcg_at_5", "value": 39.305}, {"type": "precision_at_1", "value": 51.087}, {"type": "precision_at_10", "value": 8.304}, {"type": "precision_at_100", "value": 1.059}, {"type": "precision_at_1000", "value": 0.128}, {"type": "precision_at_3", "value": 22.875999999999998}, {"type": "precision_at_5", "value": 15.033}, {"type": "recall_at_1", "value": 25.544}, {"type": "recall_at_10", "value": 41.519}, {"type": "recall_at_100", "value": 52.957}, {"type": "recall_at_1000", "value": 64.132}, {"type": "recall_at_3", "value": 34.315}, {"type": "recall_at_5", "value": 37.583}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB ImdbClassification", "type": "mteb/imdb", "config": "default", "split": "test", "revision": "8d743909f834c38949e8323a8a6ce8721ea6c7f4"}, "metrics": [{"type": "accuracy", "value": 58.6696}, {"type": "ap", "value": 55.3644880984279}, {"type": "f1", "value": 58.07942097405652}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB MSMARCO", "type": "msmarco", "config": "default", "split": "validation", "revision": "e6838a846e2408f22cf5cc337ebc83e0bcf77849"}, "metrics": [{"type": "map_at_1", "value": 14.442}, {"type": "map_at_10", "value": 22.932}, {"type": "map_at_100", "value": 24.132}, {"type": "map_at_1000", "value": 24.213}, {"type": "map_at_3", "value": 20.002}, {"type": "map_at_5", "value": 21.636}, {"type": "mrr_at_1", "value": 14.841999999999999}, {"type": "mrr_at_10", "value": 23.416}, {"type": "mrr_at_100", "value": 24.593999999999998}, {"type": "mrr_at_1000", "value": 24.669}, {"type": "mrr_at_3", "value": 20.494}, {"type": "mrr_at_5", "value": 22.14}, {"type": "ndcg_at_1", "value": 14.841999999999999}, {"type": "ndcg_at_10", "value": 27.975}, {"type": "ndcg_at_100", "value": 34.143}, {"type": "ndcg_at_1000", "value": 36.370000000000005}, {"type": "ndcg_at_3", "value": 21.944}, {"type": "ndcg_at_5", "value": 24.881}, {"type": "precision_at_1", "value": 14.841999999999999}, {"type": "precision_at_10", "value": 4.537}, {"type": "precision_at_100", "value": 0.767}, {"type": "precision_at_1000", "value": 0.096}, {"type": "precision_at_3", "value": 9.322}, {"type": "precision_at_5", "value": 7.074}, {"type": "recall_at_1", "value": 14.442}, {"type": "recall_at_10", "value": 43.557}, {"type": "recall_at_100", "value": 72.904}, {"type": "recall_at_1000", "value": 90.40700000000001}, {"type": "recall_at_3", "value": 27.088}, {"type": "recall_at_5", "value": 34.144000000000005}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPDomainClassification (en)", "type": "mteb/mtop_domain", "config": "en", "split": "test", "revision": "a7e2a951126a26fc8c6a69f835f33a346ba259e3"}, "metrics": [{"type": "accuracy", "value": 86.95622435020519}, {"type": "f1", "value": 86.58363130708494}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPDomainClassification (de)", "type": "mteb/mtop_domain", "config": "de", "split": "test", "revision": "a7e2a951126a26fc8c6a69f835f33a346ba259e3"}, "metrics": [{"type": "accuracy", "value": 62.73034657650043}, {"type": "f1", "value": 60.78623915840713}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPDomainClassification (es)", "type": "mteb/mtop_domain", "config": "es", "split": "test", "revision": "a7e2a951126a26fc8c6a69f835f33a346ba259e3"}, "metrics": [{"type": "accuracy", "value": 67.54503002001334}, {"type": "f1", "value": 65.34879794116112}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPDomainClassification (fr)", "type": "mteb/mtop_domain", "config": "fr", "split": "test", "revision": "a7e2a951126a26fc8c6a69f835f33a346ba259e3"}, "metrics": [{"type": "accuracy", "value": 65.35233322893829}, {"type": "f1", "value": 62.994001882446646}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPDomainClassification (hi)", "type": "mteb/mtop_domain", "config": "hi", "split": "test", "revision": "a7e2a951126a26fc8c6a69f835f33a346ba259e3"}, "metrics": [{"type": "accuracy", "value": 45.37110075295806}, {"type": "f1", "value": 44.26285860740745}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPDomainClassification (th)", "type": "mteb/mtop_domain", "config": "th", "split": "test", "revision": "a7e2a951126a26fc8c6a69f835f33a346ba259e3"}, "metrics": [{"type": "accuracy", "value": 55.276672694394215}, {"type": "f1", "value": 53.28388179869587}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPIntentClassification (en)", "type": "mteb/mtop_intent", "config": "en", "split": "test", "revision": "6299947a7777084cc2d4b64235bf7190381ce755"}, "metrics": [{"type": "accuracy", "value": 62.25262197902417}, {"type": "f1", "value": 43.44084037148853}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPIntentClassification (de)", "type": "mteb/mtop_intent", "config": "de", "split": "test", "revision": "6299947a7777084cc2d4b64235bf7190381ce755"}, "metrics": [{"type": "accuracy", "value": 49.56043956043956}, {"type": "f1", "value": 32.86333673498598}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPIntentClassification (es)", "type": "mteb/mtop_intent", "config": "es", "split": "test", "revision": "6299947a7777084cc2d4b64235bf7190381ce755"}, "metrics": [{"type": "accuracy", "value": 49.93995997331555}, {"type": "f1", "value": 34.726671876888126}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPIntentClassification (fr)", "type": "mteb/mtop_intent", "config": "fr", "split": "test", "revision": "6299947a7777084cc2d4b64235bf7190381ce755"}, "metrics": [{"type": "accuracy", "value": 46.32947071719386}, {"type": "f1", "value": 32.325273615982795}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPIntentClassification (hi)", "type": "mteb/mtop_intent", "config": "hi", "split": "test", "revision": "6299947a7777084cc2d4b64235bf7190381ce755"}, "metrics": [{"type": "accuracy", "value": 32.208676945141626}, {"type": "f1", "value": 21.32185122815139}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MTOPIntentClassification (th)", "type": "mteb/mtop_intent", "config": "th", "split": "test", "revision": "6299947a7777084cc2d4b64235bf7190381ce755"}, "metrics": [{"type": "accuracy", "value": 43.627486437613015}, {"type": "f1", "value": 27.04872922347508}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (af)", "type": "mteb/amazon_massive_intent", "config": "af", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 40.548083389374575}, {"type": "f1", "value": 39.490307545239716}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (am)", "type": "mteb/amazon_massive_intent", "config": "am", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 24.18291862811029}, {"type": "f1", "value": 23.437620034727473}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (ar)", "type": "mteb/amazon_massive_intent", "config": "ar", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 30.134498991257562}, {"type": "f1", "value": 28.787175191531283}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (az)", "type": "mteb/amazon_massive_intent", "config": "az", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 35.88433086751849}, {"type": "f1", "value": 36.264500398782126}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (bn)", "type": "mteb/amazon_massive_intent", "config": "bn", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 29.17283120376597}, {"type": "f1", "value": 27.8101616531901}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (cy)", "type": "mteb/amazon_massive_intent", "config": "cy", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 41.788836583725626}, {"type": "f1", "value": 39.71413181054801}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (da)", "type": "mteb/amazon_massive_intent", "config": "da", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 44.176193678547406}, {"type": "f1", "value": 42.192499826552286}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (de)", "type": "mteb/amazon_massive_intent", "config": "de", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 42.07464694014795}, {"type": "f1", "value": 39.44188259183162}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (el)", "type": "mteb/amazon_massive_intent", "config": "el", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 36.254203093476804}, {"type": "f1", "value": 34.46592715936761}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (en)", "type": "mteb/amazon_massive_intent", "config": "en", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 61.40887693342301}, {"type": "f1", "value": 59.79854802683996}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (es)", "type": "mteb/amazon_massive_intent", "config": "es", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 42.679892400807}, {"type": "f1", "value": 42.04801248338172}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (fa)", "type": "mteb/amazon_massive_intent", "config": "fa", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 35.59179556153329}, {"type": "f1", "value": 34.045862930486166}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (fi)", "type": "mteb/amazon_massive_intent", "config": "fi", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 40.036987222595826}, {"type": "f1", "value": 38.117703439362785}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (fr)", "type": "mteb/amazon_massive_intent", "config": "fr", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 43.43981170141224}, {"type": "f1", "value": 42.7084388987865}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (he)", "type": "mteb/amazon_massive_intent", "config": "he", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 31.593813046402154}, {"type": "f1", "value": 29.98550522450782}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (hi)", "type": "mteb/amazon_massive_intent", "config": "hi", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 27.044384667114997}, {"type": "f1", "value": 27.313059184832667}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (hu)", "type": "mteb/amazon_massive_intent", "config": "hu", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 38.453261600538}, {"type": "f1", "value": 37.309189326110435}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (hy)", "type": "mteb/amazon_massive_intent", "config": "hy", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 27.979152656355076}, {"type": "f1", "value": 27.430939684346445}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (id)", "type": "mteb/amazon_massive_intent", "config": "id", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 43.97108271687963}, {"type": "f1", "value": 43.40585705688761}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (is)", "type": "mteb/amazon_massive_intent", "config": "is", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 40.302622730329524}, {"type": "f1", "value": 39.108052180520744}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (it)", "type": "mteb/amazon_massive_intent", "config": "it", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 45.474108944182916}, {"type": "f1", "value": 45.85950328241134}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (ja)", "type": "mteb/amazon_massive_intent", "config": "ja", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 45.60860793544048}, {"type": "f1", "value": 43.94920708216737}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (jv)", "type": "mteb/amazon_massive_intent", "config": "jv", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 38.668459986550104}, {"type": "f1", "value": 37.6990034018859}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (ka)", "type": "mteb/amazon_massive_intent", "config": "ka", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 25.6523201075992}, {"type": "f1", "value": 25.279084273189582}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (km)", "type": "mteb/amazon_massive_intent", "config": "km", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 28.295225285810353}, {"type": "f1", "value": 26.645825638771548}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (kn)", "type": "mteb/amazon_massive_intent", "config": "kn", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 23.480161398789505}, {"type": "f1", "value": 22.275241866506732}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (ko)", "type": "mteb/amazon_massive_intent", "config": "ko", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 36.55682582380632}, {"type": "f1", "value": 36.004753171063605}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (lv)", "type": "mteb/amazon_massive_intent", "config": "lv", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 41.84936112979153}, {"type": "f1", "value": 41.38932672359119}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (ml)", "type": "mteb/amazon_massive_intent", "config": "ml", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 24.90921318090114}, {"type": "f1", "value": 23.968687483768807}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (mn)", "type": "mteb/amazon_massive_intent", "config": "mn", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 29.86213853396099}, {"type": "f1", "value": 29.977152075255407}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (ms)", "type": "mteb/amazon_massive_intent", "config": "ms", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 42.42098184263618}, {"type": "f1", "value": 41.50877432664628}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (my)", "type": "mteb/amazon_massive_intent", "config": "my", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 25.131136516476126}, {"type": "f1", "value": 23.938932214086776}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (nb)", "type": "mteb/amazon_massive_intent", "config": "nb", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 39.81506388702084}, {"type": "f1", "value": 38.809586587791664}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (nl)", "type": "mteb/amazon_massive_intent", "config": "nl", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 43.62138533960995}, {"type": "f1", "value": 42.01386842914633}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (pl)", "type": "mteb/amazon_massive_intent", "config": "pl", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 42.19569603227976}, {"type": "f1", "value": 40.00556559825827}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (pt)", "type": "mteb/amazon_massive_intent", "config": "pt", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 45.20847343644923}, {"type": "f1", "value": 44.24115005029051}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (ro)", "type": "mteb/amazon_massive_intent", "config": "ro", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 41.80901143241426}, {"type": "f1", "value": 40.474074848670085}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (ru)", "type": "mteb/amazon_massive_intent", "config": "ru", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 35.96839273705447}, {"type": "f1", "value": 35.095456843621}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (sl)", "type": "mteb/amazon_massive_intent", "config": "sl", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 40.60524546065905}, {"type": "f1", "value": 39.302383051500136}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (sq)", "type": "mteb/amazon_massive_intent", "config": "sq", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 42.75722932078009}, {"type": "f1", "value": 41.53763931497389}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (sv)", "type": "mteb/amazon_massive_intent", "config": "sv", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 42.347007397444514}, {"type": "f1", "value": 41.04366017948627}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (sw)", "type": "mteb/amazon_massive_intent", "config": "sw", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 41.12306657700067}, {"type": "f1", "value": 39.712940473289024}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (ta)", "type": "mteb/amazon_massive_intent", "config": "ta", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 24.603227975790183}, {"type": "f1", "value": 23.969236788828606}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (te)", "type": "mteb/amazon_massive_intent", "config": "te", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 25.03698722259583}, {"type": "f1", "value": 24.37196123281459}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (th)", "type": "mteb/amazon_massive_intent", "config": "th", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 35.40013449899126}, {"type": "f1", "value": 35.063600413688036}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (tl)", "type": "mteb/amazon_massive_intent", "config": "tl", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 41.19031607262945}, {"type": "f1", "value": 40.240432304273014}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (tr)", "type": "mteb/amazon_massive_intent", "config": "tr", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 36.405514458641555}, {"type": "f1", "value": 36.03844992856558}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (ur)", "type": "mteb/amazon_massive_intent", "config": "ur", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 25.934767989240076}, {"type": "f1", "value": 25.2074457023531}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (vi)", "type": "mteb/amazon_massive_intent", "config": "vi", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 38.79959650302622}, {"type": "f1", "value": 37.160233794673125}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (zh-CN)", "type": "mteb/amazon_massive_intent", "config": "zh-CN", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 46.244115669132476}, {"type": "f1", "value": 44.367480561291906}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveIntentClassification (zh-TW)", "type": "mteb/amazon_massive_intent", "config": "zh-TW", "split": "test", "revision": "072a486a144adf7f4479a4a0dddb2152e161e1ea"}, "metrics": [{"type": "accuracy", "value": 42.30665770006724}, {"type": "f1", "value": 41.9642223283514}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (af)", "type": "mteb/amazon_massive_scenario", "config": "af", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 43.2481506388702}, {"type": "f1", "value": 40.924230769590785}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (am)", "type": "mteb/amazon_massive_scenario", "config": "am", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 25.30262273032952}, {"type": "f1", "value": 24.937105830264066}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (ar)", "type": "mteb/amazon_massive_scenario", "config": "ar", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 32.07128446536651}, {"type": "f1", "value": 31.80245816594883}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (az)", "type": "mteb/amazon_massive_scenario", "config": "az", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 36.681237390719566}, {"type": "f1", "value": 36.37219042508338}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (bn)", "type": "mteb/amazon_massive_scenario", "config": "bn", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 29.56624075319435}, {"type": "f1", "value": 28.386042056362758}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (cy)", "type": "mteb/amazon_massive_scenario", "config": "cy", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 42.1049092131809}, {"type": "f1", "value": 38.926150886991294}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (da)", "type": "mteb/amazon_massive_scenario", "config": "da", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 45.44384667114997}, {"type": "f1", "value": 42.578252395460005}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (de)", "type": "mteb/amazon_massive_scenario", "config": "de", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 43.211163416274374}, {"type": "f1", "value": 41.04465858304789}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (el)", "type": "mteb/amazon_massive_scenario", "config": "el", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 36.503026227303295}, {"type": "f1", "value": 34.49785095312759}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (en)", "type": "mteb/amazon_massive_scenario", "config": "en", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 69.73772696704773}, {"type": "f1", "value": 69.21759502909043}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (es)", "type": "mteb/amazon_massive_scenario", "config": "es", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 44.078681909885674}, {"type": "f1", "value": 43.05914426901129}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (fa)", "type": "mteb/amazon_massive_scenario", "config": "fa", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 32.61264290517821}, {"type": "f1", "value": 32.02463177462754}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (fi)", "type": "mteb/amazon_massive_scenario", "config": "fi", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 40.35642232683255}, {"type": "f1", "value": 38.13642481807678}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (fr)", "type": "mteb/amazon_massive_scenario", "config": "fr", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 45.06724949562878}, {"type": "f1", "value": 43.19827608343738}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (he)", "type": "mteb/amazon_massive_scenario", "config": "he", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 32.178883658372555}, {"type": "f1", "value": 29.979761884698775}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (hi)", "type": "mteb/amazon_massive_scenario", "config": "hi", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 26.903160726294555}, {"type": "f1", "value": 25.833010434083363}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (hu)", "type": "mteb/amazon_massive_scenario", "config": "hu", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 40.379959650302624}, {"type": "f1", "value": 37.93134355292882}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (hy)", "type": "mteb/amazon_massive_scenario", "config": "hy", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 28.375924680564896}, {"type": "f1", "value": 26.96255693013172}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (id)", "type": "mteb/amazon_massive_scenario", "config": "id", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 44.361129791526565}, {"type": "f1", "value": 43.54445012295126}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (is)", "type": "mteb/amazon_massive_scenario", "config": "is", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 39.290517821116346}, {"type": "f1", "value": 37.26982052174147}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (it)", "type": "mteb/amazon_massive_scenario", "config": "it", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 46.4694014794889}, {"type": "f1", "value": 44.060986162841566}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (ja)", "type": "mteb/amazon_massive_scenario", "config": "ja", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 46.25756556825824}, {"type": "f1", "value": 45.625139456758816}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (jv)", "type": "mteb/amazon_massive_scenario", "config": "jv", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 41.12642905178212}, {"type": "f1", "value": 39.54392378396527}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (ka)", "type": "mteb/amazon_massive_scenario", "config": "ka", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 24.72763954270343}, {"type": "f1", "value": 23.337743140804484}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (km)", "type": "mteb/amazon_massive_scenario", "config": "km", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 29.741089441829182}, {"type": "f1", "value": 27.570876190083748}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (kn)", "type": "mteb/amazon_massive_scenario", "config": "kn", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 23.850033624747816}, {"type": "f1", "value": 22.86733484540032}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (ko)", "type": "mteb/amazon_massive_scenario", "config": "ko", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 36.56691324815064}, {"type": "f1", "value": 35.504081677134565}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (lv)", "type": "mteb/amazon_massive_scenario", "config": "lv", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 40.928043039677206}, {"type": "f1", "value": 39.108589131211254}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (ml)", "type": "mteb/amazon_massive_scenario", "config": "ml", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 25.527908540685946}, {"type": "f1", "value": 25.333391622280477}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (mn)", "type": "mteb/amazon_massive_scenario", "config": "mn", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 29.105581708137183}, {"type": "f1", "value": 28.478235012692814}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (ms)", "type": "mteb/amazon_massive_scenario", "config": "ms", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 43.78614660390047}, {"type": "f1", "value": 41.9640143926267}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (my)", "type": "mteb/amazon_massive_scenario", "config": "my", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 27.269670477471415}, {"type": "f1", "value": 26.228386764141852}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (nb)", "type": "mteb/amazon_massive_scenario", "config": "nb", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 39.018157363819775}, {"type": "f1", "value": 37.641949339321854}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (nl)", "type": "mteb/amazon_massive_scenario", "config": "nl", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 45.35978480161399}, {"type": "f1", "value": 42.6851176096831}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (pl)", "type": "mteb/amazon_massive_scenario", "config": "pl", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 41.89307330195023}, {"type": "f1", "value": 40.888710642615024}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (pt)", "type": "mteb/amazon_massive_scenario", "config": "pt", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 45.901143241425686}, {"type": "f1", "value": 44.496942353920545}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (ro)", "type": "mteb/amazon_massive_scenario", "config": "ro", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 44.11566913248151}, {"type": "f1", "value": 41.953945105870616}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (ru)", "type": "mteb/amazon_massive_scenario", "config": "ru", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 32.76395427034297}, {"type": "f1", "value": 31.436372571600934}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (sl)", "type": "mteb/amazon_massive_scenario", "config": "sl", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 40.504371217215876}, {"type": "f1", "value": 39.322752749628165}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (sq)", "type": "mteb/amazon_massive_scenario", "config": "sq", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 42.51849361129792}, {"type": "f1", "value": 41.4139297118463}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (sv)", "type": "mteb/amazon_massive_scenario", "config": "sv", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 42.293207800941495}, {"type": "f1", "value": 40.50409536806683}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (sw)", "type": "mteb/amazon_massive_scenario", "config": "sw", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 42.9993275050437}, {"type": "f1", "value": 41.045416224973266}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (ta)", "type": "mteb/amazon_massive_scenario", "config": "ta", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 28.32548755884331}, {"type": "f1", "value": 27.276841995561867}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (te)", "type": "mteb/amazon_massive_scenario", "config": "te", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 26.593813046402154}, {"type": "f1", "value": 25.483878616197586}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (th)", "type": "mteb/amazon_massive_scenario", "config": "th", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 36.788836583725626}, {"type": "f1", "value": 34.603932909177686}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (tl)", "type": "mteb/amazon_massive_scenario", "config": "tl", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 42.5689307330195}, {"type": "f1", "value": 40.924469309079825}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (tr)", "type": "mteb/amazon_massive_scenario", "config": "tr", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 37.09482178883658}, {"type": "f1", "value": 37.949628822857164}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (ur)", "type": "mteb/amazon_massive_scenario", "config": "ur", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 28.836583725622063}, {"type": "f1", "value": 27.806558655512344}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (vi)", "type": "mteb/amazon_massive_scenario", "config": "vi", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 37.357094821788834}, {"type": "f1", "value": 37.507918961038165}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (zh-CN)", "type": "mteb/amazon_massive_scenario", "config": "zh-CN", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 49.37794216543375}, {"type": "f1", "value": 47.20421153697707}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB MassiveScenarioClassification (zh-TW)", "type": "mteb/amazon_massive_scenario", "config": "zh-TW", "split": "test", "revision": "7d571f92784cd94a019292a1f45445077d0ef634"}, "metrics": [{"type": "accuracy", "value": 44.42165433759248}, {"type": "f1", "value": 44.34741861198931}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB MedrxivClusteringP2P", "type": "mteb/medrxiv-clustering-p2p", "config": "default", "split": "test", "revision": "dcefc037ef84348e49b0d29109e891c01067226b"}, "metrics": [{"type": "v_measure", "value": 31.374938993074252}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB MedrxivClusteringS2S", "type": "mteb/medrxiv-clustering-s2s", "config": "default", "split": "test", "revision": "3cd0e71dfbe09d4de0f9e5ecba43e7ce280959dc"}, "metrics": [{"type": "v_measure", "value": 26.871455379644093}]}, {"task": {"type": "Reranking"}, "dataset": {"name": "MTEB MindSmallReranking", "type": "mteb/mind_small", "config": "default", "split": "test", "revision": "3bdac13927fdc888b903db93b2ffdbd90b295a69"}, "metrics": [{"type": "map", "value": 30.402396942935333}, {"type": "mrr", "value": 31.42600938803256}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB NFCorpus", "type": "nfcorpus", "config": "default", "split": "test", "revision": "7eb63cc0c1eb59324d709ebed25fcab851fa7610"}, "metrics": [{"type": "map_at_1", "value": 3.7740000000000005}, {"type": "map_at_10", "value": 7.614999999999999}, {"type": "map_at_100", "value": 9.574}, {"type": "map_at_1000", "value": 10.711}, {"type": "map_at_3", "value": 5.7540000000000004}, {"type": "map_at_5", "value": 6.6659999999999995}, {"type": "mrr_at_1", "value": 33.127}, {"type": "mrr_at_10", "value": 40.351}, {"type": "mrr_at_100", "value": 41.144}, {"type": "mrr_at_1000", "value": 41.202}, {"type": "mrr_at_3", "value": 38.029}, {"type": "mrr_at_5", "value": 39.190000000000005}, {"type": "ndcg_at_1", "value": 31.579}, {"type": "ndcg_at_10", "value": 22.792}, {"type": "ndcg_at_100", "value": 21.698999999999998}, {"type": "ndcg_at_1000", "value": 30.892999999999997}, {"type": "ndcg_at_3", "value": 26.828999999999997}, {"type": "ndcg_at_5", "value": 25.119000000000003}, {"type": "precision_at_1", "value": 33.127}, {"type": "precision_at_10", "value": 16.718}, {"type": "precision_at_100", "value": 5.7090000000000005}, {"type": "precision_at_1000", "value": 1.836}, {"type": "precision_at_3", "value": 24.768}, {"type": "precision_at_5", "value": 21.3}, {"type": "recall_at_1", "value": 3.7740000000000005}, {"type": "recall_at_10", "value": 10.302999999999999}, {"type": "recall_at_100", "value": 23.013}, {"type": "recall_at_1000", "value": 54.864999999999995}, {"type": "recall_at_3", "value": 6.554}, {"type": "recall_at_5", "value": 8.087}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB NQ", "type": "nq", "config": "default", "split": "test", "revision": "6062aefc120bfe8ece5897809fb2e53bfe0d128c"}, "metrics": [{"type": "map_at_1", "value": 15.620999999999999}, {"type": "map_at_10", "value": 24.519}, {"type": "map_at_100", "value": 25.586}, {"type": "map_at_1000", "value": 25.662000000000003}, {"type": "map_at_3", "value": 21.619}, {"type": "map_at_5", "value": 23.232}, {"type": "mrr_at_1", "value": 17.497}, {"type": "mrr_at_10", "value": 26.301000000000002}, {"type": "mrr_at_100", "value": 27.235}, {"type": "mrr_at_1000", "value": 27.297}, {"type": "mrr_at_3", "value": 23.561}, {"type": "mrr_at_5", "value": 25.111}, {"type": "ndcg_at_1", "value": 17.497}, {"type": "ndcg_at_10", "value": 29.725}, {"type": "ndcg_at_100", "value": 34.824}, {"type": "ndcg_at_1000", "value": 36.907000000000004}, {"type": "ndcg_at_3", "value": 23.946}, {"type": "ndcg_at_5", "value": 26.739}, {"type": "precision_at_1", "value": 17.497}, {"type": "precision_at_10", "value": 5.2170000000000005}, {"type": "precision_at_100", "value": 0.8099999999999999}, {"type": "precision_at_1000", "value": 0.101}, {"type": "precision_at_3", "value": 11.114}, {"type": "precision_at_5", "value": 8.285}, {"type": "recall_at_1", "value": 15.620999999999999}, {"type": "recall_at_10", "value": 43.999}, {"type": "recall_at_100", "value": 67.183}, {"type": "recall_at_1000", "value": 83.174}, {"type": "recall_at_3", "value": 28.720000000000002}, {"type": "recall_at_5", "value": 35.154}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB QuoraRetrieval", "type": "quora", "config": "default", "split": "test", "revision": "6205996560df11e3a3da9ab4f926788fc30a7db4"}, "metrics": [{"type": "map_at_1", "value": 54.717000000000006}, {"type": "map_at_10", "value": 67.514}, {"type": "map_at_100", "value": 68.484}, {"type": "map_at_1000", "value": 68.523}, {"type": "map_at_3", "value": 64.169}, {"type": "map_at_5", "value": 66.054}, {"type": "mrr_at_1", "value": 62.46000000000001}, {"type": "mrr_at_10", "value": 71.503}, {"type": "mrr_at_100", "value": 71.91499999999999}, {"type": "mrr_at_1000", "value": 71.923}, {"type": "mrr_at_3", "value": 69.46799999999999}, {"type": "mrr_at_5", "value": 70.677}, {"type": "ndcg_at_1", "value": 62.480000000000004}, {"type": "ndcg_at_10", "value": 72.98}, {"type": "ndcg_at_100", "value": 76.023}, {"type": "ndcg_at_1000", "value": 76.512}, {"type": "ndcg_at_3", "value": 68.138}, {"type": "ndcg_at_5", "value": 70.458}, {"type": "precision_at_1", "value": 62.480000000000004}, {"type": "precision_at_10", "value": 11.373}, {"type": "precision_at_100", "value": 1.437}, {"type": "precision_at_1000", "value": 0.154}, {"type": "precision_at_3", "value": 29.622999999999998}, {"type": "precision_at_5", "value": 19.918}, {"type": "recall_at_1", "value": 54.717000000000006}, {"type": "recall_at_10", "value": 84.745}, {"type": "recall_at_100", "value": 96.528}, {"type": "recall_at_1000", "value": 99.39}, {"type": "recall_at_3", "value": 71.60600000000001}, {"type": "recall_at_5", "value": 77.511}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB RedditClustering", "type": "mteb/reddit-clustering", "config": "default", "split": "test", "revision": "b2805658ae38990172679479369a78b86de8c390"}, "metrics": [{"type": "v_measure", "value": 40.23390747226228}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB RedditClusteringP2P", "type": "mteb/reddit-clustering-p2p", "config": "default", "split": "test", "revision": "385e3cb46b4cfa89021f56c4380204149d0efe33"}, "metrics": [{"type": "v_measure", "value": 49.090518272935626}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB SCIDOCS", "type": "scidocs", "config": "default", "split": "test", "revision": "5c59ef3e437a0a9651c8fe6fde943e7dce59fba5"}, "metrics": [{"type": "map_at_1", "value": 3.028}, {"type": "map_at_10", "value": 6.968000000000001}, {"type": "map_at_100", "value": 8.200000000000001}, {"type": "map_at_1000", "value": 8.432}, {"type": "map_at_3", "value": 5.3069999999999995}, {"type": "map_at_5", "value": 6.099}, {"type": "mrr_at_1", "value": 14.799999999999999}, {"type": "mrr_at_10", "value": 22.425}, {"type": "mrr_at_100", "value": 23.577}, {"type": "mrr_at_1000", "value": 23.669999999999998}, {"type": "mrr_at_3", "value": 20.233}, {"type": "mrr_at_5", "value": 21.318}, {"type": "ndcg_at_1", "value": 14.799999999999999}, {"type": "ndcg_at_10", "value": 12.206}, {"type": "ndcg_at_100", "value": 17.799}, {"type": "ndcg_at_1000", "value": 22.891000000000002}, {"type": "ndcg_at_3", "value": 12.128}, {"type": "ndcg_at_5", "value": 10.212}, {"type": "precision_at_1", "value": 14.799999999999999}, {"type": "precision_at_10", "value": 6.17}, {"type": "precision_at_100", "value": 1.428}, {"type": "precision_at_1000", "value": 0.266}, {"type": "precision_at_3", "value": 11.333}, {"type": "precision_at_5", "value": 8.74}, {"type": "recall_at_1", "value": 3.028}, {"type": "recall_at_10", "value": 12.522}, {"type": "recall_at_100", "value": 28.975}, {"type": "recall_at_1000", "value": 54.038}, {"type": "recall_at_3", "value": 6.912999999999999}, {"type": "recall_at_5", "value": 8.883000000000001}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB SICK-R", "type": "mteb/sickr-sts", "config": "default", "split": "test", "revision": "20a6d6f312dd54037fe07a32d58e5e168867909d"}, "metrics": [{"type": "cos_sim_pearson", "value": 76.62983928119752}, {"type": "cos_sim_spearman", "value": 65.92910683118656}, {"type": "euclidean_pearson", "value": 71.10290039690963}, {"type": "euclidean_spearman", "value": 64.80076622426652}, {"type": "manhattan_pearson", "value": 70.8944726230188}, {"type": "manhattan_spearman", "value": 64.75082576033986}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS12", "type": "mteb/sts12-sts", "config": "default", "split": "test", "revision": "fdf84275bb8ce4b49c971d02e84dd1abc677a50f"}, "metrics": [{"type": "cos_sim_pearson", "value": 74.42679147085553}, {"type": "cos_sim_spearman", "value": 66.52980061546658}, {"type": "euclidean_pearson", "value": 74.87039477408763}, {"type": "euclidean_spearman", "value": 70.63397666902786}, {"type": "manhattan_pearson", "value": 74.97015137513088}, {"type": "manhattan_spearman", "value": 70.75951355434326}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS13", "type": "mteb/sts13-sts", "config": "default", "split": "test", "revision": "1591bfcbe8c69d4bf7fe2a16e2451017832cafb9"}, "metrics": [{"type": "cos_sim_pearson", "value": 75.62472426599543}, {"type": "cos_sim_spearman", "value": 76.1662886374236}, {"type": "euclidean_pearson", "value": 76.3297128081315}, {"type": "euclidean_spearman", "value": 77.19385151966563}, {"type": "manhattan_pearson", "value": 76.50363291423257}, {"type": "manhattan_spearman", "value": 77.37081896355399}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS14", "type": "mteb/sts14-sts", "config": "default", "split": "test", "revision": "e2125984e7df8b7871f6ae9949cf6b6795e7c54b"}, "metrics": [{"type": "cos_sim_pearson", "value": 74.48227705407035}, {"type": "cos_sim_spearman", "value": 69.04572664009687}, {"type": "euclidean_pearson", "value": 71.76138185714849}, {"type": "euclidean_spearman", "value": 68.93415452043307}, {"type": "manhattan_pearson", "value": 71.68010915543306}, {"type": "manhattan_spearman", "value": 68.99176321262806}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS15", "type": "mteb/sts15-sts", "config": "default", "split": "test", "revision": "1cd7298cac12a96a373b6a2f18738bb3e739a9b6"}, "metrics": [{"type": "cos_sim_pearson", "value": 78.1566527175902}, {"type": "cos_sim_spearman", "value": 79.23677712825851}, {"type": "euclidean_pearson", "value": 76.29138438696417}, {"type": "euclidean_spearman", "value": 77.20108266215374}, {"type": "manhattan_pearson", "value": 76.27464935799118}, {"type": "manhattan_spearman", "value": 77.15286174478099}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS16", "type": "mteb/sts16-sts", "config": "default", "split": "test", "revision": "360a0b2dff98700d09e634a01e1cc1624d3e42cd"}, "metrics": [{"type": "cos_sim_pearson", "value": 75.068454465977}, {"type": "cos_sim_spearman", "value": 76.06792422441929}, {"type": "euclidean_pearson", "value": 70.64605440627699}, {"type": "euclidean_spearman", "value": 70.21776051117844}, {"type": "manhattan_pearson", "value": 70.32479295054918}, {"type": "manhattan_spearman", "value": 69.89782458638528}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (ko-ko)", "type": "mteb/sts17-crosslingual-sts", "config": "ko-ko", "split": "test", "revision": "9fc37e8c632af1c87a3d23e685d49552a02582a0"}, "metrics": [{"type": "cos_sim_pearson", "value": 39.43327289939437}, {"type": "cos_sim_spearman", "value": 52.386010275505654}, {"type": "euclidean_pearson", "value": 46.40999904885745}, {"type": "euclidean_spearman", "value": 51.00333465175934}, {"type": "manhattan_pearson", "value": 46.55753533133655}, {"type": "manhattan_spearman", "value": 51.07550440519388}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (ar-ar)", "type": "mteb/sts17-crosslingual-sts", "config": "ar-ar", "split": "test", "revision": "9fc37e8c632af1c87a3d23e685d49552a02582a0"}, "metrics": [{"type": "cos_sim_pearson", "value": 55.54431928210687}, {"type": "cos_sim_spearman", "value": 55.61674586076298}, {"type": "euclidean_pearson", "value": 58.07442713714088}, {"type": "euclidean_spearman", "value": 55.74066216931719}, {"type": "manhattan_pearson", "value": 57.84021675638542}, {"type": "manhattan_spearman", "value": 55.20365812536853}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (en-ar)", "type": "mteb/sts17-crosslingual-sts", "config": "en-ar", "split": "test", "revision": "9fc37e8c632af1c87a3d23e685d49552a02582a0"}, "metrics": [{"type": "cos_sim_pearson", "value": 11.378463868809098}, {"type": "cos_sim_spearman", "value": 8.209569244801065}, {"type": "euclidean_pearson", "value": 1.07041700730406}, {"type": "euclidean_spearman", "value": 2.2052197108931892}, {"type": "manhattan_pearson", "value": 0.7671300251104268}, {"type": "manhattan_spearman", "value": 3.430645020535567}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (en-de)", "type": "mteb/sts17-crosslingual-sts", "config": "en-de", "split": "test", "revision": "9fc37e8c632af1c87a3d23e685d49552a02582a0"}, "metrics": [{"type": "cos_sim_pearson", "value": 32.71403560929013}, {"type": "cos_sim_spearman", "value": 30.18181775929109}, {"type": "euclidean_pearson", "value": 25.57368595910298}, {"type": "euclidean_spearman", "value": 23.316649115731376}, {"type": "manhattan_pearson", "value": 24.144200325329614}, {"type": "manhattan_spearman", "value": 21.64621546338457}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (en-en)", "type": "mteb/sts17-crosslingual-sts", "config": "en-en", "split": "test", "revision": "9fc37e8c632af1c87a3d23e685d49552a02582a0"}, "metrics": [{"type": "cos_sim_pearson", "value": 83.36340470799158}, {"type": "cos_sim_spearman", "value": 84.95398260629699}, {"type": "euclidean_pearson", "value": 80.69876969911644}, {"type": "euclidean_spearman", "value": 80.97451731130427}, {"type": "manhattan_pearson", "value": 80.65869354146945}, {"type": "manhattan_spearman", "value": 80.8540858718528}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (en-tr)", "type": "mteb/sts17-crosslingual-sts", "config": "en-tr", "split": "test", "revision": "9fc37e8c632af1c87a3d23e685d49552a02582a0"}, "metrics": [{"type": "cos_sim_pearson", "value": 1.9200044163754912}, {"type": "cos_sim_spearman", "value": 1.0393399782021342}, {"type": "euclidean_pearson", "value": 1.1376003191297994}, {"type": "euclidean_spearman", "value": 1.8947106671763914}, {"type": "manhattan_pearson", "value": 3.8362564474484335}, {"type": "manhattan_spearman", "value": 4.242750882792888}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (es-en)", "type": "mteb/sts17-crosslingual-sts", "config": "es-en", "split": "test", "revision": "9fc37e8c632af1c87a3d23e685d49552a02582a0"}, "metrics": [{"type": "cos_sim_pearson", "value": 26.561262451099577}, {"type": "cos_sim_spearman", "value": 28.776666666659906}, {"type": "euclidean_pearson", "value": 14.640410196999088}, {"type": "euclidean_spearman", "value": 16.10557011701786}, {"type": "manhattan_pearson", "value": 15.019405495911272}, {"type": "manhattan_spearman", "value": 15.37192083104197}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (es-es)", "type": "mteb/sts17-crosslingual-sts", "config": "es-es", "split": "test", "revision": "9fc37e8c632af1c87a3d23e685d49552a02582a0"}, "metrics": [{"type": "cos_sim_pearson", "value": 69.7544202001433}, {"type": "cos_sim_spearman", "value": 71.88444295144646}, {"type": "euclidean_pearson", "value": 73.84934185952773}, {"type": "euclidean_spearman", "value": 73.26911108021089}, {"type": "manhattan_pearson", "value": 74.04354196954574}, {"type": "manhattan_spearman", "value": 73.37650787943872}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (fr-en)", "type": "mteb/sts17-crosslingual-sts", "config": "fr-en", "split": "test", "revision": "9fc37e8c632af1c87a3d23e685d49552a02582a0"}, "metrics": [{"type": "cos_sim_pearson", "value": 27.70511842301491}, {"type": "cos_sim_spearman", "value": 26.339466714066447}, {"type": "euclidean_pearson", "value": 9.323158236506385}, {"type": "euclidean_spearman", "value": 7.32083231520273}, {"type": "manhattan_pearson", "value": 7.807399527573071}, {"type": "manhattan_spearman", "value": 5.525546663067113}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (it-en)", "type": "mteb/sts17-crosslingual-sts", "config": "it-en", "split": "test", "revision": "9fc37e8c632af1c87a3d23e685d49552a02582a0"}, "metrics": [{"type": "cos_sim_pearson", "value": 24.226521799447692}, {"type": "cos_sim_spearman", "value": 20.72992940458968}, {"type": "euclidean_pearson", "value": 6.753378617205011}, {"type": "euclidean_spearman", "value": 6.281654679029505}, {"type": "manhattan_pearson", "value": 7.087180250449323}, {"type": "manhattan_spearman", "value": 6.41611659259516}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS17 (nl-en)", "type": "mteb/sts17-crosslingual-sts", "config": "nl-en", "split": "test", "revision": "9fc37e8c632af1c87a3d23e685d49552a02582a0"}, "metrics": [{"type": "cos_sim_pearson", "value": 29.131412364061234}, {"type": "cos_sim_spearman", "value": 25.053429612793547}, {"type": "euclidean_pearson", "value": 10.657141303962}, {"type": "euclidean_spearman", "value": 9.712124819778452}, {"type": "manhattan_pearson", "value": 12.481782693315688}, {"type": "manhattan_spearman", "value": 11.287958480905973}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (en)", "type": "mteb/sts22-crosslingual-sts", "config": "en", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 64.04750650962879}, {"type": "cos_sim_spearman", "value": 65.66183708171826}, {"type": "euclidean_pearson", "value": 66.90887604405887}, {"type": "euclidean_spearman", "value": 66.89814072484552}, {"type": "manhattan_pearson", "value": 67.31627110509089}, {"type": "manhattan_spearman", "value": 67.01048176165322}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (de)", "type": "mteb/sts22-crosslingual-sts", "config": "de", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 19.26519187000913}, {"type": "cos_sim_spearman", "value": 21.987647321429005}, {"type": "euclidean_pearson", "value": 17.850618752342946}, {"type": "euclidean_spearman", "value": 22.86669392885474}, {"type": "manhattan_pearson", "value": 18.16183594260708}, {"type": "manhattan_spearman", "value": 23.637510352837907}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (es)", "type": "mteb/sts22-crosslingual-sts", "config": "es", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 34.221261828226936}, {"type": "cos_sim_spearman", "value": 49.811823238907664}, {"type": "euclidean_pearson", "value": 44.50394399762147}, {"type": "euclidean_spearman", "value": 50.959184495072876}, {"type": "manhattan_pearson", "value": 45.83191034038624}, {"type": "manhattan_spearman", "value": 50.190409866117946}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (pl)", "type": "mteb/sts22-crosslingual-sts", "config": "pl", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 3.620381732096531}, {"type": "cos_sim_spearman", "value": 23.30843951799194}, {"type": "euclidean_pearson", "value": 0.965453312113125}, {"type": "euclidean_spearman", "value": 24.235967620790316}, {"type": "manhattan_pearson", "value": 1.4408922275701606}, {"type": "manhattan_spearman", "value": 25.161920137046096}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (tr)", "type": "mteb/sts22-crosslingual-sts", "config": "tr", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 16.69489628726267}, {"type": "cos_sim_spearman", "value": 34.66348380997687}, {"type": "euclidean_pearson", "value": 29.415825529188606}, {"type": "euclidean_spearman", "value": 38.33011033170646}, {"type": "manhattan_pearson", "value": 31.23273195263394}, {"type": "manhattan_spearman", "value": 39.10055785755795}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (ar)", "type": "mteb/sts22-crosslingual-sts", "config": "ar", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 9.134927430889528}, {"type": "cos_sim_spearman", "value": 28.18922448944151}, {"type": "euclidean_pearson", "value": 19.86814169549051}, {"type": "euclidean_spearman", "value": 27.519588644948627}, {"type": "manhattan_pearson", "value": 21.80949221238945}, {"type": "manhattan_spearman", "value": 28.25217200494078}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (ru)", "type": "mteb/sts22-crosslingual-sts", "config": "ru", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 3.6386482942352085}, {"type": "cos_sim_spearman", "value": 9.068119621940966}, {"type": "euclidean_pearson", "value": 0.8123129118737714}, {"type": "euclidean_spearman", "value": 9.173672890166147}, {"type": "manhattan_pearson", "value": 0.754518899822658}, {"type": "manhattan_spearman", "value": 8.431719541986524}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (zh)", "type": "mteb/sts22-crosslingual-sts", "config": "zh", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 2.972091574908432}, {"type": "cos_sim_spearman", "value": 25.48511383289232}, {"type": "euclidean_pearson", "value": 12.751569670148918}, {"type": "euclidean_spearman", "value": 24.940721642439286}, {"type": "manhattan_pearson", "value": 14.310238482989826}, {"type": "manhattan_spearman", "value": 24.69821216148647}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (fr)", "type": "mteb/sts22-crosslingual-sts", "config": "fr", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 54.4745185734135}, {"type": "cos_sim_spearman", "value": 67.66493409568727}, {"type": "euclidean_pearson", "value": 60.13580336797049}, {"type": "euclidean_spearman", "value": 66.12319300814538}, {"type": "manhattan_pearson", "value": 60.816210368708155}, {"type": "manhattan_spearman", "value": 65.70010026716766}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (de-en)", "type": "mteb/sts22-crosslingual-sts", "config": "de-en", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 49.37865412588201}, {"type": "cos_sim_spearman", "value": 53.07135629778897}, {"type": "euclidean_pearson", "value": 49.29201416711091}, {"type": "euclidean_spearman", "value": 50.54523702399645}, {"type": "manhattan_pearson", "value": 51.265764141268534}, {"type": "manhattan_spearman", "value": 51.979086403193605}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (es-en)", "type": "mteb/sts22-crosslingual-sts", "config": "es-en", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 44.925652392562135}, {"type": "cos_sim_spearman", "value": 49.51253904767726}, {"type": "euclidean_pearson", "value": 48.79346518897415}, {"type": "euclidean_spearman", "value": 51.47957870101565}, {"type": "manhattan_pearson", "value": 49.51314553898044}, {"type": "manhattan_spearman", "value": 51.895207893189166}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (it)", "type": "mteb/sts22-crosslingual-sts", "config": "it", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 45.241690321111875}, {"type": "cos_sim_spearman", "value": 48.24795739512037}, {"type": "euclidean_pearson", "value": 49.22719494399897}, {"type": "euclidean_spearman", "value": 49.64102442042809}, {"type": "manhattan_pearson", "value": 49.497887732970256}, {"type": "manhattan_spearman", "value": 49.940515338096304}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (pl-en)", "type": "mteb/sts22-crosslingual-sts", "config": "pl-en", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 36.42138324083909}, {"type": "cos_sim_spearman", "value": 36.79867489417801}, {"type": "euclidean_pearson", "value": 27.760612942610084}, {"type": "euclidean_spearman", "value": 29.140966500287625}, {"type": "manhattan_pearson", "value": 28.456674031350115}, {"type": "manhattan_spearman", "value": 27.46356370924497}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (zh-en)", "type": "mteb/sts22-crosslingual-sts", "config": "zh-en", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 26.55350664089358}, {"type": "cos_sim_spearman", "value": 28.681707196975008}, {"type": "euclidean_pearson", "value": 12.613577889195138}, {"type": "euclidean_spearman", "value": 13.589493311702933}, {"type": "manhattan_pearson", "value": 11.640157427420958}, {"type": "manhattan_spearman", "value": 10.345223941212415}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (es-it)", "type": "mteb/sts22-crosslingual-sts", "config": "es-it", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 38.54682179114309}, {"type": "cos_sim_spearman", "value": 45.782560880405704}, {"type": "euclidean_pearson", "value": 46.496857002368486}, {"type": "euclidean_spearman", "value": 48.21270426410012}, {"type": "manhattan_pearson", "value": 46.871839119374044}, {"type": "manhattan_spearman", "value": 47.556987773851525}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (de-fr)", "type": "mteb/sts22-crosslingual-sts", "config": "de-fr", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 35.12956772546032}, {"type": "cos_sim_spearman", "value": 32.96920218281008}, {"type": "euclidean_pearson", "value": 34.23140384382136}, {"type": "euclidean_spearman", "value": 32.19303153191447}, {"type": "manhattan_pearson", "value": 34.189468276600635}, {"type": "manhattan_spearman", "value": 34.887065709732376}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (de-pl)", "type": "mteb/sts22-crosslingual-sts", "config": "de-pl", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 30.507667380509634}, {"type": "cos_sim_spearman", "value": 20.447284723752716}, {"type": "euclidean_pearson", "value": 29.662041381794474}, {"type": "euclidean_spearman", "value": 20.939990379746757}, {"type": "manhattan_pearson", "value": 32.5112080506328}, {"type": "manhattan_spearman", "value": 23.773047901712495}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STS22 (fr-pl)", "type": "mteb/sts22-crosslingual-sts", "config": "fr-pl", "split": "test", "revision": "2de6ce8c1921b71a755b262c6b57fef195dd7906"}, "metrics": [{"type": "cos_sim_pearson", "value": 71.10820459712156}, {"type": "cos_sim_spearman", "value": 61.97797868009122}, {"type": "euclidean_pearson", "value": 60.30910689156633}, {"type": "euclidean_spearman", "value": 61.97797868009122}, {"type": "manhattan_pearson", "value": 66.3405176964038}, {"type": "manhattan_spearman", "value": 61.97797868009122}]}, {"task": {"type": "STS"}, "dataset": {"name": "MTEB STSBenchmark", "type": "mteb/stsbenchmark-sts", "config": "default", "split": "test", "revision": "8913289635987208e6e7c72789e4be2fe94b6abd"}, "metrics": [{"type": "cos_sim_pearson", "value": 76.53032504460737}, {"type": "cos_sim_spearman", "value": 75.33716094627373}, {"type": "euclidean_pearson", "value": 69.64662673290599}, {"type": "euclidean_spearman", "value": 67.30188896368857}, {"type": "manhattan_pearson", "value": 69.45096082050807}, {"type": "manhattan_spearman", "value": 67.0718727259371}]}, {"task": {"type": "Reranking"}, "dataset": {"name": "MTEB SciDocsRR", "type": "mteb/scidocs-reranking", "config": "default", "split": "test", "revision": "56a6d0140cf6356659e2a7c1413286a774468d44"}, "metrics": [{"type": "map", "value": 71.33941904192648}, {"type": "mrr", "value": 89.73766429648782}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB SciFact", "type": "scifact", "config": "default", "split": "test", "revision": "a75ae049398addde9b70f6b268875f5cbce99089"}, "metrics": [{"type": "map_at_1", "value": 43.333}, {"type": "map_at_10", "value": 52.364}, {"type": "map_at_100", "value": 53.184}, {"type": "map_at_1000", "value": 53.234}, {"type": "map_at_3", "value": 49.832}, {"type": "map_at_5", "value": 51.244}, {"type": "mrr_at_1", "value": 45.333}, {"type": "mrr_at_10", "value": 53.455}, {"type": "mrr_at_100", "value": 54.191}, {"type": "mrr_at_1000", "value": 54.235}, {"type": "mrr_at_3", "value": 51.556000000000004}, {"type": "mrr_at_5", "value": 52.622}, {"type": "ndcg_at_1", "value": 45.333}, {"type": "ndcg_at_10", "value": 56.899}, {"type": "ndcg_at_100", "value": 60.702}, {"type": "ndcg_at_1000", "value": 62.046}, {"type": "ndcg_at_3", "value": 52.451}, {"type": "ndcg_at_5", "value": 54.534000000000006}, {"type": "precision_at_1", "value": 45.333}, {"type": "precision_at_10", "value": 7.8}, {"type": "precision_at_100", "value": 0.987}, {"type": "precision_at_1000", "value": 0.11}, {"type": "precision_at_3", "value": 20.778}, {"type": "precision_at_5", "value": 13.866999999999999}, {"type": "recall_at_1", "value": 43.333}, {"type": "recall_at_10", "value": 69.69999999999999}, {"type": "recall_at_100", "value": 86.9}, {"type": "recall_at_1000", "value": 97.6}, {"type": "recall_at_3", "value": 57.81699999999999}, {"type": "recall_at_5", "value": 62.827999999999996}]}, {"task": {"type": "PairClassification"}, "dataset": {"name": "MTEB SprintDuplicateQuestions", "type": "mteb/sprintduplicatequestions-pairclassification", "config": "default", "split": "test", "revision": "5a8256d0dff9c4bd3be3ba3e67e4e70173f802ea"}, "metrics": [{"type": "cos_sim_accuracy", "value": 99.7}, {"type": "cos_sim_ap", "value": 89.88577913120001}, {"type": "cos_sim_f1", "value": 84.62694041061593}, {"type": "cos_sim_precision", "value": 84.7542627883651}, {"type": "cos_sim_recall", "value": 84.5}, {"type": "dot_accuracy", "value": 99.24752475247524}, {"type": "dot_ap", "value": 56.81855467290009}, {"type": "dot_f1", "value": 56.084126189283936}, {"type": "dot_precision", "value": 56.16850551654965}, {"type": "dot_recall", "value": 56.00000000000001}, {"type": "euclidean_accuracy", "value": 99.7059405940594}, {"type": "euclidean_ap", "value": 90.12451226491524}, {"type": "euclidean_f1", "value": 84.44211629125196}, {"type": "euclidean_precision", "value": 88.66886688668868}, {"type": "euclidean_recall", "value": 80.60000000000001}, {"type": "manhattan_accuracy", "value": 99.7128712871287}, {"type": "manhattan_ap", "value": 90.67590584183216}, {"type": "manhattan_f1", "value": 84.85436893203884}, {"type": "manhattan_precision", "value": 82.45283018867924}, {"type": "manhattan_recall", "value": 87.4}, {"type": "max_accuracy", "value": 99.7128712871287}, {"type": "max_ap", "value": 90.67590584183216}, {"type": "max_f1", "value": 84.85436893203884}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB StackExchangeClustering", "type": "mteb/stackexchange-clustering", "config": "default", "split": "test", "revision": "70a89468f6dccacc6aa2b12a6eac54e74328f235"}, "metrics": [{"type": "v_measure", "value": 52.74481093815175}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB StackExchangeClusteringP2P", "type": "mteb/stackexchange-clustering-p2p", "config": "default", "split": "test", "revision": "d88009ab563dd0b16cfaf4436abaf97fa3550cf0"}, "metrics": [{"type": "v_measure", "value": 32.65999453562101}]}, {"task": {"type": "Reranking"}, "dataset": {"name": "MTEB StackOverflowDupQuestions", "type": "mteb/stackoverflowdupquestions-reranking", "config": "default", "split": "test", "revision": "ef807ea29a75ec4f91b50fd4191cb4ee4589a9f9"}, "metrics": [{"type": "map", "value": 44.74498464555465}, {"type": "mrr", "value": 45.333879764026825}]}, {"task": {"type": "Summarization"}, "dataset": {"name": "MTEB SummEval", "type": "mteb/summeval", "config": "default", "split": "test", "revision": "8753c2788d36c01fc6f05d03fe3f7268d63f9122"}, "metrics": [{"type": "cos_sim_pearson", "value": "29,603788751645216"}, {"type": "cos_sim_spearman", "value": 29.705103354786033}, {"type": "dot_pearson", "value": 28.07425338095399}, {"type": "dot_spearman", "value": 26.841406359135366}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB TRECCOVID", "type": "trec-covid", "config": "default", "split": "test", "revision": "2c8041b2c07a79b6f7ba8fe6acc72e5d9f92d217"}, "metrics": [{"type": "map_at_1", "value": 0.241}, {"type": "map_at_10", "value": 1.672}, {"type": "map_at_100", "value": 7.858999999999999}, {"type": "map_at_1000", "value": 17.616}, {"type": "map_at_3", "value": 0.631}, {"type": "map_at_5", "value": 0.968}, {"type": "mrr_at_1", "value": 90.0}, {"type": "mrr_at_10", "value": 92.952}, {"type": "mrr_at_100", "value": 93.036}, {"type": "mrr_at_1000", "value": 93.036}, {"type": "mrr_at_3", "value": 92.667}, {"type": "mrr_at_5", "value": 92.667}, {"type": "ndcg_at_1", "value": 83.0}, {"type": "ndcg_at_10", "value": 70.30199999999999}, {"type": "ndcg_at_100", "value": 48.149}, {"type": "ndcg_at_1000", "value": 40.709}, {"type": "ndcg_at_3", "value": 79.173}, {"type": "ndcg_at_5", "value": 75.347}, {"type": "precision_at_1", "value": 90.0}, {"type": "precision_at_10", "value": 72.6}, {"type": "precision_at_100", "value": 48.46}, {"type": "precision_at_1000", "value": 18.093999999999998}, {"type": "precision_at_3", "value": 84.0}, {"type": "precision_at_5", "value": 78.8}, {"type": "recall_at_1", "value": 0.241}, {"type": "recall_at_10", "value": 1.814}, {"type": "recall_at_100", "value": 11.141}, {"type": "recall_at_1000", "value": 37.708999999999996}, {"type": "recall_at_3", "value": 0.647}, {"type": "recall_at_5", "value": 1.015}]}, {"task": {"type": "Retrieval"}, "dataset": {"name": "MTEB Touche2020", "type": "webis-touche2020", "config": "default", "split": "test", "revision": "527b7d77e16e343303e68cb6af11d6e18b9f7b3b"}, "metrics": [{"type": "map_at_1", "value": 2.782}, {"type": "map_at_10", "value": 9.06}, {"type": "map_at_100", "value": 14.571000000000002}, {"type": "map_at_1000", "value": 16.006999999999998}, {"type": "map_at_3", "value": 5.037}, {"type": "map_at_5", "value": 6.63}, {"type": "mrr_at_1", "value": 34.694}, {"type": "mrr_at_10", "value": 48.243}, {"type": "mrr_at_100", "value": 49.065}, {"type": "mrr_at_1000", "value": 49.065}, {"type": "mrr_at_3", "value": 44.897999999999996}, {"type": "mrr_at_5", "value": 46.428999999999995}, {"type": "ndcg_at_1", "value": 31.633}, {"type": "ndcg_at_10", "value": 22.972}, {"type": "ndcg_at_100", "value": 34.777}, {"type": "ndcg_at_1000", "value": 45.639}, {"type": "ndcg_at_3", "value": 26.398}, {"type": "ndcg_at_5", "value": 24.418}, {"type": "precision_at_1", "value": 34.694}, {"type": "precision_at_10", "value": 19.796}, {"type": "precision_at_100", "value": 7.224}, {"type": "precision_at_1000", "value": 1.4449999999999998}, {"type": "precision_at_3", "value": 26.531}, {"type": "precision_at_5", "value": 23.265}, {"type": "recall_at_1", "value": 2.782}, {"type": "recall_at_10", "value": 14.841}, {"type": "recall_at_100", "value": 44.86}, {"type": "recall_at_1000", "value": 78.227}, {"type": "recall_at_3", "value": 5.959}, {"type": "recall_at_5", "value": 8.969000000000001}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB ToxicConversationsClassification", "type": "mteb/toxic_conversations_50k", "config": "default", "split": "test", "revision": "edfaf9da55d3dd50d43143d90c1ac476895ae6de"}, "metrics": [{"type": "accuracy", "value": 62.657999999999994}, {"type": "ap", "value": 10.96353161716344}, {"type": "f1", "value": 48.294226423442645}]}, {"task": {"type": "Classification"}, "dataset": {"name": "MTEB TweetSentimentExtractionClassification", "type": "mteb/tweet_sentiment_extraction", "config": "default", "split": "test", "revision": "62146448f05be9e52a36b8ee9936447ea787eede"}, "metrics": [{"type": "accuracy", "value": 52.40803621958121}, {"type": "f1", "value": 52.61009636022186}]}, {"task": {"type": "Clustering"}, "dataset": {"name": "MTEB TwentyNewsgroupsClustering", "type": "mteb/twentynewsgroups-clustering", "config": "default", "split": "test", "revision": "091a54f9a36281ce7d6590ec8c75dd485e7e01d4"}, "metrics": [{"type": "v_measure", "value": 32.12697126747911}]}, {"task": {"type": "PairClassification"}, "dataset": {"name": "MTEB TwitterSemEval2015", "type": "mteb/twittersemeval2015-pairclassification", "config": "default", "split": "test", "revision": "70970daeab8776df92f5ea462b6173c0b46fd2d1"}, "metrics": [{"type": "cos_sim_accuracy", "value": 80.69976753889253}, {"type": "cos_sim_ap", "value": 54.74680676121268}, {"type": "cos_sim_f1", "value": 53.18923998590391}, {"type": "cos_sim_precision", "value": 47.93563413084904}, {"type": "cos_sim_recall", "value": 59.73614775725594}, {"type": "dot_accuracy", "value": 79.3348036001669}, {"type": "dot_ap", "value": 48.46902128933627}, {"type": "dot_f1", "value": 50.480109739369006}, {"type": "dot_precision", "value": 42.06084051345173}, {"type": "dot_recall", "value": 63.113456464379944}, {"type": "euclidean_accuracy", "value": 79.78780473266973}, {"type": "euclidean_ap", "value": 50.258327255164815}, {"type": "euclidean_f1", "value": 49.655838666827684}, {"type": "euclidean_precision", "value": 45.78044978846582}, {"type": "euclidean_recall", "value": 54.24802110817942}, {"type": "manhattan_accuracy", "value": 79.76992310901831}, {"type": "manhattan_ap", "value": 49.89892485714363}, {"type": "manhattan_f1", "value": 49.330433787341185}, {"type": "manhattan_precision", "value": 43.56175459874672}, {"type": "manhattan_recall", "value": 56.86015831134564}, {"type": "max_accuracy", "value": 80.69976753889253}, {"type": "max_ap", "value": 54.74680676121268}, {"type": "max_f1", "value": 53.18923998590391}]}, {"task": {"type": "PairClassification"}, "dataset": {"name": "MTEB TwitterURLCorpus", "type": "mteb/twitterurlcorpus-pairclassification", "config": "default", "split": "test", "revision": "8b6510b0b1fa4e4c4f879467980e9be563ec1cdf"}, "metrics": [{"type": "cos_sim_accuracy", "value": 86.90573213800597}, {"type": "cos_sim_ap", "value": 81.05760818661524}, {"type": "cos_sim_f1", "value": 73.64688856729379}, {"type": "cos_sim_precision", "value": 69.46491946491946}, {"type": "cos_sim_recall", "value": 78.3646442870342}, {"type": "dot_accuracy", "value": 83.80680715644041}, {"type": "dot_ap", "value": 72.49774005947461}, {"type": "dot_f1", "value": 68.68460650173216}, {"type": "dot_precision", "value": 62.954647507858105}, {"type": "dot_recall", "value": 75.56205728364644}, {"type": "euclidean_accuracy", "value": 85.97430822369697}, {"type": "euclidean_ap", "value": 78.86101740829326}, {"type": "euclidean_f1", "value": 71.07960824663695}, {"type": "euclidean_precision", "value": 70.36897306270279}, {"type": "euclidean_recall", "value": 71.8047428395442}, {"type": "manhattan_accuracy", "value": 85.94132029339853}, {"type": "manhattan_ap", "value": 78.77876711171923}, {"type": "manhattan_f1", "value": 71.07869075515912}, {"type": "manhattan_precision", "value": 69.80697847067557}, {"type": "manhattan_recall", "value": 72.39759778256852}, {"type": "max_accuracy", "value": 86.90573213800597}, {"type": "max_ap", "value": 81.05760818661524}, {"type": "max_f1", "value": 73.64688856729379}]}]}]}
sentence-transformers pytorch gpt_neo feature-extraction sentence-similarity mteb arxiv:2202.08904 model-index endpoints_compatible has_space region:us
TAGS
#sentence-transformers #pytorch #gpt_neo #feature-extraction #sentence-similarity #mteb #arxiv-2202.08904 #model-index #endpoints_compatible #has_space #region-us 


# SGPT-125M-weightedmean-msmarco-specb-bitfit

## Usage

For usage instructions, refer to our codebase: URL 

## Evaluation Results

For eval results, refer to the eval folder or our paper: URL


## Training
The model was trained with the parameters:

DataLoader:

'URL.dataloader.DataLoader' of length 15600 with parameters:


Loss:

'sentence_transformers.losses.MultipleNegativesRankingLoss.MultipleNegativesRankingLoss' with parameters:
  

Parameters of the fit()-Method:



## Full Model Architecture


## Citing & Authors



# SGPT-125M-weightedmean-msmarco-specb-bitfit ## Usage

For usage instructions, refer to our codebase: URL ## Evaluation Results

For eval results, refer to the eval folder or our paper: URL ## Training
The model was trained with the parameters:

DataLoader:

'URL.dataloader.DataLoader' of length 15600 with parameters:


Loss:

'sentence_transformers.losses.MultipleNegativesRankingLoss.MultipleNegativesRankingLoss' with parameters:
  

Parameters of the fit()-Method: ## Full Model Architecture ## Citing & Authors
TAGS
#sentence-transformers #pytorch #gpt_neo #feature-extraction #sentence-similarity #mteb #arxiv-2202.08904 #model-index #endpoints_compatible #has_space #region-us 
 # SGPT-125M-weightedmean-msmarco-specb-bitfit ## Usage

For usage instructions, refer to our codebase: URL ## Evaluation Results

For eval results, refer to the eval folder or our paper: URL ## Training
The model was trained with the parameters:

DataLoader:

'URL.dataloader.DataLoader' of length 15600 with parameters:


Loss:

'sentence_transformers.losses.MultipleNegativesRankingLoss.MultipleNegativesRankingLoss' with parameters:
  

Parameters of the fit()-Method: ## Full Model Architecture ## Citing & Authors
passage: TAGS
#sentence-transformers #pytorch #gpt_neo #feature-extraction #sentence-similarity #mteb #arxiv-2202.08904 #model-index #endpoints_compatible #has_space #region-us 
# SGPT-125M-weightedmean-msmarco-specb-bitfit## Usage

For usage instructions, refer to our codebase: URL## Evaluation Results

For eval results, refer to the eval folder or our paper: URL## Training
The model was trained with the parameters:

DataLoader:

'URL.dataloader.DataLoader' of length 15600 with parameters:


Loss:

'sentence_transformers.losses.MultipleNegativesRankingLoss.MultipleNegativesRankingLoss' with parameters:
  

Parameters of the fit()-Method:## Full Model Architecture## Citing & Authors