diff --git "a/README.md" "b/README.md" --- "a/README.md" +++ "b/README.md" @@ -1,3 +1,3938 @@ ---- -license: apache-2.0 ---- +--- +tags: +- sentence-transformers +- sentence-similarity +- feature-extraction +- mteb +base_model: sbintuitions/modernbert-ja-130m +language: +- ja +- en +model-index: +- name: retrieva-jp/amber-base + results: + - dataset: + config: en + name: MTEB AmazonCounterfactualClassification (en) + revision: e8379541af4e31359cca9fbcf4b00f2671dba205 + split: test + type: mteb/amazon_counterfactual + metrics: + - type: accuracy + value: 68.1642 + - type: f1 + value: 61.9811 + - type: f1_weighted + value: 71.2157 + - type: ap + value: 30.6541 + - type: ap_weighted + value: 30.6541 + - type: main_score + value: 68.1642 + task: + type: Classification + - dataset: + config: default + name: MTEB ArXivHierarchicalClusteringP2P (default) + revision: 0bbdb47bcbe3a90093699aefeed338a0f28a7ee8 + split: test + type: mteb/arxiv-clustering-p2p + metrics: + - type: v_measure + value: 55.655100000000004 + - type: v_measure_std + value: 3.2918999999999996 + - type: main_score + value: 55.655100000000004 + task: + type: Clustering + - dataset: + config: default + name: MTEB ArXivHierarchicalClusteringS2S (default) + revision: b73bd54100e5abfa6e3a23dcafb46fe4d2438dc3 + split: test + type: mteb/arxiv-clustering-s2s + metrics: + - type: v_measure + value: 53.6493 + - type: v_measure_std + value: 3.2359 + - type: main_score + value: 53.6493 + task: + type: Clustering + - dataset: + config: default + name: MTEB ArguAna (default) + revision: c22ab2a51041ffd869aaddef7af8d8215647e41a + split: test + type: mteb/arguana + metrics: + - type: ndcg_at_1 + value: 25.249 + - type: ndcg_at_3 + value: 38.056 + - type: ndcg_at_5 + value: 43.124 + - type: ndcg_at_10 + value: 48.068 + - type: ndcg_at_20 + value: 51.461 + - type: ndcg_at_100 + value: 53.15800000000001 + - type: ndcg_at_1000 + value: 53.38 + - type: map_at_1 + value: 25.249 + - type: map_at_3 + value: 34.803 + - type: map_at_5 + value: 37.598 + - type: map_at_10 + value: 39.611000000000004 + - type: map_at_20 + value: 40.569 + - type: map_at_100 + value: 40.821000000000005 + - type: map_at_1000 + value: 40.83 + - type: recall_at_1 + value: 25.249 + - type: recall_at_3 + value: 47.510999999999996 + - type: recall_at_5 + value: 59.885999999999996 + - type: recall_at_10 + value: 75.32 + - type: recall_at_20 + value: 88.549 + - type: recall_at_100 + value: 97.44 + - type: recall_at_1000 + value: 99.14699999999999 + - type: precision_at_1 + value: 25.249 + - type: precision_at_3 + value: 15.837000000000002 + - type: precision_at_5 + value: 11.977 + - type: precision_at_10 + value: 7.532 + - type: precision_at_20 + value: 4.427 + - type: precision_at_100 + value: 0.9740000000000001 + - type: precision_at_1000 + value: 0.099 + - type: mrr_at_1 + value: 25.817899999999998 + - type: mrr_at_3 + value: 34.9692 + - type: mrr_at_5 + value: 37.7928 + - type: mrr_at_10 + value: 39.8238 + - type: mrr_at_20 + value: 40.7844 + - type: mrr_at_100 + value: 41.0403 + - type: mrr_at_1000 + value: 41.0495 + - type: nauc_ndcg_at_1_max + value: -2.6569 + - type: nauc_ndcg_at_1_std + value: -2.4726000000000004 + - type: nauc_ndcg_at_1_diff1 + value: 10.259699999999999 + - type: nauc_ndcg_at_3_max + value: -0.8151 + - type: nauc_ndcg_at_3_std + value: -3.3642 + - type: nauc_ndcg_at_3_diff1 + value: 7.884099999999999 + - type: nauc_ndcg_at_5_max + value: -0.3906 + - type: nauc_ndcg_at_5_std + value: -2.4619 + - type: nauc_ndcg_at_5_diff1 + value: 7.558 + - type: nauc_ndcg_at_10_max + value: 1.0935000000000001 + - type: nauc_ndcg_at_10_std + value: -1.8624999999999998 + - type: nauc_ndcg_at_10_diff1 + value: 8.0503 + - type: nauc_ndcg_at_20_max + value: 1.3164 + - type: nauc_ndcg_at_20_std + value: -1.3407 + - type: nauc_ndcg_at_20_diff1 + value: 7.8992 + - type: nauc_ndcg_at_100_max + value: 0.8316 + - type: nauc_ndcg_at_100_std + value: -0.8725 + - type: nauc_ndcg_at_100_diff1 + value: 8.5633 + - type: nauc_ndcg_at_1000_max + value: 0.44999999999999996 + - type: nauc_ndcg_at_1000_std + value: -1.4357 + - type: nauc_ndcg_at_1000_diff1 + value: 8.4438 + - type: nauc_map_at_1_max + value: -2.6569 + - type: nauc_map_at_1_std + value: -2.4726000000000004 + - type: nauc_map_at_1_diff1 + value: 10.259699999999999 + - type: nauc_map_at_3_max + value: -1.3567 + - type: nauc_map_at_3_std + value: -3.222 + - type: nauc_map_at_3_diff1 + value: 8.3557 + - type: nauc_map_at_5_max + value: -1.162 + - type: nauc_map_at_5_std + value: -2.7384 + - type: nauc_map_at_5_diff1 + value: 8.118400000000001 + - type: nauc_map_at_10_max + value: -0.615 + - type: nauc_map_at_10_std + value: -2.5394 + - type: nauc_map_at_10_diff1 + value: 8.283100000000001 + - type: nauc_map_at_20_max + value: -0.5492 + - type: nauc_map_at_20_std + value: -2.4076 + - type: nauc_map_at_20_diff1 + value: 8.280999999999999 + - type: nauc_map_at_100_max + value: -0.6049 + - type: nauc_map_at_100_std + value: -2.3560000000000003 + - type: nauc_map_at_100_diff1 + value: 8.3933 + - type: nauc_map_at_1000_max + value: -0.6154 + - type: nauc_map_at_1000_std + value: -2.373 + - type: nauc_map_at_1000_diff1 + value: 8.3902 + - type: nauc_recall_at_1_max + value: -2.6569 + - type: nauc_recall_at_1_std + value: -2.4726000000000004 + - type: nauc_recall_at_1_diff1 + value: 10.259699999999999 + - type: nauc_recall_at_3_max + value: 0.7234 + - type: nauc_recall_at_3_std + value: -3.7315 + - type: nauc_recall_at_3_diff1 + value: 6.6138 + - type: nauc_recall_at_5_max + value: 2.0847 + - type: nauc_recall_at_5_std + value: -1.4385000000000001 + - type: nauc_recall_at_5_diff1 + value: 5.9428 + - type: nauc_recall_at_10_max + value: 9.2417 + - type: nauc_recall_at_10_std + value: 1.6372000000000002 + - type: nauc_recall_at_10_diff1 + value: 7.6442 + - type: nauc_recall_at_20_max + value: 17.9819 + - type: nauc_recall_at_20_std + value: 9.3827 + - type: nauc_recall_at_20_diff1 + value: 5.2288 + - type: nauc_recall_at_100_max + value: 46.3576 + - type: nauc_recall_at_100_std + value: 69.5314 + - type: nauc_recall_at_100_diff1 + value: 25.2365 + - type: nauc_recall_at_1000_max + value: 47.3173 + - type: nauc_recall_at_1000_std + value: 80.3564 + - type: nauc_recall_at_1000_diff1 + value: 30.506 + - type: nauc_precision_at_1_max + value: -2.6569 + - type: nauc_precision_at_1_std + value: -2.4726000000000004 + - type: nauc_precision_at_1_diff1 + value: 10.259699999999999 + - type: nauc_precision_at_3_max + value: 0.7234 + - type: nauc_precision_at_3_std + value: -3.7315 + - type: nauc_precision_at_3_diff1 + value: 6.6138 + - type: nauc_precision_at_5_max + value: 2.0847 + - type: nauc_precision_at_5_std + value: -1.4385000000000001 + - type: nauc_precision_at_5_diff1 + value: 5.9428 + - type: nauc_precision_at_10_max + value: 9.2417 + - type: nauc_precision_at_10_std + value: 1.6372000000000002 + - type: nauc_precision_at_10_diff1 + value: 7.6442 + - type: nauc_precision_at_20_max + value: 17.9819 + - type: nauc_precision_at_20_std + value: 9.3827 + - type: nauc_precision_at_20_diff1 + value: 5.2288 + - type: nauc_precision_at_100_max + value: 46.3576 + - type: nauc_precision_at_100_std + value: 69.5314 + - type: nauc_precision_at_100_diff1 + value: 25.2365 + - type: nauc_precision_at_1000_max + value: 47.3173 + - type: nauc_precision_at_1000_std + value: 80.3564 + - type: nauc_precision_at_1000_diff1 + value: 30.506 + - type: nauc_mrr_at_1_max + value: -2.5852 + - type: nauc_mrr_at_1_std + value: -2.7133000000000003 + - type: nauc_mrr_at_1_diff1 + value: 8.3902 + - type: nauc_mrr_at_3_max + value: -2.3878 + - type: nauc_mrr_at_3_std + value: -3.1916 + - type: nauc_mrr_at_3_diff1 + value: 6.3759999999999994 + - type: nauc_mrr_at_5_max + value: -2.0079 + - type: nauc_mrr_at_5_std + value: -2.9791000000000003 + - type: nauc_mrr_at_5_diff1 + value: 6.3531 + - type: nauc_mrr_at_10_max + value: -1.41 + - type: nauc_mrr_at_10_std + value: -2.7921 + - type: nauc_mrr_at_10_diff1 + value: 6.514200000000001 + - type: nauc_mrr_at_20_max + value: -1.35 + - type: nauc_mrr_at_20_std + value: -2.6331 + - type: nauc_mrr_at_20_diff1 + value: 6.4700999999999995 + - type: nauc_mrr_at_100_max + value: -1.393 + - type: nauc_mrr_at_100_std + value: -2.5819 + - type: nauc_mrr_at_100_diff1 + value: 6.5875 + - type: nauc_mrr_at_1000_max + value: -1.4037000000000002 + - type: nauc_mrr_at_1000_std + value: -2.5989 + - type: nauc_mrr_at_1000_diff1 + value: 6.583799999999999 + - type: main_score + value: 48.068 + task: + type: Retrieval + - dataset: + config: default + name: MTEB AskUbuntuDupQuestions (default) + revision: 2000358ca161889fa9c082cb41daa8dcfb161a54 + split: test + type: mteb/askubuntudupquestions-reranking + metrics: + - type: map + value: 56.5225 + - type: mrr + value: 70.5146 + - type: nAUC_map_max + value: 18.224 + - type: nAUC_map_std + value: 12.5352 + - type: nAUC_map_diff1 + value: 14.0464 + - type: nAUC_mrr_max + value: 28.619699999999998 + - type: nAUC_mrr_std + value: 21.69 + - type: nAUC_mrr_diff1 + value: 15.8021 + - type: main_score + value: 56.5225 + task: + type: Reranking + - dataset: + config: default + name: MTEB BIOSSES (default) + revision: d3fb88f8f02e40887cd149695127462bbcf29b4a + split: test + type: mteb/biosses-sts + metrics: + - type: pearson + value: 86.6855 + - type: spearman + value: 83.17360000000001 + - type: cosine_pearson + value: 86.6855 + - type: cosine_spearman + value: 83.17360000000001 + - type: manhattan_pearson + value: 85.5442 + - type: manhattan_spearman + value: 83.9501 + - type: euclidean_pearson + value: 85.0403 + - type: euclidean_spearman + value: 83.17360000000001 + - type: main_score + value: 83.17360000000001 + task: + type: STS + - dataset: + config: default + name: MTEB Banking77Classification (default) + revision: 0fd18e25b25c072e09e0d92ab615fda904d66300 + split: test + type: mteb/banking77 + metrics: + - type: accuracy + value: 76.3312 + - type: f1 + value: 75.4609 + - type: f1_weighted + value: 75.4609 + - type: main_score + value: 76.3312 + task: + type: Classification + - dataset: + config: default + name: MTEB BiorxivClusteringP2P.v2 (default) + revision: f5dbc242e11dd8e24def4c4268607a49e02946dc + split: test + type: mteb/biorxiv-clustering-p2p + metrics: + - type: v_measure + value: 33.6692 + - type: v_measure_std + value: 0.769 + - type: main_score + value: 33.6692 + task: + type: Clustering + - dataset: + config: default + name: MTEB CQADupstackGamingRetrieval (default) + revision: 4885aa143210c98657558c04aaf3dc47cfb54340 + split: test + type: mteb/cqadupstack-gaming + metrics: + - type: ndcg_at_1 + value: 30.345 + - type: ndcg_at_3 + value: 37.726 + - type: ndcg_at_5 + value: 39.999 + - type: ndcg_at_10 + value: 42.732 + - type: ndcg_at_20 + value: 44.696000000000005 + - type: ndcg_at_100 + value: 47.461 + - type: ndcg_at_1000 + value: 49.341 + - type: map_at_1 + value: 26.484999999999996 + - type: map_at_3 + value: 34.474 + - type: map_at_5 + value: 35.94 + - type: map_at_10 + value: 37.24 + - type: map_at_20 + value: 37.852999999999994 + - type: map_at_100 + value: 38.286 + - type: map_at_1000 + value: 38.369 + - type: recall_at_1 + value: 26.484999999999996 + - type: recall_at_3 + value: 42.857 + - type: recall_at_5 + value: 48.501 + - type: recall_at_10 + value: 56.48 + - type: recall_at_20 + value: 63.81099999999999 + - type: recall_at_100 + value: 77.518 + - type: recall_at_1000 + value: 90.89 + - type: precision_at_1 + value: 30.345 + - type: precision_at_3 + value: 17.241 + - type: precision_at_5 + value: 11.962 + - type: precision_at_10 + value: 7.204000000000001 + - type: precision_at_20 + value: 4.1290000000000004 + - type: precision_at_100 + value: 1.0330000000000001 + - type: precision_at_1000 + value: 0.127 + - type: mrr_at_1 + value: 30.3448 + - type: mrr_at_3 + value: 37.5131 + - type: mrr_at_5 + value: 38.8516 + - type: mrr_at_10 + value: 39.915299999999995 + - type: mrr_at_20 + value: 40.428599999999996 + - type: mrr_at_100 + value: 40.7757 + - type: mrr_at_1000 + value: 40.8275 + - type: nauc_ndcg_at_1_max + value: 30.5442 + - type: nauc_ndcg_at_1_std + value: -10.3888 + - type: nauc_ndcg_at_1_diff1 + value: 52.476 + - type: nauc_ndcg_at_3_max + value: 28.6927 + - type: nauc_ndcg_at_3_std + value: -8.8728 + - type: nauc_ndcg_at_3_diff1 + value: 45.094699999999996 + - type: nauc_ndcg_at_5_max + value: 29.259600000000002 + - type: nauc_ndcg_at_5_std + value: -7.945399999999999 + - type: nauc_ndcg_at_5_diff1 + value: 44.600699999999996 + - type: nauc_ndcg_at_10_max + value: 29.9977 + - type: nauc_ndcg_at_10_std + value: -6.1746 + - type: nauc_ndcg_at_10_diff1 + value: 44.2832 + - type: nauc_ndcg_at_20_max + value: 30.034100000000002 + - type: nauc_ndcg_at_20_std + value: -4.8941 + - type: nauc_ndcg_at_20_diff1 + value: 43.3814 + - type: nauc_ndcg_at_100_max + value: 30.812800000000003 + - type: nauc_ndcg_at_100_std + value: -3.5000999999999998 + - type: nauc_ndcg_at_100_diff1 + value: 43.345 + - type: nauc_ndcg_at_1000_max + value: 30.9884 + - type: nauc_ndcg_at_1000_std + value: -3.9316999999999998 + - type: nauc_ndcg_at_1000_diff1 + value: 43.6512 + - type: nauc_map_at_1_max + value: 27.442800000000002 + - type: nauc_map_at_1_std + value: -9.8884 + - type: nauc_map_at_1_diff1 + value: 52.666999999999994 + - type: nauc_map_at_3_max + value: 27.897100000000002 + - type: nauc_map_at_3_std + value: -9.777 + - type: nauc_map_at_3_diff1 + value: 47.013 + - type: nauc_map_at_5_max + value: 28.3476 + - type: nauc_map_at_5_std + value: -9.3335 + - type: nauc_map_at_5_diff1 + value: 46.7246 + - type: nauc_map_at_10_max + value: 28.921000000000003 + - type: nauc_map_at_10_std + value: -8.4018 + - type: nauc_map_at_10_diff1 + value: 46.5358 + - type: nauc_map_at_20_max + value: 29.033900000000003 + - type: nauc_map_at_20_std + value: -7.985100000000001 + - type: nauc_map_at_20_diff1 + value: 46.2362 + - type: nauc_map_at_100_max + value: 29.2382 + - type: nauc_map_at_100_std + value: -7.7172 + - type: nauc_map_at_100_diff1 + value: 46.2663 + - type: nauc_map_at_1000_max + value: 29.263699999999996 + - type: nauc_map_at_1000_std + value: -7.7108 + - type: nauc_map_at_1000_diff1 + value: 46.2735 + - type: nauc_recall_at_1_max + value: 27.442800000000002 + - type: nauc_recall_at_1_std + value: -9.8884 + - type: nauc_recall_at_1_diff1 + value: 52.666999999999994 + - type: nauc_recall_at_3_max + value: 25.7102 + - type: nauc_recall_at_3_std + value: -8.2064 + - type: nauc_recall_at_3_diff1 + value: 39.145 + - type: nauc_recall_at_5_max + value: 27.244699999999998 + - type: nauc_recall_at_5_std + value: -5.943 + - type: nauc_recall_at_5_diff1 + value: 38.024 + - type: nauc_recall_at_10_max + value: 29.226000000000003 + - type: nauc_recall_at_10_std + value: -0.2402 + - type: nauc_recall_at_10_diff1 + value: 36.58 + - type: nauc_recall_at_20_max + value: 29.567500000000003 + - type: nauc_recall_at_20_std + value: 6.2502 + - type: nauc_recall_at_20_diff1 + value: 32.092999999999996 + - type: nauc_recall_at_100_max + value: 33.8086 + - type: nauc_recall_at_100_std + value: 20.092 + - type: nauc_recall_at_100_diff1 + value: 27.5754 + - type: nauc_recall_at_1000_max + value: 38.0782 + - type: nauc_recall_at_1000_std + value: 34.3309 + - type: nauc_recall_at_1000_diff1 + value: 17.712 + - type: nauc_precision_at_1_max + value: 30.5442 + - type: nauc_precision_at_1_std + value: -10.3888 + - type: nauc_precision_at_1_diff1 + value: 52.476 + - type: nauc_precision_at_3_max + value: 29.0858 + - type: nauc_precision_at_3_std + value: -5.8233 + - type: nauc_precision_at_3_diff1 + value: 33.480900000000005 + - type: nauc_precision_at_5_max + value: 30.425200000000004 + - type: nauc_precision_at_5_std + value: -2.0077000000000003 + - type: nauc_precision_at_5_diff1 + value: 29.5631 + - type: nauc_precision_at_10_max + value: 30.8693 + - type: nauc_precision_at_10_std + value: 4.5986 + - type: nauc_precision_at_10_diff1 + value: 23.346600000000002 + - type: nauc_precision_at_20_max + value: 29.6844 + - type: nauc_precision_at_20_std + value: 9.4699 + - type: nauc_precision_at_20_diff1 + value: 15.9193 + - type: nauc_precision_at_100_max + value: 29.7036 + - type: nauc_precision_at_100_std + value: 19.0186 + - type: nauc_precision_at_100_diff1 + value: 5.9221 + - type: nauc_precision_at_1000_max + value: 24.6994 + - type: nauc_precision_at_1000_std + value: 18.0033 + - type: nauc_precision_at_1000_diff1 + value: -3.2275 + - type: nauc_mrr_at_1_max + value: 30.5442 + - type: nauc_mrr_at_1_std + value: -10.3888 + - type: nauc_mrr_at_1_diff1 + value: 52.476 + - type: nauc_mrr_at_3_max + value: 29.7504 + - type: nauc_mrr_at_3_std + value: -9.5234 + - type: nauc_mrr_at_3_diff1 + value: 46.5068 + - type: nauc_mrr_at_5_max + value: 30.341099999999997 + - type: nauc_mrr_at_5_std + value: -8.4966 + - type: nauc_mrr_at_5_diff1 + value: 46.051199999999994 + - type: nauc_mrr_at_10_max + value: 30.6066 + - type: nauc_mrr_at_10_std + value: -7.8854 + - type: nauc_mrr_at_10_diff1 + value: 46.035199999999996 + - type: nauc_mrr_at_20_max + value: 30.570199999999996 + - type: nauc_mrr_at_20_std + value: -7.614700000000001 + - type: nauc_mrr_at_20_diff1 + value: 45.8861 + - type: nauc_mrr_at_100_max + value: 30.589100000000002 + - type: nauc_mrr_at_100_std + value: -7.5529 + - type: nauc_mrr_at_100_diff1 + value: 45.907 + - type: nauc_mrr_at_1000_max + value: 30.587799999999998 + - type: nauc_mrr_at_1000_std + value: -7.5716 + - type: nauc_mrr_at_1000_diff1 + value: 45.9244 + - type: main_score + value: 42.732 + task: + type: Retrieval + - dataset: + config: default + name: MTEB CQADupstackUnixRetrieval (default) + revision: 6c6430d3a6d36f8d2a829195bc5dc94d7e063e53 + split: test + type: mteb/cqadupstack-unix + metrics: + - type: ndcg_at_1 + value: 18.843 + - type: ndcg_at_3 + value: 22.131 + - type: ndcg_at_5 + value: 23.772 + - type: ndcg_at_10 + value: 25.661 + - type: ndcg_at_20 + value: 27.939999999999998 + - type: ndcg_at_100 + value: 31.645 + - type: ndcg_at_1000 + value: 34.687 + - type: map_at_1 + value: 16.194 + - type: map_at_3 + value: 20.068 + - type: map_at_5 + value: 21.075 + - type: map_at_10 + value: 21.913 + - type: map_at_20 + value: 22.569 + - type: map_at_100 + value: 23.107 + - type: map_at_1000 + value: 23.23 + - type: recall_at_1 + value: 16.194 + - type: recall_at_3 + value: 24.704 + - type: recall_at_5 + value: 28.859 + - type: recall_at_10 + value: 34.402 + - type: recall_at_20 + value: 42.714 + - type: recall_at_100 + value: 61.19799999999999 + - type: recall_at_1000 + value: 82.953 + - type: precision_at_1 + value: 18.843 + - type: precision_at_3 + value: 9.919 + - type: precision_at_5 + value: 7.071 + - type: precision_at_10 + value: 4.328 + - type: precision_at_20 + value: 2.752 + - type: precision_at_100 + value: 0.823 + - type: precision_at_1000 + value: 0.121 + - type: mrr_at_1 + value: 18.8433 + - type: mrr_at_3 + value: 22.776699999999998 + - type: mrr_at_5 + value: 23.9055 + - type: mrr_at_10 + value: 24.7244 + - type: mrr_at_20 + value: 25.3919 + - type: mrr_at_100 + value: 25.8783 + - type: mrr_at_1000 + value: 25.957900000000002 + - type: nauc_ndcg_at_1_max + value: 35.1013 + - type: nauc_ndcg_at_1_std + value: 4.116899999999999 + - type: nauc_ndcg_at_1_diff1 + value: 54.3984 + - type: nauc_ndcg_at_3_max + value: 35.1035 + - type: nauc_ndcg_at_3_std + value: 5.3618 + - type: nauc_ndcg_at_3_diff1 + value: 47.4455 + - type: nauc_ndcg_at_5_max + value: 34.3845 + - type: nauc_ndcg_at_5_std + value: 5.4364 + - type: nauc_ndcg_at_5_diff1 + value: 44.8757 + - type: nauc_ndcg_at_10_max + value: 33.4252 + - type: nauc_ndcg_at_10_std + value: 7.100099999999999 + - type: nauc_ndcg_at_10_diff1 + value: 43.0854 + - type: nauc_ndcg_at_20_max + value: 33.2135 + - type: nauc_ndcg_at_20_std + value: 7.750500000000001 + - type: nauc_ndcg_at_20_diff1 + value: 42.5065 + - type: nauc_ndcg_at_100_max + value: 34.0845 + - type: nauc_ndcg_at_100_std + value: 9.0937 + - type: nauc_ndcg_at_100_diff1 + value: 40.9634 + - type: nauc_ndcg_at_1000_max + value: 34.3716 + - type: nauc_ndcg_at_1000_std + value: 9.8049 + - type: nauc_ndcg_at_1000_diff1 + value: 41.606 + - type: nauc_map_at_1_max + value: 35.054 + - type: nauc_map_at_1_std + value: 3.4526000000000003 + - type: nauc_map_at_1_diff1 + value: 55.69840000000001 + - type: nauc_map_at_3_max + value: 34.861 + - type: nauc_map_at_3_std + value: 4.6036 + - type: nauc_map_at_3_diff1 + value: 49.338 + - type: nauc_map_at_5_max + value: 34.3213 + - type: nauc_map_at_5_std + value: 4.7856000000000005 + - type: nauc_map_at_5_diff1 + value: 47.856 + - type: nauc_map_at_10_max + value: 33.9813 + - type: nauc_map_at_10_std + value: 5.649 + - type: nauc_map_at_10_diff1 + value: 47.0563 + - type: nauc_map_at_20_max + value: 33.8854 + - type: nauc_map_at_20_std + value: 5.9026000000000005 + - type: nauc_map_at_20_diff1 + value: 46.876200000000004 + - type: nauc_map_at_100_max + value: 33.996500000000005 + - type: nauc_map_at_100_std + value: 6.094200000000001 + - type: nauc_map_at_100_diff1 + value: 46.6388 + - type: nauc_map_at_1000_max + value: 34.0082 + - type: nauc_map_at_1000_std + value: 6.1436 + - type: nauc_map_at_1000_diff1 + value: 46.643 + - type: nauc_recall_at_1_max + value: 35.054 + - type: nauc_recall_at_1_std + value: 3.4526000000000003 + - type: nauc_recall_at_1_diff1 + value: 55.69840000000001 + - type: nauc_recall_at_3_max + value: 34.2271 + - type: nauc_recall_at_3_std + value: 5.573 + - type: nauc_recall_at_3_diff1 + value: 42.0593 + - type: nauc_recall_at_5_max + value: 32.7785 + - type: nauc_recall_at_5_std + value: 6.188599999999999 + - type: nauc_recall_at_5_diff1 + value: 36.9345 + - type: nauc_recall_at_10_max + value: 29.7004 + - type: nauc_recall_at_10_std + value: 10.3771 + - type: nauc_recall_at_10_diff1 + value: 31.6352 + - type: nauc_recall_at_20_max + value: 28.474100000000004 + - type: nauc_recall_at_20_std + value: 12.3244 + - type: nauc_recall_at_20_diff1 + value: 29.6458 + - type: nauc_recall_at_100_max + value: 31.2612 + - type: nauc_recall_at_100_std + value: 19.1574 + - type: nauc_recall_at_100_diff1 + value: 19.7616 + - type: nauc_recall_at_1000_max + value: 33.2982 + - type: nauc_recall_at_1000_std + value: 36.4068 + - type: nauc_recall_at_1000_diff1 + value: 15.3188 + - type: nauc_precision_at_1_max + value: 35.1013 + - type: nauc_precision_at_1_std + value: 4.116899999999999 + - type: nauc_precision_at_1_diff1 + value: 54.3984 + - type: nauc_precision_at_3_max + value: 34.4651 + - type: nauc_precision_at_3_std + value: 7.8735 + - type: nauc_precision_at_3_diff1 + value: 39.7844 + - type: nauc_precision_at_5_max + value: 32.2792 + - type: nauc_precision_at_5_std + value: 8.465 + - type: nauc_precision_at_5_diff1 + value: 34.130700000000004 + - type: nauc_precision_at_10_max + value: 28.197699999999998 + - type: nauc_precision_at_10_std + value: 12.1518 + - type: nauc_precision_at_10_diff1 + value: 28.672900000000002 + - type: nauc_precision_at_20_max + value: 27.2073 + - type: nauc_precision_at_20_std + value: 14.113100000000001 + - type: nauc_precision_at_20_diff1 + value: 23.623 + - type: nauc_precision_at_100_max + value: 22.906399999999998 + - type: nauc_precision_at_100_std + value: 16.7201 + - type: nauc_precision_at_100_diff1 + value: 7.0853 + - type: nauc_precision_at_1000_max + value: 10.5167 + - type: nauc_precision_at_1000_std + value: 11.5017 + - type: nauc_precision_at_1000_diff1 + value: -6.6079 + - type: nauc_mrr_at_1_max + value: 35.1013 + - type: nauc_mrr_at_1_std + value: 4.116899999999999 + - type: nauc_mrr_at_1_diff1 + value: 54.3984 + - type: nauc_mrr_at_3_max + value: 35.489399999999996 + - type: nauc_mrr_at_3_std + value: 5.097700000000001 + - type: nauc_mrr_at_3_diff1 + value: 48.8783 + - type: nauc_mrr_at_5_max + value: 35.2093 + - type: nauc_mrr_at_5_std + value: 5.2317 + - type: nauc_mrr_at_5_diff1 + value: 47.3602 + - type: nauc_mrr_at_10_max + value: 34.731 + - type: nauc_mrr_at_10_std + value: 5.7762 + - type: nauc_mrr_at_10_diff1 + value: 46.495999999999995 + - type: nauc_mrr_at_20_max + value: 34.6509 + - type: nauc_mrr_at_20_std + value: 5.8511 + - type: nauc_mrr_at_20_diff1 + value: 46.386500000000005 + - type: nauc_mrr_at_100_max + value: 34.7761 + - type: nauc_mrr_at_100_std + value: 6.0355 + - type: nauc_mrr_at_100_diff1 + value: 46.2476 + - type: nauc_mrr_at_1000_max + value: 34.792699999999996 + - type: nauc_mrr_at_1000_std + value: 6.0607 + - type: nauc_mrr_at_1000_diff1 + value: 46.281800000000004 + - type: main_score + value: 25.661 + task: + type: Retrieval + - dataset: + config: default + name: MTEB ClimateFEVERHardNegatives (default) + revision: 3a309e201f3c2c4b13bd4a367a8f37eee2ec1d21 + split: test + type: mteb/ClimateFEVER_test_top_250_only_w_correct-v2 + metrics: + - type: ndcg_at_1 + value: 16.8 + - type: ndcg_at_3 + value: 15.503 + - type: ndcg_at_5 + value: 17.5 + - type: ndcg_at_10 + value: 20.642 + - type: ndcg_at_20 + value: 23.07 + - type: ndcg_at_100 + value: 27.639000000000003 + - type: ndcg_at_1000 + value: 32.041 + - type: map_at_1 + value: 7.885000000000001 + - type: map_at_3 + value: 11.128 + - type: map_at_5 + value: 12.565999999999999 + - type: map_at_10 + value: 13.876 + - type: map_at_20 + value: 14.66 + - type: map_at_100 + value: 15.432000000000002 + - type: map_at_1000 + value: 15.655 + - type: recall_at_1 + value: 7.885000000000001 + - type: recall_at_3 + value: 14.957 + - type: recall_at_5 + value: 19.675 + - type: recall_at_10 + value: 26.868 + - type: recall_at_20 + value: 33.94 + - type: recall_at_100 + value: 51.833 + - type: recall_at_1000 + value: 76.822 + - type: precision_at_1 + value: 16.8 + - type: precision_at_3 + value: 11.533 + - type: precision_at_5 + value: 9.56 + - type: precision_at_10 + value: 6.83 + - type: precision_at_20 + value: 4.41 + - type: precision_at_100 + value: 1.432 + - type: precision_at_1000 + value: 0.22499999999999998 + - type: mrr_at_1 + value: 16.8 + - type: mrr_at_3 + value: 23.2333 + - type: mrr_at_5 + value: 25.2183 + - type: mrr_at_10 + value: 26.775 + - type: mrr_at_20 + value: 27.4121 + - type: mrr_at_100 + value: 27.882299999999997 + - type: mrr_at_1000 + value: 27.9472 + - type: nauc_ndcg_at_1_max + value: 28.3609 + - type: nauc_ndcg_at_1_std + value: 10.5951 + - type: nauc_ndcg_at_1_diff1 + value: 16.566 + - type: nauc_ndcg_at_3_max + value: 33.3794 + - type: nauc_ndcg_at_3_std + value: 14.645900000000001 + - type: nauc_ndcg_at_3_diff1 + value: 15.4617 + - type: nauc_ndcg_at_5_max + value: 33.5092 + - type: nauc_ndcg_at_5_std + value: 16.209699999999998 + - type: nauc_ndcg_at_5_diff1 + value: 16.7386 + - type: nauc_ndcg_at_10_max + value: 37.101299999999995 + - type: nauc_ndcg_at_10_std + value: 20.939 + - type: nauc_ndcg_at_10_diff1 + value: 15.1232 + - type: nauc_ndcg_at_20_max + value: 38.3563 + - type: nauc_ndcg_at_20_std + value: 22.3038 + - type: nauc_ndcg_at_20_diff1 + value: 14.613100000000001 + - type: nauc_ndcg_at_100_max + value: 39.5793 + - type: nauc_ndcg_at_100_std + value: 23.3348 + - type: nauc_ndcg_at_100_diff1 + value: 13.6571 + - type: nauc_ndcg_at_1000_max + value: 39.2582 + - type: nauc_ndcg_at_1000_std + value: 22.5989 + - type: nauc_ndcg_at_1000_diff1 + value: 12.6784 + - type: nauc_map_at_1_max + value: 36.9819 + - type: nauc_map_at_1_std + value: 11.5065 + - type: nauc_map_at_1_diff1 + value: 22.4791 + - type: nauc_map_at_3_max + value: 35.324299999999994 + - type: nauc_map_at_3_std + value: 13.572000000000001 + - type: nauc_map_at_3_diff1 + value: 19.3415 + - type: nauc_map_at_5_max + value: 35.0138 + - type: nauc_map_at_5_std + value: 14.857600000000001 + - type: nauc_map_at_5_diff1 + value: 19.5352 + - type: nauc_map_at_10_max + value: 36.8267 + - type: nauc_map_at_10_std + value: 17.6287 + - type: nauc_map_at_10_diff1 + value: 18.2802 + - type: nauc_map_at_20_max + value: 37.5214 + - type: nauc_map_at_20_std + value: 18.319399999999998 + - type: nauc_map_at_20_diff1 + value: 18.0343 + - type: nauc_map_at_100_max + value: 37.933499999999995 + - type: nauc_map_at_100_std + value: 18.6864 + - type: nauc_map_at_100_diff1 + value: 17.7119 + - type: nauc_map_at_1000_max + value: 37.9509 + - type: nauc_map_at_1000_std + value: 18.6975 + - type: nauc_map_at_1000_diff1 + value: 17.5997 + - type: nauc_recall_at_1_max + value: 36.9819 + - type: nauc_recall_at_1_std + value: 11.5065 + - type: nauc_recall_at_1_diff1 + value: 22.4791 + - type: nauc_recall_at_3_max + value: 33.0875 + - type: nauc_recall_at_3_std + value: 16.3976 + - type: nauc_recall_at_3_diff1 + value: 15.6164 + - type: nauc_recall_at_5_max + value: 30.604799999999997 + - type: nauc_recall_at_5_std + value: 17.1699 + - type: nauc_recall_at_5_diff1 + value: 15.639800000000001 + - type: nauc_recall_at_10_max + value: 35.342400000000005 + - type: nauc_recall_at_10_std + value: 24.665599999999998 + - type: nauc_recall_at_10_diff1 + value: 11.9499 + - type: nauc_recall_at_20_max + value: 35.956700000000005 + - type: nauc_recall_at_20_std + value: 26.556800000000003 + - type: nauc_recall_at_20_diff1 + value: 10.0239 + - type: nauc_recall_at_100_max + value: 36.1012 + - type: nauc_recall_at_100_std + value: 27.8055 + - type: nauc_recall_at_100_diff1 + value: 6.3591 + - type: nauc_recall_at_1000_max + value: 34.7202 + - type: nauc_recall_at_1000_std + value: 26.378 + - type: nauc_recall_at_1000_diff1 + value: -0.7171000000000001 + - type: nauc_precision_at_1_max + value: 28.3609 + - type: nauc_precision_at_1_std + value: 10.5951 + - type: nauc_precision_at_1_diff1 + value: 16.566 + - type: nauc_precision_at_3_max + value: 30.490000000000002 + - type: nauc_precision_at_3_std + value: 16.270899999999997 + - type: nauc_precision_at_3_diff1 + value: 9.7026 + - type: nauc_precision_at_5_max + value: 29.3491 + - type: nauc_precision_at_5_std + value: 19.084699999999998 + - type: nauc_precision_at_5_diff1 + value: 10.7809 + - type: nauc_precision_at_10_max + value: 34.753699999999995 + - type: nauc_precision_at_10_std + value: 28.155 + - type: nauc_precision_at_10_diff1 + value: 5.6554 + - type: nauc_precision_at_20_max + value: 33.3812 + - type: nauc_precision_at_20_std + value: 27.122400000000003 + - type: nauc_precision_at_20_diff1 + value: 3.6636 + - type: nauc_precision_at_100_max + value: 28.7799 + - type: nauc_precision_at_100_std + value: 23.9905 + - type: nauc_precision_at_100_diff1 + value: -0.5301 + - type: nauc_precision_at_1000_max + value: 13.068399999999999 + - type: nauc_precision_at_1000_std + value: 12.9133 + - type: nauc_precision_at_1000_diff1 + value: -8.8717 + - type: nauc_mrr_at_1_max + value: 28.3609 + - type: nauc_mrr_at_1_std + value: 10.5951 + - type: nauc_mrr_at_1_diff1 + value: 16.566 + - type: nauc_mrr_at_3_max + value: 30.9311 + - type: nauc_mrr_at_3_std + value: 13.9549 + - type: nauc_mrr_at_3_diff1 + value: 12.851399999999998 + - type: nauc_mrr_at_5_max + value: 30.893700000000003 + - type: nauc_mrr_at_5_std + value: 14.464599999999999 + - type: nauc_mrr_at_5_diff1 + value: 13.2001 + - type: nauc_mrr_at_10_max + value: 32.277499999999996 + - type: nauc_mrr_at_10_std + value: 15.9378 + - type: nauc_mrr_at_10_diff1 + value: 12.9887 + - type: nauc_mrr_at_20_max + value: 32.3817 + - type: nauc_mrr_at_20_std + value: 16.0469 + - type: nauc_mrr_at_20_diff1 + value: 13.039200000000001 + - type: nauc_mrr_at_100_max + value: 32.386900000000004 + - type: nauc_mrr_at_100_std + value: 15.966800000000001 + - type: nauc_mrr_at_100_diff1 + value: 12.982 + - type: nauc_mrr_at_1000_max + value: 32.347300000000004 + - type: nauc_mrr_at_1000_std + value: 15.9096 + - type: nauc_mrr_at_1000_diff1 + value: 12.9742 + - type: main_score + value: 20.642 + task: + type: Retrieval + - dataset: + config: default + name: MTEB FEVERHardNegatives (default) + revision: 080c9ed6267b65029207906e815d44a9240bafca + split: test + type: mteb/FEVER_test_top_250_only_w_correct-v2 + metrics: + - type: ndcg_at_1 + value: 46.9 + - type: ndcg_at_3 + value: 57.825 + - type: ndcg_at_5 + value: 61.245000000000005 + - type: ndcg_at_10 + value: 63.836000000000006 + - type: ndcg_at_20 + value: 65.408 + - type: ndcg_at_100 + value: 66.796 + - type: ndcg_at_1000 + value: 67.216 + - type: map_at_1 + value: 43.999 + - type: map_at_3 + value: 53.813 + - type: map_at_5 + value: 55.741 + - type: map_at_10 + value: 56.852999999999994 + - type: map_at_20 + value: 57.30800000000001 + - type: map_at_100 + value: 57.54 + - type: map_at_1000 + value: 57.56099999999999 + - type: recall_at_1 + value: 43.999 + - type: recall_at_3 + value: 66.184 + - type: recall_at_5 + value: 74.557 + - type: recall_at_10 + value: 82.394 + - type: recall_at_20 + value: 88.51 + - type: recall_at_100 + value: 95.253 + - type: recall_at_1000 + value: 98.031 + - type: precision_at_1 + value: 46.9 + - type: precision_at_3 + value: 23.599999999999998 + - type: precision_at_5 + value: 15.98 + - type: precision_at_10 + value: 8.85 + - type: precision_at_20 + value: 4.760000000000001 + - type: precision_at_100 + value: 1.045 + - type: precision_at_1000 + value: 0.11 + - type: mrr_at_1 + value: 46.9 + - type: mrr_at_3 + value: 57.0167 + - type: mrr_at_5 + value: 59.046699999999994 + - type: mrr_at_10 + value: 60.1422 + - type: mrr_at_20 + value: 60.535799999999995 + - type: mrr_at_100 + value: 60.716 + - type: mrr_at_1000 + value: 60.7232 + - type: nauc_ndcg_at_1_max + value: 12.741900000000001 + - type: nauc_ndcg_at_1_std + value: -20.011000000000003 + - type: nauc_ndcg_at_1_diff1 + value: 51.02100000000001 + - type: nauc_ndcg_at_3_max + value: 17.416400000000003 + - type: nauc_ndcg_at_3_std + value: -20.9336 + - type: nauc_ndcg_at_3_diff1 + value: 46.3134 + - type: nauc_ndcg_at_5_max + value: 18.2369 + - type: nauc_ndcg_at_5_std + value: -21.5645 + - type: nauc_ndcg_at_5_diff1 + value: 46.261799999999994 + - type: nauc_ndcg_at_10_max + value: 18.8528 + - type: nauc_ndcg_at_10_std + value: -20.6893 + - type: nauc_ndcg_at_10_diff1 + value: 46.5862 + - type: nauc_ndcg_at_20_max + value: 18.0211 + - type: nauc_ndcg_at_20_std + value: -19.652 + - type: nauc_ndcg_at_20_diff1 + value: 46.5482 + - type: nauc_ndcg_at_100_max + value: 17.766000000000002 + - type: nauc_ndcg_at_100_std + value: -18.7245 + - type: nauc_ndcg_at_100_diff1 + value: 47.0345 + - type: nauc_ndcg_at_1000_max + value: 17.596500000000002 + - type: nauc_ndcg_at_1000_std + value: -19.0628 + - type: nauc_ndcg_at_1000_diff1 + value: 47.12 + - type: nauc_map_at_1_max + value: 13.017599999999998 + - type: nauc_map_at_1_std + value: -18.8296 + - type: nauc_map_at_1_diff1 + value: 49.8762 + - type: nauc_map_at_3_max + value: 16.2438 + - type: nauc_map_at_3_std + value: -20.1711 + - type: nauc_map_at_3_diff1 + value: 47.2236 + - type: nauc_map_at_5_max + value: 16.541 + - type: nauc_map_at_5_std + value: -20.4952 + - type: nauc_map_at_5_diff1 + value: 47.1971 + - type: nauc_map_at_10_max + value: 16.7266 + - type: nauc_map_at_10_std + value: -20.1189 + - type: nauc_map_at_10_diff1 + value: 47.2762 + - type: nauc_map_at_20_max + value: 16.5198 + - type: nauc_map_at_20_std + value: -19.8167 + - type: nauc_map_at_20_diff1 + value: 47.266799999999996 + - type: nauc_map_at_100_max + value: 16.467200000000002 + - type: nauc_map_at_100_std + value: -19.7016 + - type: nauc_map_at_100_diff1 + value: 47.3389 + - type: nauc_map_at_1000_max + value: 16.466900000000003 + - type: nauc_map_at_1000_std + value: -19.704 + - type: nauc_map_at_1000_diff1 + value: 47.341 + - type: nauc_recall_at_1_max + value: 13.017599999999998 + - type: nauc_recall_at_1_std + value: -18.8296 + - type: nauc_recall_at_1_diff1 + value: 49.8762 + - type: nauc_recall_at_3_max + value: 20.579700000000003 + - type: nauc_recall_at_3_std + value: -21.263399999999997 + - type: nauc_recall_at_3_diff1 + value: 40.7412 + - type: nauc_recall_at_5_max + value: 23.308799999999998 + - type: nauc_recall_at_5_std + value: -23.0915 + - type: nauc_recall_at_5_diff1 + value: 38.2001 + - type: nauc_recall_at_10_max + value: 27.296 + - type: nauc_recall_at_10_std + value: -19.2697 + - type: nauc_recall_at_10_diff1 + value: 35.9711 + - type: nauc_recall_at_20_max + value: 23.9957 + - type: nauc_recall_at_20_std + value: -10.1564 + - type: nauc_recall_at_20_diff1 + value: 30.5332 + - type: nauc_recall_at_100_max + value: 27.0148 + - type: nauc_recall_at_100_std + value: 25.655299999999997 + - type: nauc_recall_at_100_diff1 + value: 23.1136 + - type: nauc_recall_at_1000_max + value: 28.9392 + - type: nauc_recall_at_1000_std + value: 47.491 + - type: nauc_recall_at_1000_diff1 + value: 15.6225 + - type: nauc_precision_at_1_max + value: 12.741900000000001 + - type: nauc_precision_at_1_std + value: -20.011000000000003 + - type: nauc_precision_at_1_diff1 + value: 51.02100000000001 + - type: nauc_precision_at_3_max + value: 20.477999999999998 + - type: nauc_precision_at_3_std + value: -24.4646 + - type: nauc_precision_at_3_diff1 + value: 41.1551 + - type: nauc_precision_at_5_max + value: 24.364 + - type: nauc_precision_at_5_std + value: -27.1997 + - type: nauc_precision_at_5_diff1 + value: 38.9501 + - type: nauc_precision_at_10_max + value: 30.684299999999997 + - type: nauc_precision_at_10_std + value: -23.1531 + - type: nauc_precision_at_10_diff1 + value: 34.6829 + - type: nauc_precision_at_20_max + value: 24.1828 + - type: nauc_precision_at_20_std + value: -10.783800000000001 + - type: nauc_precision_at_20_diff1 + value: 22.662399999999998 + - type: nauc_precision_at_100_max + value: 12.189 + - type: nauc_precision_at_100_std + value: 10.600999999999999 + - type: nauc_precision_at_100_diff1 + value: -0.2197 + - type: nauc_precision_at_1000_max + value: 1.1533 + - type: nauc_precision_at_1000_std + value: 6.2423 + - type: nauc_precision_at_1000_diff1 + value: -10.4662 + - type: nauc_mrr_at_1_max + value: 12.741900000000001 + - type: nauc_mrr_at_1_std + value: -20.011000000000003 + - type: nauc_mrr_at_1_diff1 + value: 51.02100000000001 + - type: nauc_mrr_at_3_max + value: 16.4501 + - type: nauc_mrr_at_3_std + value: -21.337500000000002 + - type: nauc_mrr_at_3_diff1 + value: 48.4594 + - type: nauc_mrr_at_5_max + value: 16.8928 + - type: nauc_mrr_at_5_std + value: -21.7254 + - type: nauc_mrr_at_5_diff1 + value: 48.619299999999996 + - type: nauc_mrr_at_10_max + value: 17.0057 + - type: nauc_mrr_at_10_std + value: -21.465899999999998 + - type: nauc_mrr_at_10_diff1 + value: 48.848200000000006 + - type: nauc_mrr_at_20_max + value: 16.745099999999997 + - type: nauc_mrr_at_20_std + value: -21.2914 + - type: nauc_mrr_at_20_diff1 + value: 48.861900000000006 + - type: nauc_mrr_at_100_max + value: 16.653399999999998 + - type: nauc_mrr_at_100_std + value: -21.1954 + - type: nauc_mrr_at_100_diff1 + value: 48.9097 + - type: nauc_mrr_at_1000_max + value: 16.650000000000002 + - type: nauc_mrr_at_1000_std + value: -21.2048 + - type: nauc_mrr_at_1000_diff1 + value: 48.911500000000004 + - type: main_score + value: 63.836000000000006 + task: + type: Retrieval + - dataset: + config: default + name: MTEB FiQA2018 (default) + revision: 27a168819829fe9bcd655c2df245fb19452e8e06 + split: test + type: mteb/fiqa + metrics: + - type: ndcg_at_1 + value: 25.154 + - type: ndcg_at_3 + value: 22.85 + - type: ndcg_at_5 + value: 23.788999999999998 + - type: ndcg_at_10 + value: 25.657000000000004 + - type: ndcg_at_20 + value: 28.058 + - type: ndcg_at_100 + value: 32.019999999999996 + - type: ndcg_at_1000 + value: 36.124 + - type: map_at_1 + value: 12.594 + - type: map_at_3 + value: 17.345 + - type: map_at_5 + value: 18.740000000000002 + - type: map_at_10 + value: 19.871 + - type: map_at_20 + value: 20.71 + - type: map_at_100 + value: 21.404 + - type: map_at_1000 + value: 21.616 + - type: recall_at_1 + value: 12.594 + - type: recall_at_3 + value: 20.682000000000002 + - type: recall_at_5 + value: 24.735 + - type: recall_at_10 + value: 30.217 + - type: recall_at_20 + value: 37.714999999999996 + - type: recall_at_100 + value: 54.364000000000004 + - type: recall_at_1000 + value: 79.487 + - type: precision_at_1 + value: 25.154 + - type: precision_at_3 + value: 15.174999999999999 + - type: precision_at_5 + value: 11.235000000000001 + - type: precision_at_10 + value: 7.13 + - type: precision_at_20 + value: 4.522 + - type: precision_at_100 + value: 1.341 + - type: precision_at_1000 + value: 0.20500000000000002 + - type: mrr_at_1 + value: 25.154300000000003 + - type: mrr_at_3 + value: 30.324099999999998 + - type: mrr_at_5 + value: 31.581799999999998 + - type: mrr_at_10 + value: 32.5208 + - type: mrr_at_20 + value: 33.055 + - type: mrr_at_100 + value: 33.4738 + - type: mrr_at_1000 + value: 33.5533 + - type: nauc_ndcg_at_1_max + value: 20.836199999999998 + - type: nauc_ndcg_at_1_std + value: -2.4346 + - type: nauc_ndcg_at_1_diff1 + value: 41.3264 + - type: nauc_ndcg_at_3_max + value: 21.4673 + - type: nauc_ndcg_at_3_std + value: -0.35760000000000003 + - type: nauc_ndcg_at_3_diff1 + value: 36.5457 + - type: nauc_ndcg_at_5_max + value: 21.0022 + - type: nauc_ndcg_at_5_std + value: 0.30079999999999996 + - type: nauc_ndcg_at_5_diff1 + value: 35.1377 + - type: nauc_ndcg_at_10_max + value: 21.4511 + - type: nauc_ndcg_at_10_std + value: 1.9931 + - type: nauc_ndcg_at_10_diff1 + value: 35.367599999999996 + - type: nauc_ndcg_at_20_max + value: 21.9794 + - type: nauc_ndcg_at_20_std + value: 3.2666 + - type: nauc_ndcg_at_20_diff1 + value: 33.9954 + - type: nauc_ndcg_at_100_max + value: 22.666900000000002 + - type: nauc_ndcg_at_100_std + value: 6.1648000000000005 + - type: nauc_ndcg_at_100_diff1 + value: 32.5715 + - type: nauc_ndcg_at_1000_max + value: 23.9645 + - type: nauc_ndcg_at_1000_std + value: 7.031 + - type: nauc_ndcg_at_1000_diff1 + value: 32.6535 + - type: nauc_map_at_1_max + value: 13.436699999999998 + - type: nauc_map_at_1_std + value: -6.1377 + - type: nauc_map_at_1_diff1 + value: 46.1518 + - type: nauc_map_at_3_max + value: 17.6491 + - type: nauc_map_at_3_std + value: -3.3383000000000003 + - type: nauc_map_at_3_diff1 + value: 39.909800000000004 + - type: nauc_map_at_5_max + value: 18.4969 + - type: nauc_map_at_5_std + value: -1.8129 + - type: nauc_map_at_5_diff1 + value: 38.4072 + - type: nauc_map_at_10_max + value: 19.4823 + - type: nauc_map_at_10_std + value: -0.2211 + - type: nauc_map_at_10_diff1 + value: 38.1346 + - type: nauc_map_at_20_max + value: 19.9898 + - type: nauc_map_at_20_std + value: 0.6002000000000001 + - type: nauc_map_at_20_diff1 + value: 37.755100000000006 + - type: nauc_map_at_100_max + value: 20.2321 + - type: nauc_map_at_100_std + value: 1.2189999999999999 + - type: nauc_map_at_100_diff1 + value: 37.379 + - type: nauc_map_at_1000_max + value: 20.3676 + - type: nauc_map_at_1000_std + value: 1.3561999999999999 + - type: nauc_map_at_1000_diff1 + value: 37.3216 + - type: nauc_recall_at_1_max + value: 13.436699999999998 + - type: nauc_recall_at_1_std + value: -6.1377 + - type: nauc_recall_at_1_diff1 + value: 46.1518 + - type: nauc_recall_at_3_max + value: 17.4283 + - type: nauc_recall_at_3_std + value: -2.0456 + - type: nauc_recall_at_3_diff1 + value: 34.5422 + - type: nauc_recall_at_5_max + value: 18.2169 + - type: nauc_recall_at_5_std + value: 0.7002 + - type: nauc_recall_at_5_diff1 + value: 29.7798 + - type: nauc_recall_at_10_max + value: 19.6832 + - type: nauc_recall_at_10_std + value: 4.6769 + - type: nauc_recall_at_10_diff1 + value: 27.8829 + - type: nauc_recall_at_20_max + value: 20.095 + - type: nauc_recall_at_20_std + value: 6.884899999999999 + - type: nauc_recall_at_20_diff1 + value: 22.7741 + - type: nauc_recall_at_100_max + value: 20.5351 + - type: nauc_recall_at_100_std + value: 19.2636 + - type: nauc_recall_at_100_diff1 + value: 16.2238 + - type: nauc_recall_at_1000_max + value: 27.9838 + - type: nauc_recall_at_1000_std + value: 33.3099 + - type: nauc_recall_at_1000_diff1 + value: 12.701699999999999 + - type: nauc_precision_at_1_max + value: 20.836199999999998 + - type: nauc_precision_at_1_std + value: -2.4346 + - type: nauc_precision_at_1_diff1 + value: 41.3264 + - type: nauc_precision_at_3_max + value: 26.558500000000002 + - type: nauc_precision_at_3_std + value: 3.6578 + - type: nauc_precision_at_3_diff1 + value: 27.0323 + - type: nauc_precision_at_5_max + value: 28.794199999999996 + - type: nauc_precision_at_5_std + value: 8.6533 + - type: nauc_precision_at_5_diff1 + value: 21.9488 + - type: nauc_precision_at_10_max + value: 29.7713 + - type: nauc_precision_at_10_std + value: 13.645399999999999 + - type: nauc_precision_at_10_diff1 + value: 20.1386 + - type: nauc_precision_at_20_max + value: 28.0465 + - type: nauc_precision_at_20_std + value: 16.3569 + - type: nauc_precision_at_20_diff1 + value: 14.969299999999999 + - type: nauc_precision_at_100_max + value: 26.7123 + - type: nauc_precision_at_100_std + value: 19.1407 + - type: nauc_precision_at_100_diff1 + value: 5.7822 + - type: nauc_precision_at_1000_max + value: 23.6681 + - type: nauc_precision_at_1000_std + value: 16.3438 + - type: nauc_precision_at_1000_diff1 + value: -3.3699 + - type: nauc_mrr_at_1_max + value: 20.836199999999998 + - type: nauc_mrr_at_1_std + value: -2.4346 + - type: nauc_mrr_at_1_diff1 + value: 41.3264 + - type: nauc_mrr_at_3_max + value: 22.4267 + - type: nauc_mrr_at_3_std + value: -0.1948 + - type: nauc_mrr_at_3_diff1 + value: 36.9255 + - type: nauc_mrr_at_5_max + value: 22.6662 + - type: nauc_mrr_at_5_std + value: 0.4444 + - type: nauc_mrr_at_5_diff1 + value: 35.957 + - type: nauc_mrr_at_10_max + value: 22.5111 + - type: nauc_mrr_at_10_std + value: 0.7020000000000001 + - type: nauc_mrr_at_10_diff1 + value: 35.6976 + - type: nauc_mrr_at_20_max + value: 22.4416 + - type: nauc_mrr_at_20_std + value: 0.8706999999999999 + - type: nauc_mrr_at_20_diff1 + value: 35.2034 + - type: nauc_mrr_at_100_max + value: 22.4571 + - type: nauc_mrr_at_100_std + value: 1.0563 + - type: nauc_mrr_at_100_diff1 + value: 35.177 + - type: nauc_mrr_at_1000_max + value: 22.4743 + - type: nauc_mrr_at_1000_std + value: 1.0505 + - type: nauc_mrr_at_1000_diff1 + value: 35.2186 + - type: main_score + value: 25.657000000000004 + task: + type: Retrieval + - dataset: + config: default + name: MTEB HotpotQAHardNegatives (default) + revision: 617612fa63afcb60e3b134bed8b7216a99707c37 + split: test + type: mteb/HotpotQA_test_top_250_only_w_correct-v2 + metrics: + - type: ndcg_at_1 + value: 58.9 + - type: ndcg_at_3 + value: 45.092999999999996 + - type: ndcg_at_5 + value: 47.806 + - type: ndcg_at_10 + value: 50.666 + - type: ndcg_at_20 + value: 52.644000000000005 + - type: ndcg_at_100 + value: 56.071000000000005 + - type: ndcg_at_1000 + value: 58.262 + - type: map_at_1 + value: 29.45 + - type: map_at_3 + value: 37.675 + - type: map_at_5 + value: 39.562999999999995 + - type: map_at_10 + value: 41.056 + - type: map_at_20 + value: 41.765 + - type: map_at_100 + value: 42.425000000000004 + - type: map_at_1000 + value: 42.54 + - type: recall_at_1 + value: 29.45 + - type: recall_at_3 + value: 41.75 + - type: recall_at_5 + value: 47.099999999999994 + - type: recall_at_10 + value: 54.300000000000004 + - type: recall_at_20 + value: 60.699999999999996 + - type: recall_at_100 + value: 75.9 + - type: recall_at_1000 + value: 90.3 + - type: precision_at_1 + value: 58.9 + - type: precision_at_3 + value: 27.833000000000002 + - type: precision_at_5 + value: 18.84 + - type: precision_at_10 + value: 10.86 + - type: precision_at_20 + value: 6.069999999999999 + - type: precision_at_100 + value: 1.518 + - type: precision_at_1000 + value: 0.181 + - type: mrr_at_1 + value: 58.9 + - type: mrr_at_3 + value: 64.81670000000001 + - type: mrr_at_5 + value: 65.9717 + - type: mrr_at_10 + value: 66.84750000000001 + - type: mrr_at_20 + value: 67.1864 + - type: mrr_at_100 + value: 67.3796 + - type: mrr_at_1000 + value: 67.3962 + - type: nauc_ndcg_at_1_max + value: 40.6699 + - type: nauc_ndcg_at_1_std + value: -6.4051 + - type: nauc_ndcg_at_1_diff1 + value: 61.4074 + - type: nauc_ndcg_at_3_max + value: 36.086200000000005 + - type: nauc_ndcg_at_3_std + value: -3.8372 + - type: nauc_ndcg_at_3_diff1 + value: 44.0991 + - type: nauc_ndcg_at_5_max + value: 35.1661 + - type: nauc_ndcg_at_5_std + value: -3.4778000000000002 + - type: nauc_ndcg_at_5_diff1 + value: 41.2298 + - type: nauc_ndcg_at_10_max + value: 34.5689 + - type: nauc_ndcg_at_10_std + value: -0.7254 + - type: nauc_ndcg_at_10_diff1 + value: 38.9824 + - type: nauc_ndcg_at_20_max + value: 35.4153 + - type: nauc_ndcg_at_20_std + value: 0.9502999999999999 + - type: nauc_ndcg_at_20_diff1 + value: 38.5558 + - type: nauc_ndcg_at_100_max + value: 36.187799999999996 + - type: nauc_ndcg_at_100_std + value: 3.3059 + - type: nauc_ndcg_at_100_diff1 + value: 37.775 + - type: nauc_ndcg_at_1000_max + value: 36.9076 + - type: nauc_ndcg_at_1000_std + value: 3.2030000000000003 + - type: nauc_ndcg_at_1000_diff1 + value: 39.6691 + - type: nauc_map_at_1_max + value: 40.6699 + - type: nauc_map_at_1_std + value: -6.4051 + - type: nauc_map_at_1_diff1 + value: 61.4074 + - type: nauc_map_at_3_max + value: 34.8654 + - type: nauc_map_at_3_std + value: -1.9401000000000002 + - type: nauc_map_at_3_diff1 + value: 40.4559 + - type: nauc_map_at_5_max + value: 34.0362 + - type: nauc_map_at_5_std + value: -1.677 + - type: nauc_map_at_5_diff1 + value: 38.384 + - type: nauc_map_at_10_max + value: 33.8136 + - type: nauc_map_at_10_std + value: -0.2753 + - type: nauc_map_at_10_diff1 + value: 37.1326 + - type: nauc_map_at_20_max + value: 34.1981 + - type: nauc_map_at_20_std + value: 0.2882 + - type: nauc_map_at_20_diff1 + value: 36.996 + - type: nauc_map_at_100_max + value: 34.2694 + - type: nauc_map_at_100_std + value: 0.596 + - type: nauc_map_at_100_diff1 + value: 36.858200000000004 + - type: nauc_map_at_1000_max + value: 34.3301 + - type: nauc_map_at_1000_std + value: 0.6459 + - type: nauc_map_at_1000_diff1 + value: 36.9437 + - type: nauc_recall_at_1_max + value: 40.6699 + - type: nauc_recall_at_1_std + value: -6.4051 + - type: nauc_recall_at_1_diff1 + value: 61.4074 + - type: nauc_recall_at_3_max + value: 33.4227 + - type: nauc_recall_at_3_std + value: -2.6978 + - type: nauc_recall_at_3_diff1 + value: 35.5329 + - type: nauc_recall_at_5_max + value: 29.759900000000002 + - type: nauc_recall_at_5_std + value: -1.7928 + - type: nauc_recall_at_5_diff1 + value: 27.8553 + - type: nauc_recall_at_10_max + value: 27.2765 + - type: nauc_recall_at_10_std + value: 5.0284 + - type: nauc_recall_at_10_diff1 + value: 21.5188 + - type: nauc_recall_at_20_max + value: 27.456500000000002 + - type: nauc_recall_at_20_std + value: 10.4452 + - type: nauc_recall_at_20_diff1 + value: 17.377100000000002 + - type: nauc_recall_at_100_max + value: 27.960400000000003 + - type: nauc_recall_at_100_std + value: 26.0653 + - type: nauc_recall_at_100_diff1 + value: 5.9226 + - type: nauc_recall_at_1000_max + value: 33.996700000000004 + - type: nauc_recall_at_1000_std + value: 44.291199999999996 + - type: nauc_recall_at_1000_diff1 + value: 7.6986 + - type: nauc_precision_at_1_max + value: 40.6699 + - type: nauc_precision_at_1_std + value: -6.4051 + - type: nauc_precision_at_1_diff1 + value: 61.4074 + - type: nauc_precision_at_3_max + value: 33.4227 + - type: nauc_precision_at_3_std + value: -2.6978 + - type: nauc_precision_at_3_diff1 + value: 35.5329 + - type: nauc_precision_at_5_max + value: 29.759900000000002 + - type: nauc_precision_at_5_std + value: -1.7928 + - type: nauc_precision_at_5_diff1 + value: 27.8553 + - type: nauc_precision_at_10_max + value: 27.2765 + - type: nauc_precision_at_10_std + value: 5.0284 + - type: nauc_precision_at_10_diff1 + value: 21.5188 + - type: nauc_precision_at_20_max + value: 27.456500000000002 + - type: nauc_precision_at_20_std + value: 10.4452 + - type: nauc_precision_at_20_diff1 + value: 17.377100000000002 + - type: nauc_precision_at_100_max + value: 27.960400000000003 + - type: nauc_precision_at_100_std + value: 26.0653 + - type: nauc_precision_at_100_diff1 + value: 5.9226 + - type: nauc_precision_at_1000_max + value: 33.996700000000004 + - type: nauc_precision_at_1000_std + value: 44.291199999999996 + - type: nauc_precision_at_1000_diff1 + value: 7.6986 + - type: nauc_mrr_at_1_max + value: 40.6699 + - type: nauc_mrr_at_1_std + value: -6.4051 + - type: nauc_mrr_at_1_diff1 + value: 61.4074 + - type: nauc_mrr_at_3_max + value: 40.4193 + - type: nauc_mrr_at_3_std + value: -8.072899999999999 + - type: nauc_mrr_at_3_diff1 + value: 58.589400000000005 + - type: nauc_mrr_at_5_max + value: 40.6559 + - type: nauc_mrr_at_5_std + value: -8.1937 + - type: nauc_mrr_at_5_diff1 + value: 58.30650000000001 + - type: nauc_mrr_at_10_max + value: 40.515699999999995 + - type: nauc_mrr_at_10_std + value: -7.4325 + - type: nauc_mrr_at_10_diff1 + value: 58.1284 + - type: nauc_mrr_at_20_max + value: 40.63 + - type: nauc_mrr_at_20_std + value: -7.1578 + - type: nauc_mrr_at_20_diff1 + value: 58.215799999999994 + - type: nauc_mrr_at_100_max + value: 40.693 + - type: nauc_mrr_at_100_std + value: -7.0889 + - type: nauc_mrr_at_100_diff1 + value: 58.22389999999999 + - type: nauc_mrr_at_1000_max + value: 40.700900000000004 + - type: nauc_mrr_at_1000_std + value: -7.098400000000001 + - type: nauc_mrr_at_1000_diff1 + value: 58.2458 + - type: main_score + value: 50.666 + task: + type: Retrieval + - dataset: + config: default + name: MTEB ImdbClassification (default) + revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7 + split: test + type: mteb/imdb + metrics: + - type: accuracy + value: 68.1712 + - type: f1 + value: 67.982 + - type: f1_weighted + value: 67.982 + - type: ap + value: 62.572799999999994 + - type: ap_weighted + value: 62.572799999999994 + - type: main_score + value: 68.1712 + task: + type: Classification + - dataset: + config: en + name: MTEB MTOPDomainClassification (en) + revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf + split: test + type: mteb/mtop_domain + metrics: + - type: accuracy + value: 90.4423 + - type: f1 + value: 90.08840000000001 + - type: f1_weighted + value: 90.44919999999999 + - type: main_score + value: 90.4423 + task: + type: Classification + - dataset: + config: en + name: MTEB MassiveIntentClassification (en) + revision: 4672e20407010da34463acc759c162ca9734bca6 + split: test + type: mteb/amazon_massive_intent + metrics: + - type: accuracy + value: 65.4371 + - type: f1 + value: 62.8737 + - type: f1_weighted + value: 64.2218 + - type: main_score + value: 65.4371 + task: + type: Classification + - dataset: + config: en + name: MTEB MassiveScenarioClassification (en) + revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8 + split: test + type: mteb/amazon_massive_scenario + metrics: + - type: accuracy + value: 70.4371 + - type: f1 + value: 69.75200000000001 + - type: f1_weighted + value: 69.7839 + - type: main_score + value: 70.4371 + task: + type: Classification + - dataset: + config: default + name: MTEB MedrxivClusteringP2P.v2 (default) + revision: e7a26af6f3ae46b30dde8737f02c07b1505bcc73 + split: test + type: mteb/medrxiv-clustering-p2p + metrics: + - type: v_measure + value: 35.1864 + - type: v_measure_std + value: 0.7835 + - type: main_score + value: 35.1864 + task: + type: Clustering + - dataset: + config: default + name: MTEB MedrxivClusteringS2S.v2 (default) + revision: 35191c8c0dca72d8ff3efcd72aa802307d469663 + split: test + type: mteb/medrxiv-clustering-s2s + metrics: + - type: v_measure + value: 31.8693 + - type: v_measure_std + value: 0.662 + - type: main_score + value: 31.8693 + task: + type: Clustering + - dataset: + config: default + name: MTEB MindSmallReranking (default) + revision: 59042f120c80e8afa9cdbb224f67076cec0fc9a7 + split: test + type: mteb/mind_small + metrics: + - type: map + value: 29.4367 + - type: mrr + value: 30.318299999999997 + - type: nAUC_map_max + value: -21.5343 + - type: nAUC_map_std + value: -6.4848 + - type: nAUC_map_diff1 + value: 12.8559 + - type: nAUC_mrr_max + value: -15.981200000000001 + - type: nAUC_mrr_std + value: -4.2437000000000005 + - type: nAUC_mrr_diff1 + value: 12.4087 + - type: main_score + value: 29.4367 + task: + type: Reranking + - dataset: + config: default + name: MTEB SCIDOCS (default) + revision: f8c2fcf00f625baaa80f62ec5bd9e1fff3b8ae88 + split: test + type: mteb/scidocs + metrics: + - type: ndcg_at_1 + value: 19.5 + - type: ndcg_at_3 + value: 15.673 + - type: ndcg_at_5 + value: 13.389000000000001 + - type: ndcg_at_10 + value: 16.179 + - type: ndcg_at_20 + value: 18.88 + - type: ndcg_at_100 + value: 23.812 + - type: ndcg_at_1000 + value: 29.833 + - type: map_at_1 + value: 3.963 + - type: map_at_3 + value: 6.93 + - type: map_at_5 + value: 8.062 + - type: map_at_10 + value: 9.328 + - type: map_at_20 + value: 10.283000000000001 + - type: map_at_100 + value: 11.197 + - type: map_at_1000 + value: 11.522 + - type: recall_at_1 + value: 3.963 + - type: recall_at_3 + value: 8.813 + - type: recall_at_5 + value: 11.658 + - type: recall_at_10 + value: 16.803 + - type: recall_at_20 + value: 23.169999999999998 + - type: recall_at_100 + value: 39.163 + - type: recall_at_1000 + value: 68.572 + - type: precision_at_1 + value: 19.5 + - type: precision_at_3 + value: 14.499999999999998 + - type: precision_at_5 + value: 11.5 + - type: precision_at_10 + value: 8.3 + - type: precision_at_20 + value: 5.71 + - type: precision_at_100 + value: 1.9300000000000002 + - type: precision_at_1000 + value: 0.338 + - type: mrr_at_1 + value: 19.5 + - type: mrr_at_3 + value: 26.016699999999997 + - type: mrr_at_5 + value: 27.526699999999998 + - type: mrr_at_10 + value: 28.9305 + - type: mrr_at_20 + value: 29.628100000000003 + - type: mrr_at_100 + value: 30.131400000000003 + - type: mrr_at_1000 + value: 30.201800000000002 + - type: nauc_ndcg_at_1_max + value: 25.1197 + - type: nauc_ndcg_at_1_std + value: 4.7176 + - type: nauc_ndcg_at_1_diff1 + value: 24.2336 + - type: nauc_ndcg_at_3_max + value: 30.050900000000002 + - type: nauc_ndcg_at_3_std + value: 11.4719 + - type: nauc_ndcg_at_3_diff1 + value: 20.4572 + - type: nauc_ndcg_at_5_max + value: 32.224399999999996 + - type: nauc_ndcg_at_5_std + value: 15.0585 + - type: nauc_ndcg_at_5_diff1 + value: 19.991600000000002 + - type: nauc_ndcg_at_10_max + value: 33.7156 + - type: nauc_ndcg_at_10_std + value: 19.2797 + - type: nauc_ndcg_at_10_diff1 + value: 20.3735 + - type: nauc_ndcg_at_20_max + value: 34.7518 + - type: nauc_ndcg_at_20_std + value: 23.227600000000002 + - type: nauc_ndcg_at_20_diff1 + value: 19.2851 + - type: nauc_ndcg_at_100_max + value: 36.6006 + - type: nauc_ndcg_at_100_std + value: 28.511599999999998 + - type: nauc_ndcg_at_100_diff1 + value: 18.0315 + - type: nauc_ndcg_at_1000_max + value: 36.3651 + - type: nauc_ndcg_at_1000_std + value: 29.7201 + - type: nauc_ndcg_at_1000_diff1 + value: 16.5988 + - type: nauc_map_at_1_max + value: 24.954 + - type: nauc_map_at_1_std + value: 4.7878 + - type: nauc_map_at_1_diff1 + value: 24.7611 + - type: nauc_map_at_3_max + value: 30.0634 + - type: nauc_map_at_3_std + value: 9.9217 + - type: nauc_map_at_3_diff1 + value: 21.9063 + - type: nauc_map_at_5_max + value: 32.1685 + - type: nauc_map_at_5_std + value: 12.8527 + - type: nauc_map_at_5_diff1 + value: 21.033099999999997 + - type: nauc_map_at_10_max + value: 33.840199999999996 + - type: nauc_map_at_10_std + value: 16.304299999999998 + - type: nauc_map_at_10_diff1 + value: 21.9142 + - type: nauc_map_at_20_max + value: 34.2084 + - type: nauc_map_at_20_std + value: 18.709799999999998 + - type: nauc_map_at_20_diff1 + value: 21.2113 + - type: nauc_map_at_100_max + value: 35.1304 + - type: nauc_map_at_100_std + value: 20.8559 + - type: nauc_map_at_100_diff1 + value: 20.8642 + - type: nauc_map_at_1000_max + value: 35.1972 + - type: nauc_map_at_1000_std + value: 21.2306 + - type: nauc_map_at_1000_diff1 + value: 20.7425 + - type: nauc_recall_at_1_max + value: 24.954 + - type: nauc_recall_at_1_std + value: 4.7878 + - type: nauc_recall_at_1_diff1 + value: 24.7611 + - type: nauc_recall_at_3_max + value: 31.1016 + - type: nauc_recall_at_3_std + value: 14.1642 + - type: nauc_recall_at_3_diff1 + value: 18.676000000000002 + - type: nauc_recall_at_5_max + value: 33.8509 + - type: nauc_recall_at_5_std + value: 19.503899999999998 + - type: nauc_recall_at_5_diff1 + value: 17.1764 + - type: nauc_recall_at_10_max + value: 34.085300000000004 + - type: nauc_recall_at_10_std + value: 25.536199999999997 + - type: nauc_recall_at_10_diff1 + value: 16.8913 + - type: nauc_recall_at_20_max + value: 34.1879 + - type: nauc_recall_at_20_std + value: 31.5486 + - type: nauc_recall_at_20_diff1 + value: 13.852300000000001 + - type: nauc_recall_at_100_max + value: 34.313700000000004 + - type: nauc_recall_at_100_std + value: 40.6137 + - type: nauc_recall_at_100_diff1 + value: 9.043800000000001 + - type: nauc_recall_at_1000_max + value: 27.090500000000002 + - type: nauc_recall_at_1000_std + value: 42.398799999999994 + - type: nauc_recall_at_1000_diff1 + value: -0.9452999999999999 + - type: nauc_precision_at_1_max + value: 25.1197 + - type: nauc_precision_at_1_std + value: 4.7176 + - type: nauc_precision_at_1_diff1 + value: 24.2336 + - type: nauc_precision_at_3_max + value: 31.4429 + - type: nauc_precision_at_3_std + value: 14.1941 + - type: nauc_precision_at_3_diff1 + value: 18.4824 + - type: nauc_precision_at_5_max + value: 34.2219 + - type: nauc_precision_at_5_std + value: 19.703699999999998 + - type: nauc_precision_at_5_diff1 + value: 17.0964 + - type: nauc_precision_at_10_max + value: 34.380300000000005 + - type: nauc_precision_at_10_std + value: 25.6554 + - type: nauc_precision_at_10_diff1 + value: 16.8487 + - type: nauc_precision_at_20_max + value: 34.462199999999996 + - type: nauc_precision_at_20_std + value: 31.465500000000002 + - type: nauc_precision_at_20_diff1 + value: 13.9038 + - type: nauc_precision_at_100_max + value: 34.7074 + - type: nauc_precision_at_100_std + value: 40.3278 + - type: nauc_precision_at_100_diff1 + value: 9.2637 + - type: nauc_precision_at_1000_max + value: 27.213900000000002 + - type: nauc_precision_at_1000_std + value: 40.8382 + - type: nauc_precision_at_1000_diff1 + value: -0.5306 + - type: nauc_mrr_at_1_max + value: 25.1197 + - type: nauc_mrr_at_1_std + value: 4.7176 + - type: nauc_mrr_at_1_diff1 + value: 24.2336 + - type: nauc_mrr_at_3_max + value: 27.9362 + - type: nauc_mrr_at_3_std + value: 9.9578 + - type: nauc_mrr_at_3_diff1 + value: 20.809 + - type: nauc_mrr_at_5_max + value: 29.0381 + - type: nauc_mrr_at_5_std + value: 11.7807 + - type: nauc_mrr_at_5_diff1 + value: 20.8787 + - type: nauc_mrr_at_10_max + value: 28.860799999999998 + - type: nauc_mrr_at_10_std + value: 12.269 + - type: nauc_mrr_at_10_diff1 + value: 20.7762 + - type: nauc_mrr_at_20_max + value: 29.2051 + - type: nauc_mrr_at_20_std + value: 12.7588 + - type: nauc_mrr_at_20_diff1 + value: 20.9176 + - type: nauc_mrr_at_100_max + value: 29.2288 + - type: nauc_mrr_at_100_std + value: 12.7523 + - type: nauc_mrr_at_100_diff1 + value: 20.9235 + - type: nauc_mrr_at_1000_max + value: 29.2119 + - type: nauc_mrr_at_1000_std + value: 12.697600000000001 + - type: nauc_mrr_at_1000_diff1 + value: 20.9131 + - type: main_score + value: 16.179 + task: + type: Retrieval + - dataset: + config: default + name: MTEB SICK-R (default) + revision: 20a6d6f312dd54037fe07a32d58e5e168867909d + split: test + type: mteb/sickr-sts + metrics: + - type: pearson + value: 84.5347 + - type: spearman + value: 79.80850000000001 + - type: cosine_pearson + value: 84.5347 + - type: cosine_spearman + value: 79.80850000000001 + - type: manhattan_pearson + value: 81.0701 + - type: manhattan_spearman + value: 79.6721 + - type: euclidean_pearson + value: 81.20349999999999 + - type: euclidean_spearman + value: 79.80850000000001 + - type: main_score + value: 79.80850000000001 + task: + type: STS + - dataset: + config: default + name: MTEB STS12 (default) + revision: a0d554a64d88156834ff5ae9920b964011b16384 + split: test + type: mteb/sts12-sts + metrics: + - type: pearson + value: 86.88 + - type: spearman + value: 78.1076 + - type: cosine_pearson + value: 86.88 + - type: cosine_spearman + value: 78.1052 + - type: manhattan_pearson + value: 83.3712 + - type: manhattan_spearman + value: 78.0898 + - type: euclidean_pearson + value: 83.3731 + - type: euclidean_spearman + value: 78.1052 + - type: main_score + value: 78.1052 + task: + type: STS + - dataset: + config: default + name: MTEB STS13 (default) + revision: 7e90230a92c190f1bf69ae9002b8cea547a64cca + split: test + type: mteb/sts13-sts + metrics: + - type: pearson + value: 83.5938 + - type: spearman + value: 84.2951 + - type: cosine_pearson + value: 83.5938 + - type: cosine_spearman + value: 84.2951 + - type: manhattan_pearson + value: 83.2541 + - type: manhattan_spearman + value: 83.8292 + - type: euclidean_pearson + value: 83.69640000000001 + - type: euclidean_spearman + value: 84.2951 + - type: main_score + value: 84.2951 + task: + type: STS + - dataset: + config: default + name: MTEB STS14 (default) + revision: 6031580fec1f6af667f0bd2da0a551cf4f0b2375 + split: test + type: mteb/sts14-sts + metrics: + - type: pearson + value: 82.6003 + - type: spearman + value: 81.3569 + - type: cosine_pearson + value: 82.6003 + - type: cosine_spearman + value: 81.357 + - type: manhattan_pearson + value: 81.5087 + - type: manhattan_spearman + value: 81.17229999999999 + - type: euclidean_pearson + value: 81.7147 + - type: euclidean_spearman + value: 81.3569 + - type: main_score + value: 81.357 + task: + type: STS + - dataset: + config: default + name: MTEB STS15 (default) + revision: ae752c7c21bf194d8b67fd573edf7ae58183cbe3 + split: test + type: mteb/sts15-sts + metrics: + - type: pearson + value: 86.4161 + - type: spearman + value: 87.0039 + - type: cosine_pearson + value: 86.4161 + - type: cosine_spearman + value: 87.0039 + - type: manhattan_pearson + value: 86.2482 + - type: manhattan_spearman + value: 86.934 + - type: euclidean_pearson + value: 86.3344 + - type: euclidean_spearman + value: 87.0039 + - type: main_score + value: 87.0039 + task: + type: STS + - dataset: + config: en-en + name: MTEB STS17 (en-en) + revision: faeb762787bd10488a50c8b5be4a3b82e411949c + split: test + type: mteb/sts17-crosslingual-sts + metrics: + - type: pearson + value: 88.6011 + - type: spearman + value: 88.1023 + - type: cosine_pearson + value: 88.6011 + - type: cosine_spearman + value: 88.1023 + - type: manhattan_pearson + value: 88.18639999999999 + - type: manhattan_spearman + value: 88.55380000000001 + - type: euclidean_pearson + value: 88.011 + - type: euclidean_spearman + value: 88.1023 + - type: main_score + value: 88.1023 + task: + type: STS + - dataset: + config: en + name: MTEB STS22.v2 (en) + revision: d31f33a128469b20e357535c39b82fb3c3f6f2bd + split: test + type: mteb/sts22-crosslingual-sts + metrics: + - type: pearson + value: 65.7746 + - type: spearman + value: 64.7997 + - type: cosine_pearson + value: 65.7746 + - type: cosine_spearman + value: 64.7997 + - type: manhattan_pearson + value: 67.5417 + - type: manhattan_spearman + value: 65.27629999999999 + - type: euclidean_pearson + value: 67.2574 + - type: euclidean_spearman + value: 64.7997 + - type: main_score + value: 64.7997 + task: + type: STS + - dataset: + config: default + name: MTEB STSBenchmark (default) + revision: b0fddb56ed78048fa8b90373c8a3cfc37b684831 + split: test + type: mteb/stsbenchmark-sts + metrics: + - type: pearson + value: 84.4276 + - type: spearman + value: 84.9631 + - type: cosine_pearson + value: 84.4276 + - type: cosine_spearman + value: 84.9631 + - type: manhattan_pearson + value: 84.4743 + - type: manhattan_spearman + value: 84.7686 + - type: euclidean_pearson + value: 84.6058 + - type: euclidean_spearman + value: 84.9631 + - type: main_score + value: 84.9631 + task: + type: STS + - dataset: + config: default + name: MTEB SprintDuplicateQuestions (default) + revision: d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46 + split: test + type: mteb/sprintduplicatequestions-pairclassification + metrics: + - type: similarity_accuracy + value: 99.7931 + - type: similarity_accuracy_threshold + value: 69.6798 + - type: similarity_f1 + value: 89.4293 + - type: similarity_f1_threshold + value: 68.3132 + - type: similarity_precision + value: 88.76849999999999 + - type: similarity_recall + value: 90.10000000000001 + - type: similarity_ap + value: 94.3099 + - type: cosine_accuracy + value: 99.7931 + - type: cosine_accuracy_threshold + value: 69.6798 + - type: cosine_f1 + value: 89.4293 + - type: cosine_f1_threshold + value: 68.3132 + - type: cosine_precision + value: 88.76849999999999 + - type: cosine_recall + value: 90.10000000000001 + - type: cosine_ap + value: 94.3099 + - type: manhattan_accuracy + value: 99.7792 + - type: manhattan_accuracy_threshold + value: 1354.3922 + - type: manhattan_f1 + value: 88.71289999999999 + - type: manhattan_f1_threshold + value: 1389.3319999999999 + - type: manhattan_precision + value: 87.84309999999999 + - type: manhattan_recall + value: 89.60000000000001 + - type: manhattan_ap + value: 93.8459 + - type: euclidean_accuracy + value: 99.7931 + - type: euclidean_accuracy_threshold + value: 77.872 + - type: euclidean_f1 + value: 89.4293 + - type: euclidean_f1_threshold + value: 79.6075 + - type: euclidean_precision + value: 88.76849999999999 + - type: euclidean_recall + value: 90.10000000000001 + - type: euclidean_ap + value: 94.3099 + - type: dot_accuracy + value: 99.7931 + - type: dot_accuracy_threshold + value: 69.6798 + - type: dot_f1 + value: 89.4293 + - type: dot_f1_threshold + value: 68.3132 + - type: dot_precision + value: 88.76849999999999 + - type: dot_recall + value: 90.10000000000001 + - type: dot_ap + value: 94.3099 + - type: max_accuracy + value: 99.7931 + - type: max_f1 + value: 89.4293 + - type: max_precision + value: 88.76849999999999 + - type: max_recall + value: 90.10000000000001 + - type: max_ap + value: 94.3099 + - type: main_score + value: 94.3099 + task: + type: PairClassification + - dataset: + config: default + name: MTEB StackExchangeClustering.v2 (default) + revision: 6cbc1f7b2bc0622f2e39d2c77fa502909748c259 + split: test + type: mteb/stackexchange-clustering + metrics: + - type: v_measure + value: 53.9397 + - type: v_measure_std + value: 0.7764 + - type: main_score + value: 53.9397 + task: + type: Clustering + - dataset: + config: default + name: MTEB StackExchangeClusteringP2P.v2 (default) + revision: 815ca46b2622cec33ccafc3735d572c266efdb44 + split: test + type: mteb/stackexchange-clustering-p2p + metrics: + - type: v_measure + value: 40.6498 + - type: v_measure_std + value: 0.439 + - type: main_score + value: 40.6498 + task: + type: Clustering + - dataset: + config: default + name: MTEB SummEvalSummarization.v2 (default) + revision: cda12ad7615edc362dbf25a00fdd61d3b1eaf93c + split: test + type: mteb/summeval + metrics: + - type: pearson + value: 28.6283 + - type: spearman + value: 26.0828 + - type: cosine_spearman + value: 26.0828 + - type: cosine_pearson + value: 28.6283 + - type: dot_spearman + value: 26.0828 + - type: dot_pearson + value: 28.6283 + - type: main_score + value: 26.0828 + task: + type: Summarization + - dataset: + config: default + name: MTEB TRECCOVID (default) + revision: bb9466bac8153a0349341eb1b22e06409e78ef4e + split: test + type: mteb/trec-covid + metrics: + - type: ndcg_at_1 + value: 66.0 + - type: ndcg_at_3 + value: 64.592 + - type: ndcg_at_5 + value: 63.405 + - type: ndcg_at_10 + value: 60.077999999999996 + - type: ndcg_at_20 + value: 57.202 + - type: ndcg_at_100 + value: 44.643 + - type: ndcg_at_1000 + value: 42.104 + - type: map_at_1 + value: 0.193 + - type: map_at_3 + value: 0.514 + - type: map_at_5 + value: 0.783 + - type: map_at_10 + value: 1.3719999999999999 + - type: map_at_20 + value: 2.371 + - type: map_at_100 + value: 7.353 + - type: map_at_1000 + value: 17.855999999999998 + - type: recall_at_1 + value: 0.193 + - type: recall_at_3 + value: 0.563 + - type: recall_at_5 + value: 0.907 + - type: recall_at_10 + value: 1.683 + - type: recall_at_20 + value: 3.118 + - type: recall_at_100 + value: 11.051 + - type: recall_at_1000 + value: 39.973 + - type: precision_at_1 + value: 74.0 + - type: precision_at_3 + value: 71.333 + - type: precision_at_5 + value: 68.8 + - type: precision_at_10 + value: 63.800000000000004 + - type: precision_at_20 + value: 60.5 + - type: precision_at_100 + value: 45.519999999999996 + - type: precision_at_1000 + value: 18.451999999999998 + - type: mrr_at_1 + value: 74.0 + - type: mrr_at_3 + value: 83.3333 + - type: mrr_at_5 + value: 83.7333 + - type: mrr_at_10 + value: 84.3524 + - type: mrr_at_20 + value: 84.3524 + - type: mrr_at_100 + value: 84.3524 + - type: mrr_at_1000 + value: 84.3524 + - type: nauc_ndcg_at_1_max + value: 11.527800000000001 + - type: nauc_ndcg_at_1_std + value: 17.1352 + - type: nauc_ndcg_at_1_diff1 + value: 24.955199999999998 + - type: nauc_ndcg_at_3_max + value: 11.7829 + - type: nauc_ndcg_at_3_std + value: 23.1421 + - type: nauc_ndcg_at_3_diff1 + value: 20.884 + - type: nauc_ndcg_at_5_max + value: 8.8058 + - type: nauc_ndcg_at_5_std + value: 27.9156 + - type: nauc_ndcg_at_5_diff1 + value: 7.002 + - type: nauc_ndcg_at_10_max + value: 16.561 + - type: nauc_ndcg_at_10_std + value: 40.528999999999996 + - type: nauc_ndcg_at_10_diff1 + value: -6.1467 + - type: nauc_ndcg_at_20_max + value: 25.0792 + - type: nauc_ndcg_at_20_std + value: 54.0689 + - type: nauc_ndcg_at_20_diff1 + value: -9.6224 + - type: nauc_ndcg_at_100_max + value: 43.2818 + - type: nauc_ndcg_at_100_std + value: 75.4432 + - type: nauc_ndcg_at_100_diff1 + value: -11.4618 + - type: nauc_ndcg_at_1000_max + value: 50.360099999999996 + - type: nauc_ndcg_at_1000_std + value: 76.03999999999999 + - type: nauc_ndcg_at_1000_diff1 + value: -12.5796 + - type: nauc_map_at_1_max + value: 4.3809000000000005 + - type: nauc_map_at_1_std + value: -17.5338 + - type: nauc_map_at_1_diff1 + value: 24.837 + - type: nauc_map_at_3_max + value: 4.7842 + - type: nauc_map_at_3_std + value: -8.9273 + - type: nauc_map_at_3_diff1 + value: 19.7729 + - type: nauc_map_at_5_max + value: 3.6865 + - type: nauc_map_at_5_std + value: -1.1584 + - type: nauc_map_at_5_diff1 + value: 7.3548 + - type: nauc_map_at_10_max + value: 7.556400000000001 + - type: nauc_map_at_10_std + value: 11.2599 + - type: nauc_map_at_10_diff1 + value: -3.4863999999999997 + - type: nauc_map_at_20_max + value: 12.6951 + - type: nauc_map_at_20_std + value: 27.3531 + - type: nauc_map_at_20_diff1 + value: -11.968 + - type: nauc_map_at_100_max + value: 41.625099999999996 + - type: nauc_map_at_100_std + value: 66.5204 + - type: nauc_map_at_100_diff1 + value: -12.020999999999999 + - type: nauc_map_at_1000_max + value: 56.6014 + - type: nauc_map_at_1000_std + value: 80.6523 + - type: nauc_map_at_1000_diff1 + value: -11.9876 + - type: nauc_recall_at_1_max + value: 4.3809000000000005 + - type: nauc_recall_at_1_std + value: -17.5338 + - type: nauc_recall_at_1_diff1 + value: 24.837 + - type: nauc_recall_at_3_max + value: -0.8904000000000001 + - type: nauc_recall_at_3_std + value: -11.2455 + - type: nauc_recall_at_3_diff1 + value: 17.6352 + - type: nauc_recall_at_5_max + value: -4.6216 + - type: nauc_recall_at_5_std + value: -3.5367999999999995 + - type: nauc_recall_at_5_diff1 + value: 3.3192 + - type: nauc_recall_at_10_max + value: 1.8993 + - type: nauc_recall_at_10_std + value: 6.844600000000001 + - type: nauc_recall_at_10_diff1 + value: -6.0693 + - type: nauc_recall_at_20_max + value: 5.733 + - type: nauc_recall_at_20_std + value: 20.6114 + - type: nauc_recall_at_20_diff1 + value: -11.631 + - type: nauc_recall_at_100_max + value: 32.7146 + - type: nauc_recall_at_100_std + value: 55.6053 + - type: nauc_recall_at_100_diff1 + value: -10.7219 + - type: nauc_recall_at_1000_max + value: 50.7544 + - type: nauc_recall_at_1000_std + value: 68.4639 + - type: nauc_recall_at_1000_diff1 + value: -10.431600000000001 + - type: nauc_precision_at_1_max + value: 13.8681 + - type: nauc_precision_at_1_std + value: -3.4711 + - type: nauc_precision_at_1_diff1 + value: 36.945 + - type: nauc_precision_at_3_max + value: 11.6309 + - type: nauc_precision_at_3_std + value: 5.0299000000000005 + - type: nauc_precision_at_3_diff1 + value: 28.5186 + - type: nauc_precision_at_5_max + value: 10.1297 + - type: nauc_precision_at_5_std + value: 19.049599999999998 + - type: nauc_precision_at_5_diff1 + value: 7.918500000000001 + - type: nauc_precision_at_10_max + value: 21.3492 + - type: nauc_precision_at_10_std + value: 39.6679 + - type: nauc_precision_at_10_diff1 + value: -10.7691 + - type: nauc_precision_at_20_max + value: 32.4627 + - type: nauc_precision_at_20_std + value: 57.2564 + - type: nauc_precision_at_20_diff1 + value: -12.0336 + - type: nauc_precision_at_100_max + value: 47.7277 + - type: nauc_precision_at_100_std + value: 77.0329 + - type: nauc_precision_at_100_diff1 + value: -9.2173 + - type: nauc_precision_at_1000_max + value: 47.6622 + - type: nauc_precision_at_1000_std + value: 62.8329 + - type: nauc_precision_at_1000_diff1 + value: -5.9713 + - type: nauc_mrr_at_1_max + value: 13.8681 + - type: nauc_mrr_at_1_std + value: -3.4711 + - type: nauc_mrr_at_1_diff1 + value: 36.945 + - type: nauc_mrr_at_3_max + value: 9.6673 + - type: nauc_mrr_at_3_std + value: -4.3877 + - type: nauc_mrr_at_3_diff1 + value: 39.2075 + - type: nauc_mrr_at_5_max + value: 7.9742999999999995 + - type: nauc_mrr_at_5_std + value: -4.8388 + - type: nauc_mrr_at_5_diff1 + value: 38.314 + - type: nauc_mrr_at_10_max + value: 11.6962 + - type: nauc_mrr_at_10_std + value: -2.7085000000000004 + - type: nauc_mrr_at_10_diff1 + value: 37.695 + - type: nauc_mrr_at_20_max + value: 11.6962 + - type: nauc_mrr_at_20_std + value: -2.7085000000000004 + - type: nauc_mrr_at_20_diff1 + value: 37.695 + - type: nauc_mrr_at_100_max + value: 11.6962 + - type: nauc_mrr_at_100_std + value: -2.7085000000000004 + - type: nauc_mrr_at_100_diff1 + value: 37.695 + - type: nauc_mrr_at_1000_max + value: 11.6962 + - type: nauc_mrr_at_1000_std + value: -2.7085000000000004 + - type: nauc_mrr_at_1000_diff1 + value: 37.695 + - type: main_score + value: 60.077999999999996 + task: + type: Retrieval + - dataset: + config: default + name: MTEB Touche2020Retrieval.v3 (default) + revision: 431886eaecc48f067a3975b70d0949ea2862463c + split: test + type: mteb/webis-touche2020-v3 + metrics: + - type: ndcg_at_1 + value: 58.163 + - type: ndcg_at_3 + value: 58.884 + - type: ndcg_at_5 + value: 53.062 + - type: ndcg_at_10 + value: 47.571999999999996 + - type: ndcg_at_20 + value: 43.984 + - type: ndcg_at_100 + value: 51.559999999999995 + - type: ndcg_at_1000 + value: 64.25800000000001 + - type: map_at_1 + value: 2.759 + - type: map_at_3 + value: 7.310999999999999 + - type: map_at_5 + value: 10.077 + - type: map_at_10 + value: 15.722 + - type: map_at_20 + value: 21.917 + - type: map_at_100 + value: 29.582000000000004 + - type: map_at_1000 + value: 32.608 + - type: recall_at_1 + value: 2.759 + - type: recall_at_3 + value: 7.870000000000001 + - type: recall_at_5 + value: 11.26 + - type: recall_at_10 + value: 19.211 + - type: recall_at_20 + value: 30.134 + - type: recall_at_100 + value: 54.96 + - type: recall_at_1000 + value: 85.78099999999999 + - type: precision_at_1 + value: 67.34700000000001 + - type: precision_at_3 + value: 68.027 + - type: precision_at_5 + value: 59.184000000000005 + - type: precision_at_10 + value: 50.815999999999995 + - type: precision_at_20 + value: 41.939 + - type: precision_at_100 + value: 17.041 + - type: precision_at_1000 + value: 2.963 + - type: mrr_at_1 + value: 67.3469 + - type: mrr_at_3 + value: 80.6122 + - type: mrr_at_5 + value: 80.6122 + - type: mrr_at_10 + value: 80.9524 + - type: mrr_at_20 + value: 80.9524 + - type: mrr_at_100 + value: 80.9524 + - type: mrr_at_1000 + value: 80.9524 + - type: nauc_ndcg_at_1_max + value: -18.7982 + - type: nauc_ndcg_at_1_std + value: 13.605500000000001 + - type: nauc_ndcg_at_1_diff1 + value: 21.2588 + - type: nauc_ndcg_at_3_max + value: -9.0937 + - type: nauc_ndcg_at_3_std + value: 23.259900000000002 + - type: nauc_ndcg_at_3_diff1 + value: 24.2989 + - type: nauc_ndcg_at_5_max + value: -13.242300000000002 + - type: nauc_ndcg_at_5_std + value: 9.7464 + - type: nauc_ndcg_at_5_diff1 + value: 18.601799999999997 + - type: nauc_ndcg_at_10_max + value: -12.045599999999999 + - type: nauc_ndcg_at_10_std + value: 7.5604000000000005 + - type: nauc_ndcg_at_10_diff1 + value: 20.1203 + - type: nauc_ndcg_at_20_max + value: -13.2776 + - type: nauc_ndcg_at_20_std + value: 8.2692 + - type: nauc_ndcg_at_20_diff1 + value: 21.38 + - type: nauc_ndcg_at_100_max + value: -21.1315 + - type: nauc_ndcg_at_100_std + value: 8.4079 + - type: nauc_ndcg_at_100_diff1 + value: 29.3124 + - type: nauc_ndcg_at_1000_max + value: -3.7026999999999997 + - type: nauc_ndcg_at_1000_std + value: 34.970600000000005 + - type: nauc_ndcg_at_1000_diff1 + value: 22.3636 + - type: nauc_map_at_1_max + value: -36.432500000000005 + - type: nauc_map_at_1_std + value: -23.9669 + - type: nauc_map_at_1_diff1 + value: 37.2073 + - type: nauc_map_at_3_max + value: -32.8613 + - type: nauc_map_at_3_std + value: -18.0951 + - type: nauc_map_at_3_diff1 + value: 36.3228 + - type: nauc_map_at_5_max + value: -31.355 + - type: nauc_map_at_5_std + value: -21.148500000000002 + - type: nauc_map_at_5_diff1 + value: 27.999200000000002 + - type: nauc_map_at_10_max + value: -25.3787 + - type: nauc_map_at_10_std + value: -18.564700000000002 + - type: nauc_map_at_10_diff1 + value: 24.076800000000002 + - type: nauc_map_at_20_max + value: -20.954 + - type: nauc_map_at_20_std + value: -12.6847 + - type: nauc_map_at_20_diff1 + value: 24.3842 + - type: nauc_map_at_100_max + value: -15.7801 + - type: nauc_map_at_100_std + value: -2.823 + - type: nauc_map_at_100_diff1 + value: 24.8472 + - type: nauc_map_at_1000_max + value: -11.8023 + - type: nauc_map_at_1000_std + value: 3.9041 + - type: nauc_map_at_1000_diff1 + value: 23.3312 + - type: nauc_recall_at_1_max + value: -36.432500000000005 + - type: nauc_recall_at_1_std + value: -23.9669 + - type: nauc_recall_at_1_diff1 + value: 37.2073 + - type: nauc_recall_at_3_max + value: -36.3448 + - type: nauc_recall_at_3_std + value: -18.4742 + - type: nauc_recall_at_3_diff1 + value: 38.4857 + - type: nauc_recall_at_5_max + value: -35.4207 + - type: nauc_recall_at_5_std + value: -23.7906 + - type: nauc_recall_at_5_diff1 + value: 28.3854 + - type: nauc_recall_at_10_max + value: -28.4266 + - type: nauc_recall_at_10_std + value: -21.3224 + - type: nauc_recall_at_10_diff1 + value: 27.0746 + - type: nauc_recall_at_20_max + value: -23.1205 + - type: nauc_recall_at_20_std + value: -12.3539 + - type: nauc_recall_at_20_diff1 + value: 27.127499999999998 + - type: nauc_recall_at_100_max + value: -22.0703 + - type: nauc_recall_at_100_std + value: 10.1339 + - type: nauc_recall_at_100_diff1 + value: 29.759900000000002 + - type: nauc_recall_at_1000_max + value: 13.5147 + - type: nauc_recall_at_1000_std + value: 78.4907 + - type: nauc_recall_at_1000_diff1 + value: 12.151 + - type: nauc_precision_at_1_max + value: -20.1082 + - type: nauc_precision_at_1_std + value: 13.5123 + - type: nauc_precision_at_1_diff1 + value: 16.7562 + - type: nauc_precision_at_3_max + value: -11.2979 + - type: nauc_precision_at_3_std + value: 23.0876 + - type: nauc_precision_at_3_diff1 + value: 20.738 + - type: nauc_precision_at_5_max + value: -18.1198 + - type: nauc_precision_at_5_std + value: -2.4168 + - type: nauc_precision_at_5_diff1 + value: 5.1223 + - type: nauc_precision_at_10_max + value: -4.7656 + - type: nauc_precision_at_10_std + value: 1.5377 + - type: nauc_precision_at_10_diff1 + value: 8.2175 + - type: nauc_precision_at_20_max + value: 7.571999999999999 + - type: nauc_precision_at_20_std + value: 17.309 + - type: nauc_precision_at_20_diff1 + value: 5.2156 + - type: nauc_precision_at_100_max + value: 35.02 + - type: nauc_precision_at_100_std + value: 57.2867 + - type: nauc_precision_at_100_diff1 + value: -12.814200000000001 + - type: nauc_precision_at_1000_max + value: 54.8988 + - type: nauc_precision_at_1000_std + value: 55.970699999999994 + - type: nauc_precision_at_1000_diff1 + value: -36.8074 + - type: nauc_mrr_at_1_max + value: -20.1082 + - type: nauc_mrr_at_1_std + value: 13.5123 + - type: nauc_mrr_at_1_diff1 + value: 16.7562 + - type: nauc_mrr_at_3_max + value: -23.668300000000002 + - type: nauc_mrr_at_3_std + value: 16.883699999999997 + - type: nauc_mrr_at_3_diff1 + value: 20.6687 + - type: nauc_mrr_at_5_max + value: -23.668300000000002 + - type: nauc_mrr_at_5_std + value: 16.883699999999997 + - type: nauc_mrr_at_5_diff1 + value: 20.6687 + - type: nauc_mrr_at_10_max + value: -21.8234 + - type: nauc_mrr_at_10_std + value: 15.1609 + - type: nauc_mrr_at_10_diff1 + value: 19.6023 + - type: nauc_mrr_at_20_max + value: -21.8234 + - type: nauc_mrr_at_20_std + value: 15.1609 + - type: nauc_mrr_at_20_diff1 + value: 19.6023 + - type: nauc_mrr_at_100_max + value: -21.8234 + - type: nauc_mrr_at_100_std + value: 15.1609 + - type: nauc_mrr_at_100_diff1 + value: 19.6023 + - type: nauc_mrr_at_1000_max + value: -21.8234 + - type: nauc_mrr_at_1000_std + value: 15.1609 + - type: nauc_mrr_at_1000_diff1 + value: 19.6023 + - type: main_score + value: 47.571999999999996 + task: + type: Retrieval + - dataset: + config: default + name: MTEB ToxicConversationsClassification (default) + revision: edfaf9da55d3dd50d43143d90c1ac476895ae6de + split: test + type: mteb/toxic_conversations_50k + metrics: + - type: accuracy + value: 63.608399999999996 + - type: f1 + value: 48.6248 + - type: f1_weighted + value: 71.6158 + - type: ap + value: 10.9541 + - type: ap_weighted + value: 10.9541 + - type: main_score + value: 63.608399999999996 + task: + type: Classification + - dataset: + config: default + name: MTEB TweetSentimentExtractionClassification (default) + revision: d604517c81ca91fe16a244d1248fc021f9ecee7a + split: test + type: mteb/tweet_sentiment_extraction + metrics: + - type: accuracy + value: 60.506499999999996 + - type: f1 + value: 60.711499999999994 + - type: f1_weighted + value: 59.695699999999995 + - type: main_score + value: 60.506499999999996 + task: + type: Classification + - dataset: + config: default + name: MTEB TwentyNewsgroupsClustering.v2 (default) + revision: 6125ec4e24fa026cec8a478383ee943acfbd5449 + split: test + type: mteb/twentynewsgroups-clustering + metrics: + - type: v_measure + value: 33.5462 + - type: v_measure_std + value: 1.3361 + - type: main_score + value: 33.5462 + task: + type: Clustering + - dataset: + config: default + name: MTEB TwitterSemEval2015 (default) + revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1 + split: test + type: mteb/twittersemeval2015-pairclassification + metrics: + - type: similarity_accuracy + value: 82.51180000000001 + - type: similarity_accuracy_threshold + value: 69.4516 + - type: similarity_f1 + value: 58.483399999999996 + - type: similarity_f1_threshold + value: 61.3852 + - type: similarity_precision + value: 56.29880000000001 + - type: similarity_recall + value: 60.8443 + - type: similarity_ap + value: 61.8784 + - type: cosine_accuracy + value: 82.51180000000001 + - type: cosine_accuracy_threshold + value: 69.4516 + - type: cosine_f1 + value: 58.483399999999996 + - type: cosine_f1_threshold + value: 61.3852 + - type: cosine_precision + value: 56.29880000000001 + - type: cosine_recall + value: 60.8443 + - type: cosine_ap + value: 61.8784 + - type: manhattan_accuracy + value: 82.60119999999999 + - type: manhattan_accuracy_threshold + value: 1395.2354 + - type: manhattan_f1 + value: 59.3387 + - type: manhattan_f1_threshold + value: 1544.4108 + - type: manhattan_precision + value: 56.284 + - type: manhattan_recall + value: 62.7441 + - type: manhattan_ap + value: 62.407999999999994 + - type: euclidean_accuracy + value: 82.51180000000001 + - type: euclidean_accuracy_threshold + value: 78.1645 + - type: euclidean_f1 + value: 58.483399999999996 + - type: euclidean_f1_threshold + value: 87.88040000000001 + - type: euclidean_precision + value: 56.29880000000001 + - type: euclidean_recall + value: 60.8443 + - type: euclidean_ap + value: 61.8784 + - type: dot_accuracy + value: 82.51180000000001 + - type: dot_accuracy_threshold + value: 69.4516 + - type: dot_f1 + value: 58.483399999999996 + - type: dot_f1_threshold + value: 61.3852 + - type: dot_precision + value: 56.29880000000001 + - type: dot_recall + value: 60.8443 + - type: dot_ap + value: 61.8784 + - type: max_accuracy + value: 82.60119999999999 + - type: max_f1 + value: 59.3387 + - type: max_precision + value: 56.29880000000001 + - type: max_recall + value: 62.7441 + - type: max_ap + value: 62.407999999999994 + - type: main_score + value: 62.407999999999994 + task: + type: PairClassification + - dataset: + config: default + name: MTEB TwitterURLCorpus (default) + revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf + split: test + type: mteb/twitterurlcorpus-pairclassification + metrics: + - type: similarity_accuracy + value: 87.84880000000001 + - type: similarity_accuracy_threshold + value: 62.77890000000001 + - type: similarity_f1 + value: 75.968 + - type: similarity_f1_threshold + value: 57.5925 + - type: similarity_precision + value: 71.909 + - type: similarity_recall + value: 80.5128 + - type: similarity_ap + value: 83.6557 + - type: cosine_accuracy + value: 87.84880000000001 + - type: cosine_accuracy_threshold + value: 62.77890000000001 + - type: cosine_f1 + value: 75.968 + - type: cosine_f1_threshold + value: 57.5925 + - type: cosine_precision + value: 71.909 + - type: cosine_recall + value: 80.5128 + - type: cosine_ap + value: 83.6557 + - type: manhattan_accuracy + value: 87.69940000000001 + - type: manhattan_accuracy_threshold + value: 1524.1733 + - type: manhattan_f1 + value: 76.01830000000001 + - type: manhattan_f1_threshold + value: 1597.1845 + - type: manhattan_precision + value: 72.981 + - type: manhattan_recall + value: 79.3194 + - type: manhattan_ap + value: 83.63629999999999 + - type: euclidean_accuracy + value: 87.84880000000001 + - type: euclidean_accuracy_threshold + value: 86.2799 + - type: euclidean_f1 + value: 75.968 + - type: euclidean_f1_threshold + value: 92.0951 + - type: euclidean_precision + value: 71.909 + - type: euclidean_recall + value: 80.5128 + - type: euclidean_ap + value: 83.6557 + - type: dot_accuracy + value: 87.84880000000001 + - type: dot_accuracy_threshold + value: 62.77890000000001 + - type: dot_f1 + value: 75.968 + - type: dot_f1_threshold + value: 57.5925 + - type: dot_precision + value: 71.909 + - type: dot_recall + value: 80.5128 + - type: dot_ap + value: 83.6557 + - type: max_accuracy + value: 87.84880000000001 + - type: max_f1 + value: 76.01830000000001 + - type: max_precision + value: 72.981 + - type: max_recall + value: 80.5128 + - type: max_ap + value: 83.6557 + - type: main_score + value: 83.6557 + task: + type: PairClassification +--- + +# RetrievaEmbedding-01: AMBER + +The **AMBER (Adaptive Multitask Bilingual Embedding Representations)** is a text embedding model trained by Retrieva, Inc. +This model is primarily designed for Japanese, but it also supports English. +We trained this model on various datasets related to Japanese and English. + +This model size is 132M parameters (base size). + +## Model Details + +### Model Description + +The AMBER model is a text embedding model based on the [sbintuitions/modernbert-ja-130m](https://huggingface.co/sbintuitions/modernbert-ja-130m) architecture, designed for Japanese text. +This model was trained on a variety of datasets related to Japanese, and also includes English datasets. +The model can be used for English text as well. +During training, prompts (instructions) in natural language were included, allowing the model to generate embeddings tailored to specific tasks. + +- **Developed by:** Retrieva, Inc. +- **Model type:** Based on the [ModernBERT](https://arxiv.org/abs/2412.13663) Architecture. +- **Language(s) (NLP):** Primarily Japanese (optional support for English). +- **License:** Apache 2.0 +- **Finetuned from model:** `sbintuitions/modernbert-ja-130m` +- **Model Type:** Sentence Transformer +- **Maximum Sequence Length:** 512 tokens +- **Output Dimensionality:** 512 dimensions +- **Similarity Function:** Cosine Similarity + +## Uses + +## How to Get Started with the Model + +### Install Library + +First install the python library using pip: + +```bash +pip install sentence-transformers sentencepiece +``` + +### Run Inference + +Then you can load this model and run inference. + +You can specify the prompt at inference time by adding an argument called `prompt` to `model.encode`. +The prompts used in the Japanese benchmark are described in `jmteb/tasks`, and the prompts used in the English benchmark are described in `mteb/models/retrieva_en.py`. + +```python +from sentence_transformers import SentenceTransformer + +# Download from the 🤗 Hub +model = SentenceTransformer("retrieva-jp/amber-base") +# Run inference +queries = [ + "自然言語処理とはなんですか？", + "株式会社レトリバについて教えて", +] +documents = [ + "自然言語処理（しぜんげんごしょり、英語: Natural language processing、略称：NLP）は、人間が日常的に使っている自然言語をコンピュータに処理させる一連の技術であり、人工知能と言語学の一分野である。", + "株式会社レトリバは、自然言語処理と機械学習を核としたAI技術で組織の課題解決を支援するテクノロジー企業である。", +] + +queries_embeddings = model.encode(queries, prompt_name="Retrieval-query") +documents_embeddings = model.encode(documents, prompt_name="Retrieval-passage") + +similarities = model.similarity(queries_embeddings, documents_embeddings) +print(similarities.shape) +``` + +## Training Details + +### Training Data + +We used multiple datasets to train this model. +We selected datasets from [llm-jp-eval](https://github.com/llm-jp/llm-jp-eval), [llm-japanese-dataset](https://github.com/masanorihirano/llm-japanese-dataset), and [hpprc/emb](https://huggingface.co/datasets/hpprc/emb) for Japanese datasets. +For English datasets, we mainly used some of the datasets utilized in [Asai et al. (2023)](https://arxiv.org/abs/2211.09260). +Additionally, we partially used the English datasets at [the sentence-transformers repository](https://huggingface.co/sentence-transformers) and [kilt-tasks](https://huggingface.co/datasets/facebook/kilt_tasks). +To consider cross-lingual between Japanese and English, we also used translation datasets between Japanese and English. + +For Japanese, we used synthetic data created by LLM to prepare a sufficient amount of training data. + +## Evaluation + +We evaluated the model on the following benchmarks: +- Japanese Benchmark: [JMTEB](https://github.com/sbintuitions/JMTEB) +- Japanese Retrieval Tasks: [JQaRA](https://github.com/hotchpotch/JQaRA/), [JaCWIR](https://github.com/hotchpotch/JaCWIR/), [MLDR Japanese Subset](https://huggingface.co/datasets/Shitao/MLDR) +- English Benchmark: [MTEB(eng, v2)](https://github.com/embeddings-benchmark/mteb). + +The scores in the table are all calculated by us unless otherwise noted. + +### Japanese Benchmark: JMTEB + +Note that the `Mean (TaskType)` in the following leaderboard is the same as the `Avg.` in the original JMTEB leaderboard. + +The files used for evaluation are stored in the `jmteb` directory. + +| Model | # Parameters | Mean (TaskType) | Mean (Task) | Retrieval | STS | Classification | Reranking | Clustering | PairClassification | +| :--- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| base models | < 300M | | | | | | | | | +| [cl-nagoya/ruri-base](https://huggingface.co/cl-nagoya/ruri-base) | 111M | 72.60 | 71.56 | 69.53 | 82.87 | 75.49 | 92.91 | 52.40 | 62.38 | +| AMBER-base
(this model) | 130M | 72.12 | 72.12 | **73.40** | 77.81 | **76.14** | **93.27** | 48.05 | **64.03** | +| [pkshatech/GLuCoSE-base-ja-v2](https://huggingface.co/pkshatech/GLuCoSE-base-ja-v2) | 133M | **72.89** | **72.47** | 73.03 | **82.96** | 74.02 | 93.01 | 51.96 | 62.37 | +| [pkshatech/RoSEtta-base-ja](https://huggingface.co/pkshatech/RoSEtta-base-ja) | 190M | 72.49 | 72.05 | 73.14 | 81.39 | 72.37 | 92.69 | **53.60** | 61.74 | +| [intfloat/multilingual-e5-base](https://huggingface.co/intfloat/multilingual-e5-base) | 278M | 71.11 | 69.72 | 69.45 | 80.45 | 69.86 | 92.90 | 51.62 | 62.35 | +| large models | 300M < | | | | | | | | | +| [AMBER-large](https://huggingface.co/retrieva-jp/amber-large) | 315M | 72.52 | **73.22** | **75.40** | 79.32 | 77.14 | **93.54** | 48.73 | 60.97 | +| [cl-nagoya/ruri-large](https://huggingface.co/cl-nagoya/ruri-large) | 337M | **73.20** | 73.06 | 72.86 | **83.14** | **77.15** | 93.00 | 50.78 | 62.29 | +| [intfloat/multilingual-e5-large](https://huggingface.co/intfloat/multilingual-e5-large) | 560M | 72.06 | 71.29 | 71.71 | 80.87 | 72.45 | 93.29 | **51.59** | **62.42** | + + +### Japanese Retrieval Tasks: JQaRA, JaCWIR, MLDR Japanese Subset + +The files used for MLDR are stored in the `mldr` directory. + +The prompts used in JQaRA and JaCWIR are `Retrieval-query` and `Retrieval-passage` described in `config_sentence_transformers.json`. + +| Model | # Parameters | JQaRA (nDCG@10) | JaCWIR (MAP@10) | MLDR Japanese Subset (nDCG@10) | +| :--- | --- | ---: | ---: | ---: | +| base models | < 300M | | | | +| [cl-nagoya/ruri-base](https://huggingface.co/cl-nagoya/ruri-base) | 111M | 58.4 | 83.3 | TBA | +| AMBER-base
(this model) | 130M | 57.1 | 81.6 | TBA | +| [pkshatech/GLuCoSE-base-ja-v2](https://huggingface.co/pkshatech/GLuCoSE-base-ja-v2) | 133M | **60.6** | **85.3** | 33.99 | +| [intfloat/multilingual-e5-base](https://huggingface.co/intfloat/multilingual-e5-base) | 278M | 47.1 | **85.3** | TBA | +| large models | 300M < | | | | +| [AMBER-large](https://huggingface.co/retrieva-jp/amber-large) | 315M | 62.5 | 82.4 | 34.57 | +| [cl-nagoya/ruri-large](https://huggingface.co/cl-nagoya/ruri-large) | 337M | **62.8** | 82.5 | **34.78** | +| [intfloat/multilingual-e5-large](https://huggingface.co/intfloat/multilingual-e5-large) | 560M | 55.4| **87.3** | 29.95 | + +### English Benchmark: MTEB(eng, v2) + +The files used for evaluation are stored in the `mteb` directory. + +| Model | # Parameters | Mean (TaskType) | Mean (Task) | Retrieval | STS | Classification | Reranking | Clustering | PairClassification | Summarization | +| :--- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| base models | < 300M | | | | | | | | | | +| AMBER-base
(this model) | 130M | 54.75 | 58.20 | 40.11 | **81.29** | 70.39 | 42.98 | **42.27** | 80.12 | 26.08 | +| [intfloat/multilingual-e5-base](https://huggingface.co/intfloat/multilingual-e5-base) | 278M | **56.21** | **59.75** | **43.22** | 80.50 | **73.84** | **43.87** | 42.19 | **83.74** | **26.10** | +| large models | 300M < | | | | | | | | | | +| [AMBER-large](https://huggingface.co/retrieva-jp/amber-large) | 315M | 56.08 | 59.13 | 41.04 | **81.52** | 72.23 | 43.83 | **42.71** | 81.00 | **30.21** | +| [intfloat/multilingual-e5-large](https://huggingface.co/intfloat/multilingual-e5-large) | 560M | **57.06** | **60.84** | **46.17** | 81.11 | **74.88** | **44.31** | 41.91 | **84.33** | 26.67 | + +## More Information + +TBA + + +## Model Card Authors + +Satoru Katsumata, Daisuke Kimura, Jiro Nishitoba + +## Model Card Contact + +pr[at]retrieva.jp