--- language: - en license: apache-2.0 tags: - sentence-transformers - sparse-encoder - sparse - splade - generated_from_trainer - dataset_size:90000 - loss:SpladeLoss - loss:SparseMultipleNegativesRankingLoss - loss:FlopsLoss base_model: distilbert/distilbert-base-uncased widget: - text: what is chess - text: what is a hickman for? - text: 'Steps. 1 1. Gather your materials. Here''s what you need to build two regulations-size horseshoe pits that will face each other (if you only want to build one pit, halve the materials): Two 6-foot-long treated wood 2x6s (38mm x 140mm), cut in half. 2 2. Decide where you''re going to put your pit(s).' - text: who played at california jam - text: "To the Citizens of St. Bernard We chose as our motto a simple but profound\ \ declaration: â\x80\x9CWelcome to your office.â\x80\x9D Those words remind us\ \ that we are no more than the caretakers of the office of Clerk of Court for\ \ the Parish of St. Bernard." datasets: - sentence-transformers/msmarco pipeline_tag: feature-extraction library_name: sentence-transformers metrics: - dot_accuracy@1 - dot_accuracy@3 - dot_accuracy@5 - dot_accuracy@10 - dot_precision@1 - dot_precision@3 - dot_precision@5 - dot_precision@10 - dot_recall@1 - dot_recall@3 - dot_recall@5 - dot_recall@10 - dot_ndcg@10 - dot_mrr@10 - dot_map@100 - query_active_dims - query_sparsity_ratio - corpus_active_dims - corpus_sparsity_ratio co2_eq_emissions: emissions: 20.864216098626564 energy_consumed: 0.05652200756224921 source: codecarbon training_type: fine-tuning on_cloud: false cpu_model: AMD EPYC 7R13 Processor ram_total_size: 248.0 hours_used: 0.179 hardware_used: 1 x NVIDIA H100 80GB HBM3 model-index: - name: splade-distilbert-base-uncased trained on MS MARCO triplets results: - task: type: sparse-information-retrieval name: Sparse Information Retrieval dataset: name: NanoMSMARCO type: NanoMSMARCO metrics: - type: dot_accuracy@1 value: 0.44 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.66 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.72 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.82 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.44 name: Dot Precision@1 - type: dot_precision@3 value: 0.22 name: Dot Precision@3 - type: dot_precision@5 value: 0.14400000000000002 name: Dot Precision@5 - type: dot_precision@10 value: 0.08199999999999999 name: Dot Precision@10 - type: dot_recall@1 value: 0.44 name: Dot Recall@1 - type: dot_recall@3 value: 0.66 name: Dot Recall@3 - type: dot_recall@5 value: 0.72 name: Dot Recall@5 - type: dot_recall@10 value: 0.82 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.6223979987260191 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.5599444444444444 name: Dot Mrr@10 - type: dot_map@100 value: 0.5701364200315813 name: Dot Map@100 - type: query_active_dims value: 25.260000228881836 name: Query Active Dims - type: query_sparsity_ratio value: 0.9991724002283965 name: Query Sparsity Ratio - type: corpus_active_dims value: 89.06385040283203 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9970819785596348 name: Corpus Sparsity Ratio - type: dot_accuracy@1 value: 0.44 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.6 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.74 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.84 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.44 name: Dot Precision@1 - type: dot_precision@3 value: 0.2 name: Dot Precision@3 - type: dot_precision@5 value: 0.14800000000000002 name: Dot Precision@5 - type: dot_precision@10 value: 0.08399999999999999 name: Dot Precision@10 - type: dot_recall@1 value: 0.44 name: Dot Recall@1 - type: dot_recall@3 value: 0.6 name: Dot Recall@3 - type: dot_recall@5 value: 0.74 name: Dot Recall@5 - type: dot_recall@10 value: 0.84 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.6241753240638171 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.5571349206349206 name: Dot Mrr@10 - type: dot_map@100 value: 0.5639260419913368 name: Dot Map@100 - type: query_active_dims value: 20.5 name: Query Active Dims - type: query_sparsity_ratio value: 0.9993283533189175 name: Query Sparsity Ratio - type: corpus_active_dims value: 81.87666320800781 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9973174541901578 name: Corpus Sparsity Ratio - task: type: sparse-information-retrieval name: Sparse Information Retrieval dataset: name: NanoNFCorpus type: NanoNFCorpus metrics: - type: dot_accuracy@1 value: 0.4 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.52 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.54 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.66 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.4 name: Dot Precision@1 - type: dot_precision@3 value: 0.3666666666666667 name: Dot Precision@3 - type: dot_precision@5 value: 0.332 name: Dot Precision@5 - type: dot_precision@10 value: 0.27 name: Dot Precision@10 - type: dot_recall@1 value: 0.023282599806398227 name: Dot Recall@1 - type: dot_recall@3 value: 0.07519782108259539 name: Dot Recall@3 - type: dot_recall@5 value: 0.09254782270412643 name: Dot Recall@5 - type: dot_recall@10 value: 0.12120665375595915 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.32050254842735026 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.4703888888888889 name: Dot Mrr@10 - type: dot_map@100 value: 0.13331879084552362 name: Dot Map@100 - type: query_active_dims value: 17.639999389648438 name: Query Active Dims - type: query_sparsity_ratio value: 0.9994220562417387 name: Query Sparsity Ratio - type: corpus_active_dims value: 165.31358337402344 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9945837892872674 name: Corpus Sparsity Ratio - type: dot_accuracy@1 value: 0.36 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.46 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.54 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.68 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.36 name: Dot Precision@1 - type: dot_precision@3 value: 0.34 name: Dot Precision@3 - type: dot_precision@5 value: 0.32799999999999996 name: Dot Precision@5 - type: dot_precision@10 value: 0.27 name: Dot Precision@10 - type: dot_recall@1 value: 0.02081925669789383 name: Dot Recall@1 - type: dot_recall@3 value: 0.07064967781220355 name: Dot Recall@3 - type: dot_recall@5 value: 0.09055307754310991 name: Dot Recall@5 - type: dot_recall@10 value: 0.14403725441385476 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.3196380424829849 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.4414444444444445 name: Dot Mrr@10 - type: dot_map@100 value: 0.13569627052041464 name: Dot Map@100 - type: query_active_dims value: 18.299999237060547 name: Query Active Dims - type: query_sparsity_ratio value: 0.9994004324999325 name: Query Sparsity Ratio - type: corpus_active_dims value: 156.04843139648438 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9948873458031424 name: Corpus Sparsity Ratio - task: type: sparse-information-retrieval name: Sparse Information Retrieval dataset: name: NanoNQ type: NanoNQ metrics: - type: dot_accuracy@1 value: 0.48 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.68 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.72 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.76 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.48 name: Dot Precision@1 - type: dot_precision@3 value: 0.22666666666666668 name: Dot Precision@3 - type: dot_precision@5 value: 0.14400000000000002 name: Dot Precision@5 - type: dot_precision@10 value: 0.08199999999999999 name: Dot Precision@10 - type: dot_recall@1 value: 0.46 name: Dot Recall@1 - type: dot_recall@3 value: 0.65 name: Dot Recall@3 - type: dot_recall@5 value: 0.68 name: Dot Recall@5 - type: dot_recall@10 value: 0.74 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.6136977374010735 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.585079365079365 name: Dot Mrr@10 - type: dot_map@100 value: 0.5730967720685111 name: Dot Map@100 - type: query_active_dims value: 24.299999237060547 name: Query Active Dims - type: query_sparsity_ratio value: 0.9992038529835181 name: Query Sparsity Ratio - type: corpus_active_dims value: 103.79106140136719 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9965994672235972 name: Corpus Sparsity Ratio - type: dot_accuracy@1 value: 0.48 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.68 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.74 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.76 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.48 name: Dot Precision@1 - type: dot_precision@3 value: 0.22666666666666668 name: Dot Precision@3 - type: dot_precision@5 value: 0.15200000000000002 name: Dot Precision@5 - type: dot_precision@10 value: 0.08 name: Dot Precision@10 - type: dot_recall@1 value: 0.47 name: Dot Recall@1 - type: dot_recall@3 value: 0.64 name: Dot Recall@3 - type: dot_recall@5 value: 0.7 name: Dot Recall@5 - type: dot_recall@10 value: 0.73 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.6150809765850531 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.5864999999999999 name: Dot Mrr@10 - type: dot_map@100 value: 0.5841443871983568 name: Dot Map@100 - type: query_active_dims value: 22.200000762939453 name: Query Active Dims - type: query_sparsity_ratio value: 0.9992726557642704 name: Query Sparsity Ratio - type: corpus_active_dims value: 103.72532653808594 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9966016209115365 name: Corpus Sparsity Ratio - task: type: sparse-nano-beir name: Sparse Nano BEIR dataset: name: NanoBEIR mean type: NanoBEIR_mean metrics: - type: dot_accuracy@1 value: 0.44 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.6200000000000001 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.66 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.7466666666666667 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.44 name: Dot Precision@1 - type: dot_precision@3 value: 0.27111111111111114 name: Dot Precision@3 - type: dot_precision@5 value: 0.2066666666666667 name: Dot Precision@5 - type: dot_precision@10 value: 0.14466666666666664 name: Dot Precision@10 - type: dot_recall@1 value: 0.30776086660213275 name: Dot Recall@1 - type: dot_recall@3 value: 0.4617326070275318 name: Dot Recall@3 - type: dot_recall@5 value: 0.49751594090137546 name: Dot Recall@5 - type: dot_recall@10 value: 0.5604022179186531 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.5188660948514809 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.5384708994708994 name: Dot Mrr@10 - type: dot_map@100 value: 0.42551732764853867 name: Dot Map@100 - type: query_active_dims value: 22.399999618530273 name: Query Active Dims - type: query_sparsity_ratio value: 0.9992661031512178 name: Query Sparsity Ratio - type: corpus_active_dims value: 112.03345893951784 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9963294194699063 name: Corpus Sparsity Ratio - type: dot_accuracy@1 value: 0.5241130298273154 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.6799372056514913 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.7415070643642072 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.8169230769230769 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.5241130298273154 name: Dot Precision@1 - type: dot_precision@3 value: 0.3215384615384615 name: Dot Precision@3 - type: dot_precision@5 value: 0.2547566718995291 name: Dot Precision@5 - type: dot_precision@10 value: 0.17874411302982732 name: Dot Precision@10 - type: dot_recall@1 value: 0.30856930592565196 name: Dot Recall@1 - type: dot_recall@3 value: 0.4441119539769697 name: Dot Recall@3 - type: dot_recall@5 value: 0.5092929381431597 name: Dot Recall@5 - type: dot_recall@10 value: 0.5878231569460904 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.5577320367017354 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.6173593605940545 name: Dot Mrr@10 - type: dot_map@100 value: 0.48084758588880655 name: Dot Map@100 - type: query_active_dims value: 38.07395960627058 name: Query Active Dims - type: query_sparsity_ratio value: 0.9987525732387698 name: Query Sparsity Ratio - type: corpus_active_dims value: 105.05153383516846 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9965581700466821 name: Corpus Sparsity Ratio - task: type: sparse-information-retrieval name: Sparse Information Retrieval dataset: name: NanoClimateFEVER type: NanoClimateFEVER metrics: - type: dot_accuracy@1 value: 0.24 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.42 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.56 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.64 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.24 name: Dot Precision@1 - type: dot_precision@3 value: 0.14666666666666664 name: Dot Precision@3 - type: dot_precision@5 value: 0.12 name: Dot Precision@5 - type: dot_precision@10 value: 0.07400000000000001 name: Dot Precision@10 - type: dot_recall@1 value: 0.11833333333333332 name: Dot Recall@1 - type: dot_recall@3 value: 0.21166666666666664 name: Dot Recall@3 - type: dot_recall@5 value: 0.26233333333333336 name: Dot Recall@5 - type: dot_recall@10 value: 0.29966666666666664 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.25712162589613363 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.35861111111111116 name: Dot Mrr@10 - type: dot_map@100 value: 0.20460406106488077 name: Dot Map@100 - type: query_active_dims value: 51.47999954223633 name: Query Active Dims - type: query_sparsity_ratio value: 0.9983133477641624 name: Query Sparsity Ratio - type: corpus_active_dims value: 134.2989959716797 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9955999280528248 name: Corpus Sparsity Ratio - task: type: sparse-information-retrieval name: Sparse Information Retrieval dataset: name: NanoDBPedia type: NanoDBPedia metrics: - type: dot_accuracy@1 value: 0.7 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.82 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.88 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.92 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.7 name: Dot Precision@1 - type: dot_precision@3 value: 0.6133333333333333 name: Dot Precision@3 - type: dot_precision@5 value: 0.58 name: Dot Precision@5 - type: dot_precision@10 value: 0.52 name: Dot Precision@10 - type: dot_recall@1 value: 0.05306233623739282 name: Dot Recall@1 - type: dot_recall@3 value: 0.16391544714816778 name: Dot Recall@3 - type: dot_recall@5 value: 0.23662708539883293 name: Dot Recall@5 - type: dot_recall@10 value: 0.3543605851621492 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.6137764330075132 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.771888888888889 name: Dot Mrr@10 - type: dot_map@100 value: 0.4604772150699302 name: Dot Map@100 - type: query_active_dims value: 20.520000457763672 name: Query Active Dims - type: query_sparsity_ratio value: 0.999327698038865 name: Query Sparsity Ratio - type: corpus_active_dims value: 111.07841491699219 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9963607098185902 name: Corpus Sparsity Ratio - task: type: sparse-information-retrieval name: Sparse Information Retrieval dataset: name: NanoFEVER type: NanoFEVER metrics: - type: dot_accuracy@1 value: 0.74 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.9 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.92 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.98 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.74 name: Dot Precision@1 - type: dot_precision@3 value: 0.3133333333333333 name: Dot Precision@3 - type: dot_precision@5 value: 0.19599999999999995 name: Dot Precision@5 - type: dot_precision@10 value: 0.10399999999999998 name: Dot Precision@10 - type: dot_recall@1 value: 0.7066666666666667 name: Dot Recall@1 - type: dot_recall@3 value: 0.8666666666666667 name: Dot Recall@3 - type: dot_recall@5 value: 0.8933333333333333 name: Dot Recall@5 - type: dot_recall@10 value: 0.9433333333333332 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.8368149756149829 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.8170000000000001 name: Dot Mrr@10 - type: dot_map@100 value: 0.7993556466302367 name: Dot Map@100 - type: query_active_dims value: 44.84000015258789 name: Query Active Dims - type: query_sparsity_ratio value: 0.9985308957423306 name: Query Sparsity Ratio - type: corpus_active_dims value: 154.09767150878906 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9949512590423697 name: Corpus Sparsity Ratio - task: type: sparse-information-retrieval name: Sparse Information Retrieval dataset: name: NanoFiQA2018 type: NanoFiQA2018 metrics: - type: dot_accuracy@1 value: 0.34 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.5 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.58 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.68 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.34 name: Dot Precision@1 - type: dot_precision@3 value: 0.21333333333333332 name: Dot Precision@3 - type: dot_precision@5 value: 0.17600000000000002 name: Dot Precision@5 - type: dot_precision@10 value: 0.11199999999999999 name: Dot Precision@10 - type: dot_recall@1 value: 0.1770793650793651 name: Dot Recall@1 - type: dot_recall@3 value: 0.3069920634920635 name: Dot Recall@3 - type: dot_recall@5 value: 0.3936825396825397 name: Dot Recall@5 - type: dot_recall@10 value: 0.48673809523809525 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.3901649596140352 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.4438809523809523 name: Dot Mrr@10 - type: dot_map@100 value: 0.32670074884185174 name: Dot Map@100 - type: query_active_dims value: 18.920000076293945 name: Query Active Dims - type: query_sparsity_ratio value: 0.9993801192557403 name: Query Sparsity Ratio - type: corpus_active_dims value: 75.49989318847656 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9975263779179453 name: Corpus Sparsity Ratio - task: type: sparse-information-retrieval name: Sparse Information Retrieval dataset: name: NanoHotpotQA type: NanoHotpotQA metrics: - type: dot_accuracy@1 value: 0.88 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.92 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.94 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.96 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.88 name: Dot Precision@1 - type: dot_precision@3 value: 0.4866666666666666 name: Dot Precision@3 - type: dot_precision@5 value: 0.324 name: Dot Precision@5 - type: dot_precision@10 value: 0.16999999999999996 name: Dot Precision@10 - type: dot_recall@1 value: 0.44 name: Dot Recall@1 - type: dot_recall@3 value: 0.73 name: Dot Recall@3 - type: dot_recall@5 value: 0.81 name: Dot Recall@5 - type: dot_recall@10 value: 0.85 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.8077539978128343 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.9041666666666667 name: Dot Mrr@10 - type: dot_map@100 value: 0.74474463747389 name: Dot Map@100 - type: query_active_dims value: 43.880001068115234 name: Query Active Dims - type: query_sparsity_ratio value: 0.9985623484349612 name: Query Sparsity Ratio - type: corpus_active_dims value: 120.78840637207031 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9960425789144856 name: Corpus Sparsity Ratio - task: type: sparse-information-retrieval name: Sparse Information Retrieval dataset: name: NanoQuoraRetrieval type: NanoQuoraRetrieval metrics: - type: dot_accuracy@1 value: 0.84 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.92 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.94 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.96 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.84 name: Dot Precision@1 - type: dot_precision@3 value: 0.32666666666666666 name: Dot Precision@3 - type: dot_precision@5 value: 0.22 name: Dot Precision@5 - type: dot_precision@10 value: 0.11999999999999998 name: Dot Precision@10 - type: dot_recall@1 value: 0.7873333333333333 name: Dot Recall@1 - type: dot_recall@3 value: 0.8540000000000001 name: Dot Recall@3 - type: dot_recall@5 value: 0.898 name: Dot Recall@5 - type: dot_recall@10 value: 0.9313333333333332 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.8841170132005264 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.8805555555555554 name: Dot Mrr@10 - type: dot_map@100 value: 0.8625873756339163 name: Dot Map@100 - type: query_active_dims value: 18.760000228881836 name: Query Active Dims - type: query_sparsity_ratio value: 0.9993853613711787 name: Query Sparsity Ratio - type: corpus_active_dims value: 20.381887435913086 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9993322230707059 name: Corpus Sparsity Ratio - task: type: sparse-information-retrieval name: Sparse Information Retrieval dataset: name: NanoSCIDOCS type: NanoSCIDOCS metrics: - type: dot_accuracy@1 value: 0.42 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.6 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.64 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.76 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.42 name: Dot Precision@1 - type: dot_precision@3 value: 0.2866666666666667 name: Dot Precision@3 - type: dot_precision@5 value: 0.21999999999999997 name: Dot Precision@5 - type: dot_precision@10 value: 0.152 name: Dot Precision@10 - type: dot_recall@1 value: 0.086 name: Dot Recall@1 - type: dot_recall@3 value: 0.17666666666666664 name: Dot Recall@3 - type: dot_recall@5 value: 0.22466666666666665 name: Dot Recall@5 - type: dot_recall@10 value: 0.3116666666666667 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.31329169156104253 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.5258253968253969 name: Dot Mrr@10 - type: dot_map@100 value: 0.24015404586272074 name: Dot Map@100 - type: query_active_dims value: 38.599998474121094 name: Query Active Dims - type: query_sparsity_ratio value: 0.9987353384943936 name: Query Sparsity Ratio - type: corpus_active_dims value: 120.28081512451172 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9960592092548158 name: Corpus Sparsity Ratio - task: type: sparse-information-retrieval name: Sparse Information Retrieval dataset: name: NanoArguAna type: NanoArguAna metrics: - type: dot_accuracy@1 value: 0.1 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.34 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.46 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.66 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.1 name: Dot Precision@1 - type: dot_precision@3 value: 0.11333333333333333 name: Dot Precision@3 - type: dot_precision@5 value: 0.09200000000000001 name: Dot Precision@5 - type: dot_precision@10 value: 0.06600000000000002 name: Dot Precision@10 - type: dot_recall@1 value: 0.1 name: Dot Recall@1 - type: dot_recall@3 value: 0.34 name: Dot Recall@3 - type: dot_recall@5 value: 0.46 name: Dot Recall@5 - type: dot_recall@10 value: 0.66 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.35624387960476495 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.2620238095238095 name: Dot Mrr@10 - type: dot_map@100 value: 0.27408886435627244 name: Dot Map@100 - type: query_active_dims value: 121.0199966430664 name: Query Active Dims - type: query_sparsity_ratio value: 0.9960349912639058 name: Query Sparsity Ratio - type: corpus_active_dims value: 107.16836547851562 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9964888157565521 name: Corpus Sparsity Ratio - task: type: sparse-information-retrieval name: Sparse Information Retrieval dataset: name: NanoSciFact type: NanoSciFact metrics: - type: dot_accuracy@1 value: 0.6 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.72 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.72 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 0.78 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.6 name: Dot Precision@1 - type: dot_precision@3 value: 0.24666666666666665 name: Dot Precision@3 - type: dot_precision@5 value: 0.16399999999999998 name: Dot Precision@5 - type: dot_precision@10 value: 0.088 name: Dot Precision@10 - type: dot_recall@1 value: 0.565 name: Dot Recall@1 - type: dot_recall@3 value: 0.68 name: Dot Recall@3 - type: dot_recall@5 value: 0.71 name: Dot Recall@5 - type: dot_recall@10 value: 0.77 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.6798182226611048 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.6625 name: Dot Mrr@10 - type: dot_map@100 value: 0.6532896014216637 name: Dot Map@100 - type: query_active_dims value: 57.41999816894531 name: Query Active Dims - type: query_sparsity_ratio value: 0.9981187340879056 name: Query Sparsity Ratio - type: corpus_active_dims value: 158.03323364257812 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9948223172255234 name: Corpus Sparsity Ratio - task: type: sparse-information-retrieval name: Sparse Information Retrieval dataset: name: NanoTouche2020 type: NanoTouche2020 metrics: - type: dot_accuracy@1 value: 0.673469387755102 name: Dot Accuracy@1 - type: dot_accuracy@3 value: 0.9591836734693877 name: Dot Accuracy@3 - type: dot_accuracy@5 value: 0.9795918367346939 name: Dot Accuracy@5 - type: dot_accuracy@10 value: 1.0 name: Dot Accuracy@10 - type: dot_precision@1 value: 0.673469387755102 name: Dot Precision@1 - type: dot_precision@3 value: 0.6666666666666667 name: Dot Precision@3 - type: dot_precision@5 value: 0.5918367346938777 name: Dot Precision@5 - type: dot_precision@10 value: 0.4836734693877551 name: Dot Precision@10 - type: dot_recall@1 value: 0.04710668568549065 name: Dot Recall@1 - type: dot_recall@3 value: 0.13289821324817133 name: Dot Recall@3 - type: dot_recall@5 value: 0.20161215990326012 name: Dot Recall@5 - type: dot_recall@10 value: 0.3205651054850781 name: Dot Recall@10 - type: dot_ndcg@10 value: 0.5525193350177682 name: Dot Ndcg@10 - type: dot_mrr@10 value: 0.814139941690962 name: Dot Mrr@10 - type: dot_map@100 value: 0.40124972048901353 name: Dot Map@100 - type: query_active_dims value: 18.12244987487793 name: Query Active Dims - type: query_sparsity_ratio value: 0.9994062495945587 name: Query Sparsity Ratio - type: corpus_active_dims value: 84.7328109741211 name: Corpus Active Dims - type: corpus_sparsity_ratio value: 0.9972238774990461 name: Corpus Sparsity Ratio --- # splade-distilbert-base-uncased trained on MS MARCO triplets This is a [SPLADE Sparse Encoder](https://www.sbert.net/docs/sparse_encoder/usage/usage.html) model finetuned from [distilbert/distilbert-base-uncased](https://huggingface.co/distilbert/distilbert-base-uncased) on the [msmarco](https://huggingface.co/datasets/sentence-transformers/msmarco) dataset using the [sentence-transformers](https://www.SBERT.net) library. It maps sentences & paragraphs to a 30522-dimensional sparse vector space and can be used for semantic search and sparse retrieval. ## Model Details ### Model Description - **Model Type:** SPLADE Sparse Encoder - **Base model:** [distilbert/distilbert-base-uncased](https://huggingface.co/distilbert/distilbert-base-uncased) - **Maximum Sequence Length:** 256 tokens - **Output Dimensionality:** 30522 dimensions - **Similarity Function:** Dot Product - **Training Dataset:** - [msmarco](https://huggingface.co/datasets/sentence-transformers/msmarco) - **Language:** en - **License:** apache-2.0 ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Documentation:** [Sparse Encoder Documentation](https://www.sbert.net/docs/sparse_encoder/usage/usage.html) - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) - **Hugging Face:** [Sparse Encoders on Hugging Face](https://huggingface.co/models?library=sentence-transformers&other=sparse-encoder) ### Full Model Architecture ``` SparseEncoder( (0): MLMTransformer({'max_seq_length': 256, 'do_lower_case': False}) with MLMTransformer model: DistilBertForMaskedLM (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': 30522}) ) ``` ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import SparseEncoder # Download from the 🤗 Hub model = SparseEncoder("arthurbresnu/splade-distilbert-base-uncased-msmarco-mrl") # Run inference queries = [ "meaning of the name bernard", ] documents = [ 'English Meaning: The name Bernard is an English baby name. In English the meaning of the name Bernard is: Strong as a bear. See also Bjorn. American Meaning: The name Bernard is an American baby name. In American the meaning of the name Bernard is: Strong as a bear.', 'To the Citizens of St. Bernard We chose as our motto a simple but profound declaration: â\x80\x9cWelcome to your office.â\x80\x9d Those words remind us that we are no more than the caretakers of the office of Clerk of Court for the Parish of St. Bernard.', "Get Your Prior Years Tax Information from the IRS. IRS Tax Tip 2012-18, January 27, 2012. Sometimes taxpayers need a copy of an old tax return, but can't find or don't have their own records. There are three easy and convenient options for getting tax return transcripts and tax account transcripts from the IRS: on the web, by phone or by mail.", ] query_embeddings = model.encode_query(queries) document_embeddings = model.encode_document(documents) print(query_embeddings.shape, document_embeddings.shape) # [1, 30522] [3, 30522] # Get the similarity scores for the embeddings similarities = model.similarity(query_embeddings, document_embeddings) print(similarities) # tensor([[18.6221, 10.0646, 0.0000]]) ``` ## Evaluation ### Metrics #### Sparse Information Retrieval * Datasets: `NanoMSMARCO`, `NanoNFCorpus`, `NanoNQ`, `NanoClimateFEVER`, `NanoDBPedia`, `NanoFEVER`, `NanoFiQA2018`, `NanoHotpotQA`, `NanoMSMARCO`, `NanoNFCorpus`, `NanoNQ`, `NanoQuoraRetrieval`, `NanoSCIDOCS`, `NanoArguAna`, `NanoSciFact` and `NanoTouche2020` * Evaluated with [SparseInformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sparse_encoder/evaluation.html#sentence_transformers.sparse_encoder.evaluation.SparseInformationRetrievalEvaluator) | Metric | NanoMSMARCO | NanoNFCorpus | NanoNQ | NanoClimateFEVER | NanoDBPedia | NanoFEVER | NanoFiQA2018 | NanoHotpotQA | NanoQuoraRetrieval | NanoSCIDOCS | NanoArguAna | NanoSciFact | NanoTouche2020 | |:----------------------|:------------|:-------------|:-----------|:-----------------|:------------|:-----------|:-------------|:-------------|:-------------------|:------------|:------------|:------------|:---------------| | dot_accuracy@1 | 0.44 | 0.36 | 0.48 | 0.24 | 0.7 | 0.74 | 0.34 | 0.88 | 0.84 | 0.42 | 0.1 | 0.6 | 0.6735 | | dot_accuracy@3 | 0.6 | 0.46 | 0.68 | 0.42 | 0.82 | 0.9 | 0.5 | 0.92 | 0.92 | 0.6 | 0.34 | 0.72 | 0.9592 | | dot_accuracy@5 | 0.74 | 0.54 | 0.74 | 0.56 | 0.88 | 0.92 | 0.58 | 0.94 | 0.94 | 0.64 | 0.46 | 0.72 | 0.9796 | | dot_accuracy@10 | 0.84 | 0.68 | 0.76 | 0.64 | 0.92 | 0.98 | 0.68 | 0.96 | 0.96 | 0.76 | 0.66 | 0.78 | 1.0 | | dot_precision@1 | 0.44 | 0.36 | 0.48 | 0.24 | 0.7 | 0.74 | 0.34 | 0.88 | 0.84 | 0.42 | 0.1 | 0.6 | 0.6735 | | dot_precision@3 | 0.2 | 0.34 | 0.2267 | 0.1467 | 0.6133 | 0.3133 | 0.2133 | 0.4867 | 0.3267 | 0.2867 | 0.1133 | 0.2467 | 0.6667 | | dot_precision@5 | 0.148 | 0.328 | 0.152 | 0.12 | 0.58 | 0.196 | 0.176 | 0.324 | 0.22 | 0.22 | 0.092 | 0.164 | 0.5918 | | dot_precision@10 | 0.084 | 0.27 | 0.08 | 0.074 | 0.52 | 0.104 | 0.112 | 0.17 | 0.12 | 0.152 | 0.066 | 0.088 | 0.4837 | | dot_recall@1 | 0.44 | 0.0208 | 0.47 | 0.1183 | 0.0531 | 0.7067 | 0.1771 | 0.44 | 0.7873 | 0.086 | 0.1 | 0.565 | 0.0471 | | dot_recall@3 | 0.6 | 0.0706 | 0.64 | 0.2117 | 0.1639 | 0.8667 | 0.307 | 0.73 | 0.854 | 0.1767 | 0.34 | 0.68 | 0.1329 | | dot_recall@5 | 0.74 | 0.0906 | 0.7 | 0.2623 | 0.2366 | 0.8933 | 0.3937 | 0.81 | 0.898 | 0.2247 | 0.46 | 0.71 | 0.2016 | | dot_recall@10 | 0.84 | 0.144 | 0.73 | 0.2997 | 0.3544 | 0.9433 | 0.4867 | 0.85 | 0.9313 | 0.3117 | 0.66 | 0.77 | 0.3206 | | **dot_ndcg@10** | **0.6242** | **0.3196** | **0.6151** | **0.2571** | **0.6138** | **0.8368** | **0.3902** | **0.8078** | **0.8841** | **0.3133** | **0.3562** | **0.6798** | **0.5525** | | dot_mrr@10 | 0.5571 | 0.4414 | 0.5865 | 0.3586 | 0.7719 | 0.817 | 0.4439 | 0.9042 | 0.8806 | 0.5258 | 0.262 | 0.6625 | 0.8141 | | dot_map@100 | 0.5639 | 0.1357 | 0.5841 | 0.2046 | 0.4605 | 0.7994 | 0.3267 | 0.7447 | 0.8626 | 0.2402 | 0.2741 | 0.6533 | 0.4012 | | query_active_dims | 20.5 | 18.3 | 22.2 | 51.48 | 20.52 | 44.84 | 18.92 | 43.88 | 18.76 | 38.6 | 121.02 | 57.42 | 18.1224 | | query_sparsity_ratio | 0.9993 | 0.9994 | 0.9993 | 0.9983 | 0.9993 | 0.9985 | 0.9994 | 0.9986 | 0.9994 | 0.9987 | 0.996 | 0.9981 | 0.9994 | | corpus_active_dims | 81.8767 | 156.0484 | 103.7253 | 134.299 | 111.0784 | 154.0977 | 75.4999 | 120.7884 | 20.3819 | 120.2808 | 107.1684 | 158.0332 | 84.7328 | | corpus_sparsity_ratio | 0.9973 | 0.9949 | 0.9966 | 0.9956 | 0.9964 | 0.995 | 0.9975 | 0.996 | 0.9993 | 0.9961 | 0.9965 | 0.9948 | 0.9972 | #### Sparse Nano BEIR * Dataset: `NanoBEIR_mean` * Evaluated with [SparseNanoBEIREvaluator](https://sbert.net/docs/package_reference/sparse_encoder/evaluation.html#sentence_transformers.sparse_encoder.evaluation.SparseNanoBEIREvaluator) with these parameters: ```json { "dataset_names": [ "msmarco", "nfcorpus", "nq" ] } ``` | Metric | Value | |:----------------------|:-----------| | dot_accuracy@1 | 0.44 | | dot_accuracy@3 | 0.62 | | dot_accuracy@5 | 0.66 | | dot_accuracy@10 | 0.7467 | | dot_precision@1 | 0.44 | | dot_precision@3 | 0.2711 | | dot_precision@5 | 0.2067 | | dot_precision@10 | 0.1447 | | dot_recall@1 | 0.3078 | | dot_recall@3 | 0.4617 | | dot_recall@5 | 0.4975 | | dot_recall@10 | 0.5604 | | **dot_ndcg@10** | **0.5189** | | dot_mrr@10 | 0.5385 | | dot_map@100 | 0.4255 | | query_active_dims | 22.4 | | query_sparsity_ratio | 0.9993 | | corpus_active_dims | 112.0335 | | corpus_sparsity_ratio | 0.9963 | #### Sparse Nano BEIR * Dataset: `NanoBEIR_mean` * Evaluated with [SparseNanoBEIREvaluator](https://sbert.net/docs/package_reference/sparse_encoder/evaluation.html#sentence_transformers.sparse_encoder.evaluation.SparseNanoBEIREvaluator) with these parameters: ```json { "dataset_names": [ "climatefever", "dbpedia", "fever", "fiqa2018", "hotpotqa", "msmarco", "nfcorpus", "nq", "quoraretrieval", "scidocs", "arguana", "scifact", "touche2020" ] } ``` | Metric | Value | |:----------------------|:-----------| | dot_accuracy@1 | 0.5241 | | dot_accuracy@3 | 0.6799 | | dot_accuracy@5 | 0.7415 | | dot_accuracy@10 | 0.8169 | | dot_precision@1 | 0.5241 | | dot_precision@3 | 0.3215 | | dot_precision@5 | 0.2548 | | dot_precision@10 | 0.1787 | | dot_recall@1 | 0.3086 | | dot_recall@3 | 0.4441 | | dot_recall@5 | 0.5093 | | dot_recall@10 | 0.5878 | | **dot_ndcg@10** | **0.5577** | | dot_mrr@10 | 0.6174 | | dot_map@100 | 0.4808 | | query_active_dims | 38.074 | | query_sparsity_ratio | 0.9988 | | corpus_active_dims | 105.0515 | | corpus_sparsity_ratio | 0.9966 | ## Training Details ### Training Dataset #### msmarco * Dataset: [msmarco](https://huggingface.co/datasets/sentence-transformers/msmarco) at [9e329ed](https://huggingface.co/datasets/sentence-transformers/msmarco/tree/9e329ed2e649c9d37b0d91dd6b764ff6fe671d83) * Size: 90,000 training samples * Columns: query, positive, and negative * Approximate statistics based on the first 1000 samples: | | query | positive | negative | |:--------|:---------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------| | type | string | string | string | | details | | | | * Samples: | query | positive | negative | |:--------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | yosemite temperature in september | Here are the average temp in Yosemite Valley (where CV is located) by month: www.nps.gov/yose/planyourvisit/climate.htm. Also beginning of September is usually still quite warm. Nights can have a bit of a chill, but nothing a couple of blankets can't handle. | Guide to Switzerland weather in September. The average maximum daytime temperature in Switzerland in September is a comfortable 18°C (64°F). The average night-time temperature is usually a cool 9°C (48°F). There are usually 6 hours of bright sunshine each day, which represents 45% of the 13 hours of daylight. | | what is genus | Intermediate minor rankings are not shown. A genus (/ˈdʒiːnəs/, pl. genera) is a taxonomic rank used in the biological classification of living and fossil organisms in biology. In the hierarchy of biological classification, genus comes above species and below family. In binomial nomenclature, the genus name forms the first part of the binomial species name for each species within the genus. The composition of a genus is determined by a taxonomist. | The genus is the first part of a scientific name. Note that the genus is always capitalised. An example: Lemur catta is the scientific name of the Ringtailed lemur and Lemur … is the genus.Another example: Sphyrna zygaena is the scientific name of one species of Hammerhead shark and Sphyrna is the genus. name used all around the world to classify a living organism. It is composed of a genus and species name. A sceintific name can also be considered for non living things, the … se are usually called scientific jargon, or very simply 'proper names for the things around you'. 4 people found this useful. | | what did johannes kepler discover about the motion of the planets? | Johannes Kepler devised his three laws of motion from his observations of planets that are fundamental to our understanding of orbital motions. | Little Street, Johannes Vermeer, c. 1658. New stop on Delft tourist trail after Vermeer's Little Street identified. Few artists have left such a deep imprint on their birthplace as Johannes Vermeer on Delft. In the summer, tour parties weave through the Dutch town’s cobbled streets ticking off Vermeer landmarks. | * Loss: [SpladeLoss](https://sbert.net/docs/package_reference/sparse_encoder/losses.html#spladeloss) with these parameters: ```json { "loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score')", "lambda_corpus": 0.001, "lambda_query": 5e-05 } ``` ### Evaluation Dataset #### msmarco * Dataset: [msmarco](https://huggingface.co/datasets/sentence-transformers/msmarco) at [9e329ed](https://huggingface.co/datasets/sentence-transformers/msmarco/tree/9e329ed2e649c9d37b0d91dd6b764ff6fe671d83) * Size: 10,000 evaluation samples * Columns: query, positive, and negative * Approximate statistics based on the first 1000 samples: | | query | positive | negative | |:--------|:---------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------| | type | string | string | string | | details | | | | * Samples: | query | positive | negative | |:---------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | scarehouse cast | The Scarehouse. The Scarehouse is a 2014 Canadian horror film directed by Gavin Michael Booth. It stars Sarah Booth and Kimberly-Sue Murray as two women who seek revenge against their former sorority. | Nathalie Emmanuel joined the TV series as a recurring cast member in Season 3, and continued as a recurring cast member into Season 4. Emmanuel was later promoted to a starring cast member for seasons 5 and 6. | | population of bellemont arizona | The 2016 Bellemont (zip 86015), Arizona, population is 300. There are 55 people per square mile (population density). The median age is 29.9. The US median is 37.4. 38.19% of people in Bellemont (zip 86015), Arizona, are married. | • Arizona: A 2010 University of Arizona report estimates that 40% of the state's kissing bugs carry a parasite strain related to the Chagas disease but rarely transmit the disease to humans. The Arizona Department of Health Services reported one Chagas disease-related death in 2013, reports The Arizona Republic. | | does air transat check bag size | • Weight must be 10kg (22 lb) in Economy class and in Option Plus and 15 kg (33lb) in Club Class. Checked Baggage Air Transat allows for multiple pieces, as long as the combined weight does not exceed weight limitations. • Length + width + height must not exceed 158cm (62 in). | Bag-valve masks come in different sizes to fit infants, children, and adults. The face mask size may be independent of the bag size; for example, a single pediatric-sized bag might be used with different masks for multiple face sizes, or a pediatric mask might be used with an adult bag for patients with small faces. | * Loss: [SpladeLoss](https://sbert.net/docs/package_reference/sparse_encoder/losses.html#spladeloss) with these parameters: ```json { "loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score')", "lambda_corpus": 0.001, "lambda_query": 5e-05 } ``` ### Training Hyperparameters #### Non-Default Hyperparameters - `eval_strategy`: steps - `per_device_train_batch_size`: 16 - `per_device_eval_batch_size`: 16 - `learning_rate`: 2e-05 - `num_train_epochs`: 1 - `warmup_ratio`: 0.1 - `bf16`: True - `load_best_model_at_end`: True - `batch_sampler`: no_duplicates #### All Hyperparameters
Click to expand - `overwrite_output_dir`: False - `do_predict`: False - `eval_strategy`: steps - `prediction_loss_only`: True - `per_device_train_batch_size`: 16 - `per_device_eval_batch_size`: 16 - `per_gpu_train_batch_size`: None - `per_gpu_eval_batch_size`: None - `gradient_accumulation_steps`: 1 - `eval_accumulation_steps`: None - `torch_empty_cache_steps`: None - `learning_rate`: 2e-05 - `weight_decay`: 0.0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `max_grad_norm`: 1.0 - `num_train_epochs`: 1 - `max_steps`: -1 - `lr_scheduler_type`: linear - `lr_scheduler_kwargs`: {} - `warmup_ratio`: 0.1 - `warmup_steps`: 0 - `log_level`: passive - `log_level_replica`: warning - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `save_safetensors`: True - `save_on_each_node`: False - `save_only_model`: False - `restore_callback_states_from_checkpoint`: False - `no_cuda`: False - `use_cpu`: False - `use_mps_device`: False - `seed`: 42 - `data_seed`: None - `jit_mode_eval`: False - `use_ipex`: False - `bf16`: True - `fp16`: False - `fp16_opt_level`: O1 - `half_precision_backend`: auto - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: None - `local_rank`: 0 - `ddp_backend`: None - `tpu_num_cores`: None - `tpu_metrics_debug`: False - `debug`: [] - `dataloader_drop_last`: False - `dataloader_num_workers`: 0 - `dataloader_prefetch_factor`: None - `past_index`: -1 - `disable_tqdm`: False - `remove_unused_columns`: True - `label_names`: None - `load_best_model_at_end`: True - `ignore_data_skip`: False - `fsdp`: [] - `fsdp_min_num_params`: 0 - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `tp_size`: 0 - `fsdp_transformer_layer_cls_to_wrap`: None - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `deepspeed`: None - `label_smoothing_factor`: 0.0 - `optim`: adamw_torch - `optim_args`: None - `adafactor`: False - `group_by_length`: False - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `skip_memory_metrics`: True - `use_legacy_prediction_loop`: False - `push_to_hub`: False - `resume_from_checkpoint`: None - `hub_model_id`: None - `hub_strategy`: every_save - `hub_private_repo`: None - `hub_always_push`: False - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `include_inputs_for_metrics`: False - `include_for_metrics`: [] - `eval_do_concat_batches`: True - `fp16_backend`: auto - `push_to_hub_model_id`: None - `push_to_hub_organization`: None - `mp_parameters`: - `auto_find_batch_size`: False - `full_determinism`: False - `torchdynamo`: None - `ray_scope`: last - `ddp_timeout`: 1800 - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `include_tokens_per_second`: False - `include_num_input_tokens_seen`: False - `neftune_noise_alpha`: None - `optim_target_modules`: None - `batch_eval_metrics`: False - `eval_on_start`: False - `use_liger_kernel`: False - `eval_use_gather_object`: False - `average_tokens_across_devices`: False - `prompts`: None - `batch_sampler`: no_duplicates - `multi_dataset_batch_sampler`: proportional - `router_mapping`: {} - `learning_rate_mapping`: {}
### Training Logs | Epoch | Step | Training Loss | Validation Loss | NanoMSMARCO_dot_ndcg@10 | NanoNFCorpus_dot_ndcg@10 | NanoNQ_dot_ndcg@10 | NanoBEIR_mean_dot_ndcg@10 | NanoClimateFEVER_dot_ndcg@10 | NanoDBPedia_dot_ndcg@10 | NanoFEVER_dot_ndcg@10 | NanoFiQA2018_dot_ndcg@10 | NanoHotpotQA_dot_ndcg@10 | NanoQuoraRetrieval_dot_ndcg@10 | NanoSCIDOCS_dot_ndcg@10 | NanoArguAna_dot_ndcg@10 | NanoSciFact_dot_ndcg@10 | NanoTouche2020_dot_ndcg@10 | |:----------:|:--------:|:-------------:|:---------------:|:-----------------------:|:------------------------:|:------------------:|:-------------------------:|:----------------------------:|:-----------------------:|:---------------------:|:------------------------:|:------------------------:|:------------------------------:|:-----------------------:|:-----------------------:|:-----------------------:|:--------------------------:| | 0.0178 | 100 | 199.0423 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.0356 | 200 | 11.3558 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.0533 | 300 | 0.9845 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.0711 | 400 | 0.4726 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.0889 | 500 | 0.2639 | 0.2407 | 0.5514 | 0.3061 | 0.5649 | 0.4741 | - | - | - | - | - | - | - | - | - | - | | 0.1067 | 600 | 0.2931 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.1244 | 700 | 0.2301 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.1422 | 800 | 0.2168 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.16 | 900 | 0.1741 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.1778 | 1000 | 0.1852 | 0.1878 | 0.5868 | 0.2975 | 0.5648 | 0.4830 | - | - | - | - | - | - | - | - | - | - | | 0.1956 | 1100 | 0.1684 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.2133 | 1200 | 0.1629 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.2311 | 1300 | 0.1736 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.2489 | 1400 | 0.1813 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.2667 | 1500 | 0.1826 | 0.1382 | 0.5941 | 0.3251 | 0.5911 | 0.5035 | - | - | - | - | - | - | - | - | - | - | | 0.2844 | 1600 | 0.177 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.3022 | 1700 | 0.1568 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.32 | 1800 | 0.1707 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.3378 | 1900 | 0.1554 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.3556 | 2000 | 0.1643 | 0.1553 | 0.6157 | 0.2997 | 0.5807 | 0.4987 | - | - | - | - | - | - | - | - | - | - | | 0.3733 | 2100 | 0.1564 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.3911 | 2200 | 0.1334 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.4089 | 2300 | 0.1349 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.4267 | 2400 | 0.1228 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | **0.4444** | **2500** | **0.1473** | **0.1239** | **0.6242** | **0.3196** | **0.6151** | **0.5196** | **-** | **-** | **-** | **-** | **-** | **-** | **-** | **-** | **-** | **-** | | 0.4622 | 2600 | 0.1506 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.48 | 2700 | 0.1436 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.4978 | 2800 | 0.1471 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.5156 | 2900 | 0.1378 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.5333 | 3000 | 0.1248 | 0.1328 | 0.6077 | 0.3073 | 0.6022 | 0.5057 | - | - | - | - | - | - | - | - | - | - | | 0.5511 | 3100 | 0.1672 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.5689 | 3200 | 0.1301 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.5867 | 3300 | 0.1325 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.6044 | 3400 | 0.1335 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.6222 | 3500 | 0.122 | 0.1163 | 0.6081 | 0.3302 | 0.6190 | 0.5191 | - | - | - | - | - | - | - | - | - | - | | 0.64 | 3600 | 0.1369 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.6578 | 3700 | 0.1651 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.6756 | 3800 | 0.1243 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.6933 | 3900 | 0.1122 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.7111 | 4000 | 0.1308 | 0.1307 | 0.6013 | 0.3232 | 0.5981 | 0.5075 | - | - | - | - | - | - | - | - | - | - | | 0.7289 | 4100 | 0.1708 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.7467 | 4200 | 0.1143 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.7644 | 4300 | 0.167 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.7822 | 4400 | 0.1119 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.8 | 4500 | 0.1128 | 0.1177 | 0.6082 | 0.3228 | 0.5866 | 0.5058 | - | - | - | - | - | - | - | - | - | - | | 0.8178 | 4600 | 0.125 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.8356 | 4700 | 0.1252 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.8533 | 4800 | 0.1066 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.8711 | 4900 | 0.1196 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.8889 | 5000 | 0.1291 | 0.1120 | 0.6134 | 0.3230 | 0.6115 | 0.5160 | - | - | - | - | - | - | - | - | - | - | | 0.9067 | 5100 | 0.1219 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.9244 | 5200 | 0.1492 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.9422 | 5300 | 0.1138 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.96 | 5400 | 0.1583 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | 0.9778 | 5500 | 0.1516 | 0.1125 | 0.6224 | 0.3205 | 0.6137 | 0.5189 | - | - | - | - | - | - | - | - | - | - | | 0.9956 | 5600 | 0.1227 | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | | -1 | -1 | - | - | 0.6242 | 0.3196 | 0.6151 | 0.5577 | 0.2571 | 0.6138 | 0.8368 | 0.3902 | 0.8078 | 0.8841 | 0.3133 | 0.3562 | 0.6798 | 0.5525 | * The bold row denotes the saved checkpoint. ### Environmental Impact Carbon emissions were measured using [CodeCarbon](https://github.com/mlco2/codecarbon). - **Energy Consumed**: 0.057 kWh - **Carbon Emitted**: 0.021 kg of CO2 - **Hours Used**: 0.179 hours ### Training Hardware - **On Cloud**: No - **GPU Model**: 1 x NVIDIA H100 80GB HBM3 - **CPU Model**: AMD EPYC 7R13 Processor - **RAM Size**: 248.00 GB ### Framework Versions - Python: 3.13.3 - Sentence Transformers: 4.2.0.dev0 - Transformers: 4.51.3 - PyTorch: 2.7.1+cu126 - Accelerate: 0.26.0 - Datasets: 2.21.0 - Tokenizers: 0.21.1 ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ``` #### SpladeLoss ```bibtex @misc{formal2022distillationhardnegativesampling, title={From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective}, author={Thibault Formal and Carlos Lassance and Benjamin Piwowarski and Stéphane Clinchant}, year={2022}, eprint={2205.04733}, archivePrefix={arXiv}, primaryClass={cs.IR}, url={https://arxiv.org/abs/2205.04733}, } ``` #### SparseMultipleNegativesRankingLoss ```bibtex @misc{henderson2017efficient, title={Efficient Natural Language Response Suggestion for Smart Reply}, author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil}, year={2017}, eprint={1705.00652}, archivePrefix={arXiv}, primaryClass={cs.CL} } ``` #### FlopsLoss ```bibtex @article{paria2020minimizing, title={Minimizing flops to learn efficient sparse representations}, author={Paria, Biswajit and Yeh, Chih-Kuan and Yen, Ian EH and Xu, Ning and Ravikumar, Pradeep and P{'o}czos, Barnab{'a}s}, journal={arXiv preprint arXiv:2004.05665}, year={2020} } ```