sparse-encoder/example-splade-distilbert-base-uncased-quora-duplicates

splade-distilbert-base-uncased trained on Quora Duplicates Questions

This is a SPLADE Sparse Encoder model finetuned from distilbert/distilbert-base-uncased on the quora-duplicates dataset using the sentence-transformers library. It maps sentences & paragraphs to a 30522-dimensional sparse vector space and can be used for semantic search and sparse retrieval.

Model Details

Model Description

• Model Type: SPLADE Sparse Encoder
• Base model: distilbert/distilbert-base-uncased
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 30522 dimensions
• Similarity Function: Dot Product
• Training Dataset:
  • quora-duplicates
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Sparse Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sparse Encoders on Hugging Face

Full Model Architecture

SparseEncoder(
  (0): MLMTransformer({'max_seq_length': 256, 'do_lower_case': False}) with MLMTransformer model: DistilBertForMaskedLM 
  (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': 30522})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SparseEncoder

# Download from the 🤗 Hub
model = SparseEncoder("arthurbresnu/splade-distilbert-base-uncased-quora-duplicates")
# Run inference
sentences = [
    'What accomplishments did Hillary Clinton achieve during her time as Secretary of State?',
    "What are Hillary Clinton's most recognized accomplishments while Secretary of State?",
    'What are Hillary Clinton’s qualifications to be President?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# (3, 30522)

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Sparse Binary Classification

• Dataset: quora_duplicates_dev
• Evaluated with SparseBinaryClassificationEvaluator

Metric	Value
cosine_accuracy	0.758
cosine_accuracy_threshold	0.8166
cosine_f1	0.6793
cosine_f1_threshold	0.5696
cosine_precision	0.5488
cosine_recall	0.8913
cosine_ap	0.6888
cosine_mcc	0.5082
dot_accuracy	0.765
dot_accuracy_threshold	51.6699
dot_f1	0.6762
dot_f1_threshold	46.5249
dot_precision	0.5817
dot_recall	0.8075
dot_ap	0.6336
dot_mcc	0.4996
euclidean_accuracy	0.677
euclidean_accuracy_threshold	-14.2724
euclidean_f1	0.486
euclidean_f1_threshold	-0.6445
euclidean_precision	0.3213
euclidean_recall	0.9969
euclidean_ap	0.2033
euclidean_mcc	-0.0459
manhattan_accuracy	0.677
manhattan_accuracy_threshold	-161.7768
manhattan_f1	0.486
manhattan_f1_threshold	-3.0495
manhattan_precision	0.3213
manhattan_recall	0.9969
manhattan_ap	0.2044
manhattan_mcc	-0.0459
max_accuracy	0.765
max_accuracy_threshold	51.6699
max_f1	0.6793
max_f1_threshold	46.5249
max_precision	0.5817
max_recall	0.9969
max_ap	0.6888
max_mcc	0.5082
active_dims	78.3228
sparsity_ratio	0.9974

Sparse Information Retrieval

• Datasets: NanoMSMARCO, NanoNQ, NanoNFCorpus, NanoQuoraRetrieval, NanoClimateFEVER, NanoDBPedia, NanoFEVER, NanoFiQA2018, NanoHotpotQA, NanoMSMARCO, NanoNFCorpus, NanoNQ, NanoQuoraRetrieval, NanoSCIDOCS, NanoArguAna, NanoSciFact and NanoTouche2020
• Evaluated with SparseInformationRetrievalEvaluator

Metric	NanoMSMARCO	NanoNQ	NanoNFCorpus	NanoQuoraRetrieval	NanoClimateFEVER	NanoDBPedia	NanoFEVER	NanoFiQA2018	NanoHotpotQA	NanoSCIDOCS	NanoArguAna	NanoSciFact	NanoTouche2020
dot_accuracy@1	0.22	0.38	0.34	0.94	0.18	0.6	0.58	0.28	0.78	0.36	0.1	0.52	0.3878
dot_accuracy@3	0.42	0.54	0.5	0.98	0.32	0.84	0.76	0.42	0.84	0.58	0.34	0.62	0.7551
dot_accuracy@5	0.52	0.62	0.54	0.98	0.4	0.84	0.8	0.46	0.92	0.68	0.42	0.64	0.8367
dot_accuracy@10	0.76	0.62	0.58	0.98	0.48	0.92	0.86	0.5	0.98	0.76	0.44	0.76	0.9592
dot_precision@1	0.22	0.38	0.34	0.94	0.18	0.6	0.58	0.28	0.78	0.36	0.1	0.52	0.3878
dot_precision@3	0.14	0.18	0.3067	0.3933	0.1067	0.5267	0.2667	0.18	0.3733	0.2733	0.1133	0.2133	0.4694
dot_precision@5	0.104	0.124	0.26	0.248	0.084	0.456	0.168	0.136	0.284	0.212	0.084	0.14	0.4163
dot_precision@10	0.076	0.064	0.198	0.132	0.054	0.422	0.09	0.084	0.16	0.152	0.044	0.084	0.3388
dot_recall@1	0.22	0.36	0.0116	0.8173	0.085	0.0457	0.5467	0.1418	0.39	0.0757	0.1	0.475	0.0238
dot_recall@3	0.42	0.52	0.0606	0.928	0.1467	0.1537	0.7467	0.2429	0.56	0.1697	0.34	0.58	0.0893
dot_recall@5	0.52	0.6	0.0826	0.946	0.1783	0.1908	0.7867	0.2772	0.71	0.2177	0.42	0.615	0.1288
dot_recall@10	0.76	0.61	0.098	0.97	0.215	0.2936	0.8467	0.3289	0.8	0.3107	0.44	0.74	0.2166
dot_ndcg@10	0.4532	0.4828	0.2467	0.9467	0.1845	0.5071	0.707	0.2881	0.7143	0.3029	0.2782	0.6021	0.3762
dot_mrr@10	0.3601	0.4537	0.422	0.96	0.2674	0.7147	0.6765	0.3562	0.8362	0.4944	0.2247	0.5799	0.5807
dot_map@100	0.3733	0.4454	0.094	0.9291	0.1476	0.3907	0.6651	0.2415	0.6181	0.2167	0.2333	0.5551	0.2561
query_active_dims	74.76	74.74	79.7	76.58	89.86	69.52	89.88	82.86	91.98	94.3	189.1	82.46	79.1225
query_sparsity_ratio	0.9976	0.9976	0.9974	0.9975	0.9971	0.9977	0.9971	0.9973	0.997	0.9969	0.9938	0.9973	0.9974
corpus_active_dims	103.0652	141.3148	202.1727	77.5906	221.7553	135.9335	221.2156	130.937	152.0157	199.6463	164.0333	194.2494	135.0078
corpus_sparsity_ratio	0.9966	0.9954	0.9934	0.9975	0.9927	0.9955	0.9928	0.9957	0.995	0.9935	0.9946	0.9936	0.9956

Sparse Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with SparseNanoBEIREvaluator with these parameters:

  {
      "dataset_names": [
          "msmarco",
          "nq",
          "nfcorpus",
          "quoraretrieval"
      ]
  }
  

Metric	Value
dot_accuracy@1	0.47
dot_accuracy@3	0.61
dot_accuracy@5	0.665
dot_accuracy@10	0.735
dot_precision@1	0.47
dot_precision@3	0.255
dot_precision@5	0.184
dot_precision@10	0.1175
dot_recall@1	0.3522
dot_recall@3	0.4821
dot_recall@5	0.5372
dot_recall@10	0.6095
dot_ndcg@10	0.5324
dot_mrr@10	0.5489
dot_map@100	0.4605
query_active_dims	76.445
query_sparsity_ratio	0.9975
corpus_active_dims	122.7978
corpus_sparsity_ratio	0.996

Sparse Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with SparseNanoBEIREvaluator with these parameters:

  {
      "dataset_names": [
          "climatefever",
          "dbpedia",
          "fever",
          "fiqa2018",
          "hotpotqa",
          "msmarco",
          "nfcorpus",
          "nq",
          "quoraretrieval",
          "scidocs",
          "arguana",
          "scifact",
          "touche2020"
      ]
  }
  

Metric	Value
dot_accuracy@1	0.436
dot_accuracy@3	0.6089
dot_accuracy@5	0.6659
dot_accuracy@10	0.7384
dot_precision@1	0.436
dot_precision@3	0.2725
dot_precision@5	0.2089
dot_precision@10	0.1461
dot_recall@1	0.2533
dot_recall@3	0.3813
dot_recall@5	0.4364
dot_recall@10	0.51
dot_ndcg@10	0.4685
dot_mrr@10	0.5328
dot_map@100	0.3974
query_active_dims	90.3914
query_sparsity_ratio	0.997
corpus_active_dims	152.3669
corpus_sparsity_ratio	0.995

Training Details

Training Dataset

quora-duplicates

• Dataset: quora-duplicates at 451a485
• Size: 99,000 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 6 tokens mean: 14.1 tokens max: 39 tokens	min: 6 tokens mean: 13.83 tokens max: 41 tokens	min: 6 tokens mean: 15.21 tokens max: 75 tokens

• Samples:

  anchor	positive	negative
  What are the best GMAT coaching institutes in Delhi NCR?	Which are the best GMAT coaching institutes in Delhi/NCR?	What are the best GMAT coaching institutes in Delhi-Noida Area?
  Is a third world war coming?	Is World War 3 more imminent than expected?	Since the UN is unable to control terrorism and groups like ISIS, al-Qaeda and countries that promote terrorism (even though it consumed those countries), can we assume that the world is heading towards World War III?
  Should I build iOS or Android apps first?	Should people choose Android or iOS first to build their App?	How much more effort is it to build your app on both iOS and Android?

• Loss: SpladeLoss with these parameters:

  {
      "loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score')",
      "lambda_corpus": 3e-05,
      "lambda_query": 5e-05
  }
  

Evaluation Dataset

quora-duplicates

• Dataset: quora-duplicates at 451a485
• Size: 1,000 evaluation samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 6 tokens mean: 14.05 tokens max: 40 tokens	min: 6 tokens mean: 14.14 tokens max: 44 tokens	min: 6 tokens mean: 14.56 tokens max: 60 tokens

• Samples:

  anchor	positive	negative
  What happens if we use petrol in diesel vehicles?	Why can't we use petrol in diesel?	Why are diesel engines noisier than petrol engines?
  Why is Saltwater taffy candy imported in Switzerland?	Why is Saltwater taffy candy imported in Laos?	Is salt a consumer product?
  Which is your favourite film in 2016?	What movie is the best movie of 2016?	What will the best movie of 2017 be?

• Loss: SpladeLoss with these parameters:

  {
      "loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score')",
      "lambda_corpus": 3e-05,
      "lambda_query": 5e-05
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 12
• per_device_eval_batch_size: 12
• learning_rate: 2e-05
• num_train_epochs: 1
• bf16: True
• load_best_model_at_end: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 12
• per_device_eval_batch_size: 12
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• tp_size: 0
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	Validation Loss	quora_duplicates_dev_max_ap	NanoMSMARCO_dot_ndcg@10	NanoNQ_dot_ndcg@10	NanoNFCorpus_dot_ndcg@10	NanoQuoraRetrieval_dot_ndcg@10	NanoBEIR_mean_dot_ndcg@10	NanoClimateFEVER_dot_ndcg@10	NanoDBPedia_dot_ndcg@10	NanoFEVER_dot_ndcg@10	NanoFiQA2018_dot_ndcg@10	NanoHotpotQA_dot_ndcg@10	NanoSCIDOCS_dot_ndcg@10	NanoArguAna_dot_ndcg@10	NanoSciFact_dot_ndcg@10	NanoTouche2020_dot_ndcg@10
0.0242	200	8.3389	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0485	400	0.4397	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0727	600	0.3737	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0970	800	0.2666	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1212	1000	0.288	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1455	1200	0.1977	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1697	1400	0.2707	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1939	1600	0.1951	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2	1650	-	0.1669	0.6472	0.3052	0.2793	0.1711	0.9281	0.4209	-	-	-	-	-	-	-	-	-
0.2182	1800	0.2178	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2424	2000	0.2174	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2667	2200	0.1832	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2909	2400	0.1879	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3152	2600	0.1723	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3394	2800	0.1543	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3636	3000	0.1559	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3879	3200	0.1575	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4	3300	-	0.1149	0.6749	0.3894	0.4467	0.2360	0.9292	0.5003	-	-	-	-	-	-	-	-	-
0.4121	3400	0.1395	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4364	3600	0.1596	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4606	3800	0.1595	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4848	4000	0.1211	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5091	4200	0.1163	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5333	4400	0.1182	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5576	4600	0.1337	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5818	4800	0.1362	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6	4950	-	0.1001	0.6802	0.4093	0.4269	0.2341	0.9365	0.5017	-	-	-	-	-	-	-	-	-
0.6061	5000	0.1112	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6303	5200	0.1064	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6545	5400	0.119	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6788	5600	0.1077	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7030	5800	0.1398	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7273	6000	0.09	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7515	6200	0.0903	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7758	6400	0.1082	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8	6600	0.1122	0.0901	0.6941	0.4451	0.4757	0.2542	0.9411	0.5290	-	-	-	-	-	-	-	-	-
0.8242	6800	0.0708	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8485	7000	0.1291	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8727	7200	0.1165	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8970	7400	0.0735	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9212	7600	0.0775	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9455	7800	0.0945	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9697	8000	0.0912	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9939	8200	0.104	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
1.0	8250	-	0.0686	0.6888	0.4532	0.4828	0.2467	0.9467	0.5324	-	-	-	-	-	-	-	-	-
-1	-1	-	-	-	0.4532	0.4828	0.2467	0.9467	0.4685	0.1845	0.5071	0.7070	0.2881	0.7143	0.3029	0.2782	0.6021	0.3762

• The bold row denotes the saved checkpoint.

Environmental Impact

Carbon emissions were measured using CodeCarbon.

• Energy Consumed: 0.025 kWh
• Carbon Emitted: 0.001 kg of CO2
• Hours Used: 0.222 hours

Training Hardware

• On Cloud: No
• GPU Model: 1 x NVIDIA GeForce RTX 3070 Ti Laptop GPU
• CPU Model: AMD Ryzen 9 6900HX with Radeon Graphics
• RAM Size: 30.61 GB

Framework Versions

• Python: 3.12.9
• Sentence Transformers: 4.2.0.dev0
• Transformers: 4.50.3
• PyTorch: 2.6.0+cu124
• Accelerate: 1.6.0
• Datasets: 3.5.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

SpladeLoss

@misc{formal2022distillationhardnegativesampling,
      title={From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective},
      author={Thibault Formal and Carlos Lassance and Benjamin Piwowarski and Stéphane Clinchant},
      year={2022},
      eprint={2205.04733},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2205.04733},
}

SparseMultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

FlopsLoss

@article{paria2020minimizing,
    title={Minimizing flops to learn efficient sparse representations},
    author={Paria, Biswajit and Yeh, Chih-Kuan and Yen, Ian EH and Xu, Ning and Ravikumar, Pradeep and P{'o}czos, Barnab{'a}s},
    journal={arXiv preprint arXiv:2004.05665},
    year={2020}
    }