CATIE-AQ/CSR_Sparse_Encoder_camembert-large_STS

CSR Sparse Encoder

This is a CSR Sparse Encoder model finetuned from almanach/camembert-large on the french_sts dataset using the sentence-transformers library. It maps sentences & paragraphs to a 4096-dimensional sparse vector space with 256 maximum active dimensions and can be used for semantic search and sparse retrieval.

Model Details

Model Description

• Model Type: CSR Sparse Encoder
• Base model: almanach/camembert-large
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 4096 dimensions (trained with 256 maximum active dimensions)
• Similarity Function: Cosine Similarity
• Training Dataset:
  • french_sts
• Language: fr

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Sparse Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sparse Encoders on Hugging Face

Full Model Architecture

SparseEncoder(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'CamembertModel'})
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): SparseAutoEncoder({'input_dim': 1024, 'hidden_dim': 4096, 'k': 256, 'k_aux': 512, 'normalize': False, 'dead_threshold': 30})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SparseEncoder

# Download from the 🤗 Hub
model = SparseEncoder("bourdoiscatie/SPLADE_camembert-large_STS")
# Run inference
sentences = [
    "Oui, je peux vous dire d'après mon expérience personnelle qu'ils ont certainement sifflé.",
    "Il est vrai que les bombes de la Seconde Guerre mondiale faisaient un bruit de sifflet lorsqu'elles tombaient.",
    "J'envisage de dépenser les 48 dollars par mois pour le système GTD (Getting things done) annoncé par David Allen.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 4096]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.3673, 0.2794],
#         [0.3673, 1.0000, 0.2023],
#         [0.2794, 0.2023, 1.0000]])

Evaluation

Metrics

Semantic Similarity

• Datasets: sts-dev and sts-test
• Evaluated with SparseEmbeddingSimilarityEvaluator

Metric	sts-dev	sts-test
pearson_cosine	0.7307	0.7537
spearman_cosine	0.723	0.7256
active_dims	239.0452	229.7225
sparsity_ratio	0.9416	0.9439

Training Details

Training Dataset

french_sts

• Dataset: french_sts at 47128cc
• Size: 12,227 training samples
• Columns: sentence1, sentence2, and score
• Approximate statistics based on the first 1000 samples:

  	sentence1	sentence2	score
  type	string	string	float
  details	min: 6 tokens mean: 11.75 tokens max: 30 tokens	min: 6 tokens mean: 11.79 tokens max: 35 tokens	min: 0.0 mean: 0.44 max: 1.0

• Samples:

  sentence1	sentence2	score
  Un avion est en train de décoller.	Un avion est en train de décoller.	1.0
  Un homme est en train de fumer.	Un homme fait du patinage.	0.10000000149011612
  Une personne jette un chat au plafond.	Une personne jette un chat au plafond.	1.0

• Loss: SpladeLoss with these parameters:

  {
      "loss": "SparseCosineSimilarityLoss(loss_fct='torch.nn.modules.loss.MSELoss')",
      "document_regularizer_weight": 0.003
  }
  

Evaluation Dataset

french_sts

• Dataset: french_sts at 47128cc
• Size: 3,526 evaluation samples
• Columns: sentence1, sentence2, and score
• Approximate statistics based on the first 1000 samples:

  	sentence1	sentence2	score
  type	string	string	float
  details	min: 6 tokens mean: 19.13 tokens max: 50 tokens	min: 6 tokens mean: 19.05 tokens max: 55 tokens	min: 0.0 mean: 0.43 max: 1.0

• Samples:

  sentence1	sentence2	score
  Un homme avec un casque de sécurité est en train de danser.	Un homme portant un casque de sécurité est en train de danser.	1.0
  Un jeune enfant monte à cheval.	Un enfant monte à cheval.	0.949999988079071
  Un homme donne une souris à un serpent.	L'homme donne une souris au serpent.	1.0

• Loss: SpladeLoss with these parameters:

  {
      "loss": "SparseCosineSimilarityLoss(loss_fct='torch.nn.modules.loss.MSELoss')",
      "document_regularizer_weight": 0.003
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• bf16: True

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 3
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• tp_size: 0
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	sts-dev_spearman_cosine	sts-test_spearman_cosine
-1	-1	-	-	0.4890	-
0.1307	100	0.0458	-	-	-
0.2614	200	0.0447	-	-	-
0.3922	300	0.0468	-	-	-
0.5229	400	0.0416	-	-	-
0.6536	500	0.0398	-	-	-
0.7843	600	0.0397	-	-	-
0.9150	700	0.0398	-	-	-
1.0	765	-	0.0417	0.6801	-
1.0458	800	0.0368	-	-	-
1.1765	900	0.0296	-	-	-
1.3072	1000	0.0288	-	-	-
1.4379	1100	0.0285	-	-	-
1.5686	1200	0.0264	-	-	-
1.6993	1300	0.0251	-	-	-
1.8301	1400	0.0256	-	-	-
1.9608	1500	0.0253	-	-	-
2.0	1530	-	0.0368	0.7083	-
2.0915	1600	0.0197	-	-	-
2.2222	1700	0.0151	-	-	-
2.3529	1800	0.0156	-	-	-
2.4837	1900	0.0155	-	-	-
2.6144	2000	0.0141	-	-	-
2.7451	2100	0.0134	-	-	-
2.8758	2200	0.0137	-	-	-
3.0	2295	-	0.0352	0.7230	-
-1	-1	-	-	-	0.7256

Framework Versions

• Python: 3.12.3
• Sentence Transformers: 5.0.0
• Transformers: 4.51.3
• PyTorch: 2.6.0+cu124
• Accelerate: 1.6.0
• Datasets: 2.16.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

SpladeLoss

@misc{formal2022distillationhardnegativesampling,
      title={From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective},
      author={Thibault Formal and Carlos Lassance and Benjamin Piwowarski and Stéphane Clinchant},
      year={2022},
      eprint={2205.04733},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2205.04733},
}

FlopsLoss

@article{paria2020minimizing,
    title={Minimizing flops to learn efficient sparse representations},
    author={Paria, Biswajit and Yeh, Chih-Kuan and Yen, Ian EH and Xu, Ning and Ravikumar, Pradeep and P{'o}czos, Barnab{'a}s},
    journal={arXiv preprint arXiv:2004.05665},
    year={2020}
}