README.md

metadata

base_model: agentlans/deberta-v3-base-zyda-2
library_name: sentence-transformers
pipeline_tag: sentence-similarity
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - generated_from_trainer
  - dataset_size:1079040
  - loss:CoSENTLoss
widget:
  - source_sentence: Can you describe the steps involved in reconciling a bank statement?
    sentences:
      - How can we reduce delay in wowza live stream?
      - Did India meet Abdul Kalam's 2020 vision so far?
      - >-
        Why US people supporting a old woman president despite a lots of the
        young men?
  - source_sentence: He also met Theodoros Kolokotronis , whom he began to admire later .
    sentences:
      - How do you make pancakes without the batter?
      - >-
        The original team was produced by the writer Sal Buscema and artist Roy
        Thomas in `` The Avengers '' # 71 ( December 1969 ) .
      - He also met Theodoros Kolokotronis , whom he later came to admire .
  - source_sentence: >-
      To enable phone networks to monitor usage, Carrier IQ is a common
      pre-installed feature on U.S. handsets.
    sentences:
      - What does the Pope do?
      - >-
        Carrier IQ is installed as standard on many U.S. handsets to allow phone
        networks to monitor usage.
      - Is the risk of disease higher with dead rats than with live ones?
  - source_sentence: >-
      Many other individuals, communities, and countries perceive "black" as a
      derogatory, outdated, reductionist, or otherwise unrepresentative label,
      and as a result, it is neither used nor defined.
    sentences:
      - Have you been friendzoned?
      - >-
        According to Sandoval, Raul Jr. fled from the car but was allegedly shot
        in the back.
      - Do you lose nutrients & calories when you have diarrhea?
  - source_sentence: >-
      While holidaying with her partner and his son, Mrs Searle, an
      administrator, stated: 'If I had been informed, I would have instructed
      her to desist.'
    sentences:
      - >-
        Mechanical digestion involves breaking down large food pieces into
        smaller ones that can be acted upon by digestive enzymes.
      - >-
        From prehistoric times, the region was home to tribes such as the
        Koilis, Halis, Dagis, Dhaugris, Dasa, Khasas, Kinnars, and Kirats.
      - >-
        Mrs Searle, an administrator, who was on holiday with her partner and
        his son, added: 'If I had known I would have told her to stop.
license: mit
datasets:
  - agentlans/sentence-paraphrases
language:
  - en

SentenceTransformer based on agentlans/deberta-v3-base-zyda-2

This is a sentence-transformers model finetuned from agentlans/deberta-v3-base-zyda-2. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

This model has been finetuned on the agentlans/sentence-paraphrases dataset.

• Text were truncated to 500 characters to save VRAM.
• Two negative examples were generated for each row in the dataset.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: agentlans/deberta-v3-base-zyda-2
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 tokens
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: DebertaV2Model 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("agentlans/deberta-v3-base-zyda-2-v2")
# Run inference
sentences = [
    "While holidaying with her partner and his son, Mrs Searle, an administrator, stated: 'If I had been informed, I would have instructed her to desist.'",
    "Mrs Searle, an administrator, who was on holiday with her partner and his son, added: 'If I had known I would have told her to stop.",
    'Mechanical digestion involves breaking down large food pieces into smaller ones that can be acted upon by digestive enzymes.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Unnamed Dataset

• Size: 1,079,040 training samples
• Columns: sentence_0, sentence_1, and label
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1	label
  type	string	string	float
  details	min: 6 tokens mean: 22.64 tokens max: 75 tokens	min: 6 tokens mean: 20.55 tokens max: 71 tokens	min: 0.0 mean: 0.33 max: 1.0

• Samples:

  sentence_0	sentence_1	label
  Mrs Alper's late spouse, Sam, had an extensive art collection that was also destroyed.	The art collection of Mrs Alper's deceased husband, Sam, was also destroyed.	1.0
  Is the 1 x 1 Rubik's cube intended to be comical?	What is the least fuve digit decimal number with five significant figures?	0.0
  What is the Prasoon Joshi song that you enjoy the most and what is the rationale behind your preference?	What are some interesting facts about human brain?	0.0

• Loss: CoSENTLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "pairwise_cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 3
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• eval_use_gather_object: False
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin

Training Logs

Click to expand

Epoch	Step	Training Loss
0.0074	500	1.1791
0.0148	1000	0.4056
0.0222	1500	0.2729
0.0297	2000	0.1775
0.0371	2500	0.1513
0.0445	3000	0.1329
0.0519	3500	0.1503
0.0593	4000	0.1325
0.0667	4500	0.1187
0.0741	5000	0.1012
0.0816	5500	0.1504
0.0890	6000	0.1161
0.0964	6500	0.1194
0.1038	7000	0.1172
0.1112	7500	0.1391
0.1186	8000	0.1056
0.1260	8500	0.0697
0.1335	9000	0.1157
0.1409	9500	0.1009
0.1483	10000	0.0996
0.1557	10500	0.1076
0.1631	11000	0.1057
0.1705	11500	0.084
0.1779	12000	0.0711
0.1853	12500	0.1026
0.1928	13000	0.0841
0.2002	13500	0.1149
0.2076	14000	0.0871
0.2150	14500	0.1201
0.2224	15000	0.0851
0.2298	15500	0.073
0.2372	16000	0.0893
0.2447	16500	0.1083
0.2521	17000	0.0824
0.2595	17500	0.0721
0.2669	18000	0.056
0.2743	18500	0.1062
0.2817	19000	0.094
0.2891	19500	0.0887
0.2966	20000	0.0756
0.3040	20500	0.0932
0.3114	21000	0.0718
0.3188	21500	0.067
0.3262	22000	0.0792
0.3336	22500	0.0639
0.3410	23000	0.0987
0.3485	23500	0.0682
0.3559	24000	0.0769
0.3633	24500	0.1255
0.3707	25000	0.0929
0.3781	25500	0.0948
0.3855	26000	0.0983
0.3929	26500	0.1228
0.4004	27000	0.1028
0.4078	27500	0.0856
0.4152	28000	0.1173
0.4226	28500	0.0718
0.4300	29000	0.0964
0.4374	29500	0.0844
0.4448	30000	0.0871
0.4523	30500	0.0943
0.4597	31000	0.1353
0.4671	31500	0.0634
0.4745	32000	0.1263
0.4819	32500	0.1098
0.4893	33000	0.098
0.4967	33500	0.1182
0.5042	34000	0.0818
0.5116	34500	0.1207
0.5190	35000	0.097
0.5264	35500	0.0904
0.5338	36000	0.1011
0.5412	36500	0.1323
0.5486	37000	0.0699
0.5560	37500	0.0803
0.5635	38000	0.0737
0.5709	38500	0.0798
0.5783	39000	0.1348
0.5857	39500	0.0914
0.5931	40000	0.0654
0.6005	40500	0.0729
0.6079	41000	0.0737
0.6154	41500	0.1018
0.6228	42000	0.0809
0.6302	42500	0.0906
0.6376	43000	0.0955
0.6450	43500	0.0759
0.6524	44000	0.1055
0.6598	44500	0.0924
0.6673	45000	0.1027
0.6747	45500	0.0826
0.6821	46000	0.0763
0.6895	46500	0.1035
0.6969	47000	0.0969
0.7043	47500	0.0714
0.7117	48000	0.0826
0.7192	48500	0.0923
0.7266	49000	0.0651
0.7340	49500	0.0901
0.7414	50000	0.1001
0.7488	50500	0.0961
0.7562	51000	0.085
0.7636	51500	0.0633
0.7711	52000	0.0879
0.7785	52500	0.0717
0.7859	53000	0.0589
0.7933	53500	0.0822
0.8007	54000	0.0857
0.8081	54500	0.0994
0.8155	55000	0.0752
0.8230	55500	0.0965
0.8304	56000	0.0776
0.8378	56500	0.089
0.8452	57000	0.0638
0.8526	57500	0.111
0.8600	58000	0.072
0.8674	58500	0.0755
0.8749	59000	0.096
0.8823	59500	0.1205
0.8897	60000	0.0728
0.8971	60500	0.1014
0.9045	61000	0.0987
0.9119	61500	0.0756
0.9193	62000	0.0746
0.9267	62500	0.0992
0.9342	63000	0.0961
0.9416	63500	0.0861
0.9490	64000	0.0723
0.9564	64500	0.0765
0.9638	65000	0.0859
0.9712	65500	0.0839
0.9786	66000	0.085
0.9861	66500	0.1136
0.9935	67000	0.0735
1.0009	67500	0.0791
1.0083	68000	0.0747
1.0157	68500	0.1148
1.0231	69000	0.1022
1.0305	69500	0.0501
1.0380	70000	0.0735
1.0454	70500	0.0734
1.0528	71000	0.0705
1.0602	71500	0.0854
1.0676	72000	0.0858
1.0750	72500	0.0453
1.0824	73000	0.0768
1.0899	73500	0.0949
1.0973	74000	0.1028
1.1047	74500	0.1192
1.1121	75000	0.0754
1.1195	75500	0.0818
1.1269	76000	0.0662
1.1343	76500	0.0659
1.1418	77000	0.0913
1.1492	77500	0.071
1.1566	78000	0.0682
1.1640	78500	0.0858
1.1714	79000	0.0781
1.1788	79500	0.0782
1.1862	80000	0.0722
1.1937	80500	0.0686
1.2011	81000	0.0751
1.2085	81500	0.0611
1.2159	82000	0.1114
1.2233	82500	0.0856
1.2307	83000	0.0789
1.2381	83500	0.0932
1.2456	84000	0.0873
1.2530	84500	0.0691
1.2604	85000	0.0609
1.2678	85500	0.0568
1.2752	86000	0.0797
1.2826	86500	0.0968
1.2900	87000	0.1113
1.2974	87500	0.0936
1.3049	88000	0.091
1.3123	88500	0.0482
1.3197	89000	0.0898
1.3271	89500	0.0766
1.3345	90000	0.0859
1.3419	90500	0.0851
1.3493	91000	0.0695
1.3568	91500	0.0881
1.3642	92000	0.1095
1.3716	92500	0.0676
1.3790	93000	0.094
1.3864	93500	0.0986
1.3938	94000	0.0844
1.4012	94500	0.0929
1.4087	95000	0.0783
1.4161	95500	0.0963
1.4235	96000	0.1003
1.4309	96500	0.0817
1.4383	97000	0.0754
1.4457	97500	0.0858
1.4531	98000	0.0746
1.4606	98500	0.0916
1.4680	99000	0.0738
1.4754	99500	0.0778
1.4828	100000	0.0897
1.4902	100500	0.1028
1.4976	101000	0.0914
1.5050	101500	0.0771
1.5125	102000	0.0716
1.5199	102500	0.1127
1.5273	103000	0.0785
1.5347	103500	0.0868
1.5421	104000	0.118
1.5495	104500	0.0838
1.5569	105000	0.0963
1.5644	105500	0.0579
1.5718	106000	0.0738
1.5792	106500	0.1182
1.5866	107000	0.1025
1.5940	107500	0.0747
1.6014	108000	0.0604
1.6088	108500	0.0607
1.6163	109000	0.0794
1.6237	109500	0.0793
1.6311	110000	0.084
1.6385	110500	0.1315
1.6459	111000	0.0782
1.6533	111500	0.0724
1.6607	112000	0.0864
1.6681	112500	0.0791
1.6756	113000	0.0772
1.6830	113500	0.0923
1.6904	114000	0.0897
1.6978	114500	0.0833
1.7052	115000	0.0819
1.7126	115500	0.0695
1.7200	116000	0.0919
1.7275	116500	0.074
1.7349	117000	0.0893
1.7423	117500	0.1042
1.7497	118000	0.0648
1.7571	118500	0.0965
1.7645	119000	0.0634
1.7719	119500	0.0705
1.7794	120000	0.0928
1.7868	120500	0.0817
1.7942	121000	0.0756
1.8016	121500	0.0769
1.8090	122000	0.0877
1.8164	122500	0.0697
1.8238	123000	0.1095
1.8313	123500	0.1056
1.8387	124000	0.0931
1.8461	124500	0.0772
1.8535	125000	0.0867
1.8609	125500	0.0706
1.8683	126000	0.091
1.8757	126500	0.0751
1.8832	127000	0.0732
1.8906	127500	0.0615
1.8980	128000	0.0947
1.9054	128500	0.1067
1.9128	129000	0.0692
1.9202	129500	0.064
1.9276	130000	0.109
1.9351	130500	0.0843
1.9425	131000	0.0897
1.9499	131500	0.0999
1.9573	132000	0.0866
1.9647	132500	0.083
1.9721	133000	0.0859
1.9795	133500	0.0761
1.9870	134000	0.1089
1.9944	134500	0.1053
2.0018	135000	0.0581
2.0092	135500	0.0781
2.0166	136000	0.1286
2.0240	136500	0.1309
2.0314	137000	0.0476
2.0388	137500	0.0695
2.0463	138000	0.0746
2.0537	138500	0.063
2.0611	139000	0.0816
2.0685	139500	0.0821
2.0759	140000	0.0671
2.0833	140500	0.0865
2.0907	141000	0.0638
2.0982	141500	0.0803
2.1056	142000	0.0872
2.1130	142500	0.0968
2.1204	143000	0.1052
2.1278	143500	0.0554
2.1352	144000	0.1057
2.1426	144500	0.0565
2.1501	145000	0.0798
2.1575	145500	0.098
2.1649	146000	0.0832
2.1723	146500	0.067
2.1797	147000	0.0604
2.1871	147500	0.0808
2.1945	148000	0.0921
2.2020	148500	0.0767
2.2094	149000	0.0856
2.2168	149500	0.0966
2.2242	150000	0.0643
2.2316	150500	0.068
2.2390	151000	0.1007
2.2464	151500	0.0765
2.2539	152000	0.0662
2.2613	152500	0.067
2.2687	153000	0.0547
2.2761	153500	0.0833
2.2835	154000	0.1087
2.2909	154500	0.0868
2.2983	155000	0.0836
2.3058	155500	0.063
2.3132	156000	0.0459
2.3206	156500	0.0771
2.3280	157000	0.0856
2.3354	157500	0.0513
2.3428	158000	0.0584
2.3502	158500	0.0817
2.3577	159000	0.0948
2.3651	159500	0.0945
2.3725	160000	0.0746
2.3799	160500	0.0923
2.3873	161000	0.0933
2.3947	161500	0.08
2.4021	162000	0.1343
2.4095	162500	0.0699
2.4170	163000	0.0861
2.4244	163500	0.0811
2.4318	164000	0.0671
2.4392	164500	0.0877
2.4466	165000	0.0741
2.4540	165500	0.0834
2.4614	166000	0.0966
2.4689	166500	0.0739
2.4763	167000	0.0916
2.4837	167500	0.087
2.4911	168000	0.0974
2.4985	168500	0.0876
2.5059	169000	0.0954
2.5133	169500	0.0936
2.5208	170000	0.0866
2.5282	170500	0.0789
2.5356	171000	0.0932
2.5430	171500	0.094
2.5504	172000	0.0897
2.5578	172500	0.08
2.5652	173000	0.0664
2.5727	173500	0.0807
2.5801	174000	0.1157
2.5875	174500	0.1272
2.5949	175000	0.0843
2.6023	175500	0.067
2.6097	176000	0.084
2.6171	176500	0.0848
2.6246	177000	0.0805
2.6320	177500	0.0828
2.6394	178000	0.1059
2.6468	178500	0.0912
2.6542	179000	0.0683
2.6616	179500	0.0754
2.6690	180000	0.0844
2.6765	180500	0.0824
2.6839	181000	0.0729
2.6913	181500	0.0771
2.6987	182000	0.0993
2.7061	182500	0.0895
2.7135	183000	0.0706
2.7209	183500	0.0731
2.7284	184000	0.0682
2.7358	184500	0.0775
2.7432	185000	0.0956
2.7506	185500	0.0801
2.7580	186000	0.106
2.7654	186500	0.079
2.7728	187000	0.0636
2.7802	187500	0.0819
2.7877	188000	0.0763
2.7951	188500	0.0963
2.8025	189000	0.0714
2.8099	189500	0.0721
2.8173	190000	0.0599
2.8247	190500	0.0998
2.8321	191000	0.0629
2.8396	191500	0.1043
2.8470	192000	0.0973
2.8544	192500	0.1069
2.8618	193000	0.0557
2.8692	193500	0.07
2.8766	194000	0.1252
2.8840	194500	0.0801
2.8915	195000	0.0759
2.8989	195500	0.105
2.9063	196000	0.0806
2.9137	196500	0.0737
2.9211	197000	0.0841
2.9285	197500	0.1031
2.9359	198000	0.0828
2.9434	198500	0.0842
2.9508	199000	0.083
2.9582	199500	0.0745
2.9656	200000	0.0946
2.9730	200500	0.0802
2.9804	201000	0.0847
2.9878	201500	0.0991
2.9953	202000	0.0747

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 3.1.0
• Transformers: 4.43.3
• PyTorch: 2.3.0+cu121
• Accelerate: 0.33.0
• Datasets: 3.2.0
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CoSENTLoss

@online{kexuefm-8847,
    title={CoSENT: A more efficient sentence vector scheme than Sentence-BERT},
    author={Su Jianlin},
    year={2022},
    month={Jan},
    url={https://kexue.fm/archives/8847},
}