Hugging Face's logo

AhmedZaky1
/
DIMI-embedding-v4

Sentence Similarity
sentence-transformers
Safetensors
Arabic
English
new
feature-extraction
Generated from Trainer
dataset_size:34436
loss:MatryoshkaLoss
loss:CoSENTLoss
custom_code
text-embeddings-inference
Model card Files Community
AhmedZaky1 commited on
Commit
6bcaa89
·
verified ·
1 Parent(s): 0718c82

Add new SentenceTransformer model

Browse files
Files changed (11) hide show
  1. .gitattributes +1 -0
  2. 1_Pooling/config.json +10 -0
  3. README.md +626 -0
  4. config.json +49 -0
  5. config_sentence_transformers.json +10 -0
  6. model.safetensors +3 -0
  7. modules.json +20 -0
  8. sentence_bert_config.json +4 -0
  9. special_tokens_map.json +51 -0
  10. tokenizer.json +3 -0
  11. tokenizer_config.json +62 -0
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
1_Pooling/config.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "word_embedding_dimension": 768,
3
+ "pooling_mode_cls_token": true,
4
+ "pooling_mode_mean_tokens": false,
5
+ "pooling_mode_max_tokens": false,
6
+ "pooling_mode_mean_sqrt_len_tokens": false,
7
+ "pooling_mode_weightedmean_tokens": false,
8
+ "pooling_mode_lasttoken": false,
9
+ "include_prompt": true
10
+ }
README.md ADDED
@@ -0,0 +1,626 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - ar
4
+ - en
5
+ tags:
6
+ - sentence-transformers
7
+ - sentence-similarity
8
+ - feature-extraction
9
+ - generated_from_trainer
10
+ - dataset_size:34436
11
+ - loss:MatryoshkaLoss
12
+ - loss:CoSENTLoss
13
+ base_model: AhmedZaky1/DIMI-embedding-v2
14
+ widget:
15
+ - source_sentence: الرجل يركب حصاناً
16
+ sentences:
17
+ - رجل يُبث الجبن الممزق على البيتزا
18
+ - ar-ar
19
+ - رجل يركب حصاناً
20
+ - source_sentence: المرأة تقلي لحم خنزير مشوي
21
+ sentences:
22
+ - ar-ar
23
+ - امرأة تطبخ لحم خنزير مخبوز
24
+ - طائرة طيران تقلع
25
+ - source_sentence: امرأة تحمل في ذراعها طفل كنغر
26
+ sentences:
27
+ - امرأة تعزف على الغيتار
28
+ - ar-ar
29
+ - امرأة تحمل و تحمل طفل كنغر
30
+ - source_sentence: رجل يعزف على الناي
31
+ sentences:
32
+ - طائرة ستقلع
33
+ - ar-ar
34
+ - رجل يعزف على فرقة الخيزران
35
+ - source_sentence: ثلاثة رجال يلعبون الشطرنج.
36
+ sentences:
37
+ - رجلين يلعبان الشطرنج
38
+ - بعض الرجال يقاتلون
39
+ - ar-ar
40
+ datasets:
41
+ - silma-ai/silma-arabic-english-sts-dataset-v1.0
42
+ pipeline_tag: sentence-similarity
43
+ library_name: sentence-transformers
44
+ metrics:
45
+ - pearson_cosine
46
+ - spearman_cosine
47
+ model-index:
48
+ - name: SentenceTransformer based on AhmedZaky1/DIMI-embedding-v2
49
+ results:
50
+ - task:
51
+ type: semantic-similarity
52
+ name: Semantic Similarity
53
+ dataset:
54
+ name: silma sts dev 768
55
+ type: silma-sts-dev-768
56
+ metrics:
57
+ - type: pearson_cosine
58
+ value: 0.8894298077237747
59
+ name: Pearson Cosine
60
+ - type: spearman_cosine
61
+ value: 0.8357984695231979
62
+ name: Spearman Cosine
63
+ - task:
64
+ type: semantic-similarity
65
+ name: Semantic Similarity
66
+ dataset:
67
+ name: silma sts dev 512
68
+ type: silma-sts-dev-512
69
+ metrics:
70
+ - type: pearson_cosine
71
+ value: 0.8958835653694187
72
+ name: Pearson Cosine
73
+ - type: spearman_cosine
74
+ value: 0.8394578198917563
75
+ name: Spearman Cosine
76
+ - task:
77
+ type: semantic-similarity
78
+ name: Semantic Similarity
79
+ dataset:
80
+ name: silma sts dev 256
81
+ type: silma-sts-dev-256
82
+ metrics:
83
+ - type: pearson_cosine
84
+ value: 0.9078743376141943
85
+ name: Pearson Cosine
86
+ - type: spearman_cosine
87
+ value: 0.8470163055535588
88
+ name: Spearman Cosine
89
+ - task:
90
+ type: semantic-similarity
91
+ name: Semantic Similarity
92
+ dataset:
93
+ name: silma sts dev 128
94
+ type: silma-sts-dev-128
95
+ metrics:
96
+ - type: pearson_cosine
97
+ value: 0.9181556833949818
98
+ name: Pearson Cosine
99
+ - type: spearman_cosine
100
+ value: 0.856188415278301
101
+ name: Spearman Cosine
102
+ - task:
103
+ type: semantic-similarity
104
+ name: Semantic Similarity
105
+ dataset:
106
+ name: silma sts dev 64
107
+ type: silma-sts-dev-64
108
+ metrics:
109
+ - type: pearson_cosine
110
+ value: 0.9066219844975816
111
+ name: Pearson Cosine
112
+ - type: spearman_cosine
113
+ value: 0.8434430083292863
114
+ name: Spearman Cosine
115
+ - task:
116
+ type: semantic-similarity
117
+ name: Semantic Similarity
118
+ dataset:
119
+ name: sts17 ar test 768
120
+ type: sts17-ar-test-768
121
+ metrics:
122
+ - type: pearson_cosine
123
+ value: 0.8205269118955641
124
+ name: Pearson Cosine
125
+ - type: spearman_cosine
126
+ value: 0.8258003312254673
127
+ name: Spearman Cosine
128
+ - task:
129
+ type: semantic-similarity
130
+ name: Semantic Similarity
131
+ dataset:
132
+ name: sts17 ar test 512
133
+ type: sts17-ar-test-512
134
+ metrics:
135
+ - type: pearson_cosine
136
+ value: 0.8193403796404517
137
+ name: Pearson Cosine
138
+ - type: spearman_cosine
139
+ value: 0.8226611918447921
140
+ name: Spearman Cosine
141
+ - task:
142
+ type: semantic-similarity
143
+ name: Semantic Similarity
144
+ dataset:
145
+ name: sts17 ar test 256
146
+ type: sts17-ar-test-256
147
+ metrics:
148
+ - type: pearson_cosine
149
+ value: 0.8190666923783347
150
+ name: Pearson Cosine
151
+ - type: spearman_cosine
152
+ value: 0.8245760514866052
153
+ name: Spearman Cosine
154
+ - task:
155
+ type: semantic-similarity
156
+ name: Semantic Similarity
157
+ dataset:
158
+ name: sts17 ar test 128
159
+ type: sts17-ar-test-128
160
+ metrics:
161
+ - type: pearson_cosine
162
+ value: 0.8114629254813825
163
+ name: Pearson Cosine
164
+ - type: spearman_cosine
165
+ value: 0.8183273799928091
166
+ name: Spearman Cosine
167
+ - task:
168
+ type: semantic-similarity
169
+ name: Semantic Similarity
170
+ dataset:
171
+ name: sts17 ar test 64
172
+ type: sts17-ar-test-64
173
+ metrics:
174
+ - type: pearson_cosine
175
+ value: 0.796172574267003
176
+ name: Pearson Cosine
177
+ - type: spearman_cosine
178
+ value: 0.8077141358495715
179
+ name: Spearman Cosine
180
+ ---
181
+
182
+ # SentenceTransformer based on AhmedZaky1/DIMI-embedding-v2
183
+
184
+ This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [AhmedZaky1/DIMI-embedding-v2](https://huggingface.co/AhmedZaky1/DIMI-embedding-v2) on the [silma-arabic-english-sts-dataset-v1.0](https://huggingface.co/datasets/silma-ai/silma-arabic-english-sts-dataset-v1.0) dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
185
+
186
+ ## Model Details
187
+
188
+ ### Model Description
189
+ - **Model Type:** Sentence Transformer
190
+ - **Base model:** [AhmedZaky1/DIMI-embedding-v2](https://huggingface.co/AhmedZaky1/DIMI-embedding-v2) <!-- at revision d4a6e4faaea9d9a2ad374fea48b093946166e753 -->
191
+ - **Maximum Sequence Length:** 8192 tokens
192
+ - **Output Dimensionality:** 768 dimensions
193
+ - **Similarity Function:** Cosine Similarity
194
+ - **Training Dataset:**
195
+ - [silma-arabic-english-sts-dataset-v1.0](https://huggingface.co/datasets/silma-ai/silma-arabic-english-sts-dataset-v1.0)
196
+ - **Languages:** ar, en
197
+ <!-- - **License:** Unknown -->
198
+
199
+ ### Model Sources
200
+
201
+ - **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
202
+ - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
203
+ - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)
204
+
205
+ ### Full Model Architecture
206
+
207
+ ```
208
+ SentenceTransformer(
209
+ (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: NewModel
210
+ (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
211
+ (2): Normalize()
212
+ )
213
+ ```
214
+
215
+ ## Usage
216
+
217
+ ### Direct Usage (Sentence Transformers)
218
+
219
+ First install the Sentence Transformers library:
220
+
221
+ ```bash
222
+ pip install -U sentence-transformers
223
+ ```
224
+
225
+ Then you can load this model and run inference.
226
+ ```python
227
+ from sentence_transformers import SentenceTransformer
228
+
229
+ # Download from the 🤗 Hub
230
+ model = SentenceTransformer("AhmedZaky1/DIMI-embedding-v2-silma-sts-matryoshka")
231
+ # Run inference
232
+ sentences = [
233
+ 'ثلاثة رجال يلعبون الشطرنج.',
234
+ 'رجلين يلعبان الشطرنج',
235
+ 'ar-ar',
236
+ ]
237
+ embeddings = model.encode(sentences)
238
+ print(embeddings.shape)
239
+ # [3, 768]
240
+
241
+ # Get the similarity scores for the embeddings
242
+ similarities = model.similarity(embeddings, embeddings)
243
+ print(similarities.shape)
244
+ # [3, 3]
245
+ ```
246
+
247
+ <!--
248
+ ### Direct Usage (Transformers)
249
+
250
+ <details><summary>Click to see the direct usage in Transformers</summary>
251
+
252
+ </details>
253
+ -->
254
+
255
+ <!--
256
+ ### Downstream Usage (Sentence Transformers)
257
+
258
+ You can finetune this model on your own dataset.
259
+
260
+ <details><summary>Click to expand</summary>
261
+
262
+ </details>
263
+ -->
264
+
265
+ <!--
266
+ ### Out-of-Scope Use
267
+
268
+ *List how the model may foreseeably be misused and address what users ought not to do with the model.*
269
+ -->
270
+
271
+ ## Evaluation
272
+
273
+ ### Metrics
274
+
275
+ #### Semantic Similarity
276
+
277
+ * Datasets: `silma-sts-dev-768`, `silma-sts-dev-512`, `silma-sts-dev-256`, `silma-sts-dev-128`, `silma-sts-dev-64`, `sts17-ar-test-768`, `sts17-ar-test-512`, `sts17-ar-test-256`, `sts17-ar-test-128` and `sts17-ar-test-64`
278
+ * Evaluated with [<code>EmbeddingSimilarityEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.EmbeddingSimilarityEvaluator)
279
+
280
+ | Metric | silma-sts-dev-768 | silma-sts-dev-512 | silma-sts-dev-256 | silma-sts-dev-128 | silma-sts-dev-64 | sts17-ar-test-768 | sts17-ar-test-512 | sts17-ar-test-256 | sts17-ar-test-128 | sts17-ar-test-64 |
281
+ |:--------------------|:------------------|:------------------|:------------------|:------------------|:-----------------|:------------------|:------------------|:------------------|:------------------|:-----------------|
282
+ | pearson_cosine | 0.8894 | 0.8959 | 0.9079 | 0.9182 | 0.9066 | 0.8205 | 0.8193 | 0.8191 | 0.8115 | 0.7962 |
283
+ | **spearman_cosine** | **0.8358** | **0.8395** | **0.847** | **0.8562** | **0.8434** | **0.8258** | **0.8227** | **0.8246** | **0.8183** | **0.8077** |
284
+
285
+ <!--
286
+ ## Bias, Risks and Limitations
287
+
288
+ *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
289
+ -->
290
+
291
+ <!--
292
+ ### Recommendations
293
+
294
+ *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
295
+ -->
296
+
297
+ ## Training Details
298
+
299
+ ### Training Dataset
300
+
301
+ #### silma-arabic-english-sts-dataset-v1.0
302
+
303
+ * Dataset: [silma-arabic-english-sts-dataset-v1.0](https://huggingface.co/datasets/silma-ai/silma-arabic-english-sts-dataset-v1.0) at [1885690](https://huggingface.co/datasets/silma-ai/silma-arabic-english-sts-dataset-v1.0/tree/18856908c58bc3779ad089ec327093c8761d2523)
304
+ * Size: 34,436 training samples
305
+ * Columns: <code>sentence1</code>, <code>sentence2</code>, <code>score</code>, and <code>langs</code>
306
+ * Approximate statistics based on the first 1000 samples:
307
+ | | sentence1 | sentence2 | score | langs |
308
+ |:--------|:---------------------------------------------------------------------------------|:---------------------------------------------------------------------------------|:---------------------------------------------------------------|:-------------------------------------------------------------------------------|
309
+ | type | string | string | float | string |
310
+ | details | <ul><li>min: 4 tokens</li><li>mean: 9.68 tokens</li><li>max: 26 tokens</li></ul> | <ul><li>min: 4 tokens</li><li>mean: 9.68 tokens</li><li>max: 26 tokens</li></ul> | <ul><li>min: 0.0</li><li>mean: 0.47</li><li>max: 1.0</li></ul> | <ul><li>min: 5 tokens</li><li>mean: 5.0 tokens</li><li>max: 5 tokens</li></ul> |
311
+ * Samples:
312
+ | sentence1 | sentence2 | score | langs |
313
+ |:-----------------------------------|:-----------------------------------|:-----------------|:-------------------|
314
+ | <code>رجل يعزف على البيانو</code> | <code>امرأة تعزف على الكمان</code> | <code>0.2</code> | <code>ar-ar</code> |
315
+ | <code>امرأة تعزف على الكمان</code> | <code>رجل يعزف على البيانو</code> | <code>0.2</code> | <code>ar-ar</code> |
316
+ | <code>امرأة تعزف على الناي.</code> | <code>رجل يعزف على الغيتار</code> | <code>0.2</code> | <code>ar-ar</code> |
317
+ * Loss: [<code>MatryoshkaLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters:
318
+ ```json
319
+ {
320
+ "loss": "CoSENTLoss",
321
+ "matryoshka_dims": [
322
+ 768,
323
+ 512,
324
+ 256,
325
+ 128,
326
+ 64
327
+ ],
328
+ "matryoshka_weights": [
329
+ 1,
330
+ 1,
331
+ 1,
332
+ 1,
333
+ 1
334
+ ],
335
+ "n_dims_per_step": -1
336
+ }
337
+ ```
338
+
339
+ ### Evaluation Dataset
340
+
341
+ #### silma-arabic-english-sts-dataset-v1.0
342
+
343
+ * Dataset: [silma-arabic-english-sts-dataset-v1.0](https://huggingface.co/datasets/silma-ai/silma-arabic-english-sts-dataset-v1.0) at [1885690](https://huggingface.co/datasets/silma-ai/silma-arabic-english-sts-dataset-v1.0/tree/18856908c58bc3779ad089ec327093c8761d2523)
344
+ * Size: 100 evaluation samples
345
+ * Columns: <code>sentence1</code>, <code>sentence2</code>, <code>score</code>, and <code>langs</code>
346
+ * Approximate statistics based on the first 100 samples:
347
+ | | sentence1 | sentence2 | score | langs |
348
+ |:--------|:---------------------------------------------------------------------------------|:---------------------------------------------------------------------------------|:---------------------------------------------------------------|:-------------------------------------------------------------------------------|
349
+ | type | string | string | float | string |
350
+ | details | <ul><li>min: 5 tokens</li><li>mean: 9.49 tokens</li><li>max: 19 tokens</li></ul> | <ul><li>min: 5 tokens</li><li>mean: 9.49 tokens</li><li>max: 19 tokens</li></ul> | <ul><li>min: 0.1</li><li>mean: 0.74</li><li>max: 1.0</li></ul> | <ul><li>min: 5 tokens</li><li>mean: 5.0 tokens</li><li>max: 5 tokens</li></ul> |
351
+ * Samples:
352
+ | sentence1 | sentence2 | score | langs |
353
+ |:-----------------------------------|:--------------------------------|:------------------|:-------------------|
354
+ | <code>طائرة ستقلع</code> | <code>طائرة طيران تقلع</code> | <code>1.0</code> | <code>ar-ar</code> |
355
+ | <code>طائرة طيران تقلع</code> | <code>طائرة ستقلع</code> | <code>1.0</code> | <code>ar-ar</code> |
356
+ | <code>رجل يعزف على ناي كبير</code> | <code>رجل يعزف على الناي</code> | <code>0.76</code> | <code>ar-ar</code> |
357
+ * Loss: [<code>MatryoshkaLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters:
358
+ ```json
359
+ {
360
+ "loss": "CoSENTLoss",
361
+ "matryoshka_dims": [
362
+ 768,
363
+ 512,
364
+ 256,
365
+ 128,
366
+ 64
367
+ ],
368
+ "matryoshka_weights": [
369
+ 1,
370
+ 1,
371
+ 1,
372
+ 1,
373
+ 1
374
+ ],
375
+ "n_dims_per_step": -1
376
+ }
377
+ ```
378
+
379
+ ### Training Hyperparameters
380
+ #### Non-Default Hyperparameters
381
+
382
+ - `eval_strategy`: steps
383
+ - `per_device_train_batch_size`: 16
384
+ - `per_device_eval_batch_size`: 16
385
+ - `num_train_epochs`: 4
386
+ - `warmup_ratio`: 0.1
387
+ - `save_only_model`: True
388
+ - `fp16`: True
389
+ - `load_best_model_at_end`: True
390
+
391
+ #### All Hyperparameters
392
+ <details><summary>Click to expand</summary>
393
+
394
+ - `overwrite_output_dir`: False
395
+ - `do_predict`: False
396
+ - `eval_strategy`: steps
397
+ - `prediction_loss_only`: True
398
+ - `per_device_train_batch_size`: 16
399
+ - `per_device_eval_batch_size`: 16
400
+ - `per_gpu_train_batch_size`: None
401
+ - `per_gpu_eval_batch_size`: None
402
+ - `gradient_accumulation_steps`: 1
403
+ - `eval_accumulation_steps`: None
404
+ - `torch_empty_cache_steps`: None
405
+ - `learning_rate`: 5e-05
406
+ - `weight_decay`: 0.0
407
+ - `adam_beta1`: 0.9
408
+ - `adam_beta2`: 0.999
409
+ - `adam_epsilon`: 1e-08
410
+ - `max_grad_norm`: 1.0
411
+ - `num_train_epochs`: 4
412
+ - `max_steps`: -1
413
+ - `lr_scheduler_type`: linear
414
+ - `lr_scheduler_kwargs`: {}
415
+ - `warmup_ratio`: 0.1
416
+ - `warmup_steps`: 0
417
+ - `log_level`: passive
418
+ - `log_level_replica`: warning
419
+ - `log_on_each_node`: True
420
+ - `logging_nan_inf_filter`: True
421
+ - `save_safetensors`: True
422
+ - `save_on_each_node`: False
423
+ - `save_only_model`: True
424
+ - `restore_callback_states_from_checkpoint`: False
425
+ - `no_cuda`: False
426
+ - `use_cpu`: False
427
+ - `use_mps_device`: False
428
+ - `seed`: 42
429
+ - `data_seed`: None
430
+ - `jit_mode_eval`: False
431
+ - `use_ipex`: False
432
+ - `bf16`: False
433
+ - `fp16`: True
434
+ - `fp16_opt_level`: O1
435
+ - `half_precision_backend`: auto
436
+ - `bf16_full_eval`: False
437
+ - `fp16_full_eval`: False
438
+ - `tf32`: None
439
+ - `local_rank`: 0
440
+ - `ddp_backend`: None
441
+ - `tpu_num_cores`: None
442
+ - `tpu_metrics_debug`: False
443
+ - `debug`: []
444
+ - `dataloader_drop_last`: False
445
+ - `dataloader_num_workers`: 0
446
+ - `dataloader_prefetch_factor`: None
447
+ - `past_index`: -1
448
+ - `disable_tqdm`: False
449
+ - `remove_unused_columns`: True
450
+ - `label_names`: None
451
+ - `load_best_model_at_end`: True
452
+ - `ignore_data_skip`: False
453
+ - `fsdp`: []
454
+ - `fsdp_min_num_params`: 0
455
+ - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
456
+ - `tp_size`: 0
457
+ - `fsdp_transformer_layer_cls_to_wrap`: None
458
+ - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
459
+ - `deepspeed`: None
460
+ - `label_smoothing_factor`: 0.0
461
+ - `optim`: adamw_torch
462
+ - `optim_args`: None
463
+ - `adafactor`: False
464
+ - `group_by_length`: False
465
+ - `length_column_name`: length
466
+ - `ddp_find_unused_parameters`: None
467
+ - `ddp_bucket_cap_mb`: None
468
+ - `ddp_broadcast_buffers`: False
469
+ - `dataloader_pin_memory`: True
470
+ - `dataloader_persistent_workers`: False
471
+ - `skip_memory_metrics`: True
472
+ - `use_legacy_prediction_loop`: False
473
+ - `push_to_hub`: False
474
+ - `resume_from_checkpoint`: None
475
+ - `hub_model_id`: None
476
+ - `hub_strategy`: every_save
477
+ - `hub_private_repo`: None
478
+ - `hub_always_push`: False
479
+ - `gradient_checkpointing`: False
480
+ - `gradient_checkpointing_kwargs`: None
481
+ - `include_inputs_for_metrics`: False
482
+ - `include_for_metrics`: []
483
+ - `eval_do_concat_batches`: True
484
+ - `fp16_backend`: auto
485
+ - `push_to_hub_model_id`: None
486
+ - `push_to_hub_organization`: None
487
+ - `mp_parameters`:
488
+ - `auto_find_batch_size`: False
489
+ - `full_determinism`: False
490
+ - `torchdynamo`: None
491
+ - `ray_scope`: last
492
+ - `ddp_timeout`: 1800
493
+ - `torch_compile`: False
494
+ - `torch_compile_backend`: None
495
+ - `torch_compile_mode`: None
496
+ - `include_tokens_per_second`: False
497
+ - `include_num_input_tokens_seen`: False
498
+ - `neftune_noise_alpha`: None
499
+ - `optim_target_modules`: None
500
+ - `batch_eval_metrics`: False
501
+ - `eval_on_start`: False
502
+ - `use_liger_kernel`: False
503
+ - `eval_use_gather_object`: False
504
+ - `average_tokens_across_devices`: False
505
+ - `prompts`: None
506
+ - `batch_sampler`: batch_sampler
507
+ - `multi_dataset_batch_sampler`: proportional
508
+
509
+ </details>
510
+
511
+ ### Training Logs
512
+ | Epoch | Step | Training Loss | Validation Loss | silma-sts-dev-768_spearman_cosine | silma-sts-dev-512_spearman_cosine | silma-sts-dev-256_spearman_cosine | silma-sts-dev-128_spearman_cosine | silma-sts-dev-64_spearman_cosine | sts17-ar-test-768_spearman_cosine | sts17-ar-test-512_spearman_cosine | sts17-ar-test-256_spearman_cosine | sts17-ar-test-128_spearman_cosine | sts17-ar-test-64_spearman_cosine |
513
+ |:----------:|:--------:|:-------------:|:---------------:|:---------------------------------:|:---------------------------------:|:---------------------------------:|:---------------------------------:|:--------------------------------:|:---------------------------------:|:---------------------------------:|:---------------------------------:|:---------------------------------:|:--------------------------------:|
514
+ | 0.0929 | 100 | 39.5796 | 45.0982 | 0.7199 | 0.7173 | 0.7292 | 0.7433 | 0.7196 | - | - | - | - | - |
515
+ | 0.1857 | 200 | 31.3305 | 29.9877 | 0.7233 | 0.7248 | 0.7344 | 0.7337 | 0.7192 | - | - | - | - | - |
516
+ | 0.2786 | 300 | 27.7756 | 31.4644 | 0.7288 | 0.7268 | 0.7331 | 0.7388 | 0.7169 | - | - | - | - | - |
517
+ | 0.3714 | 400 | 27.7405 | 33.3315 | 0.7172 | 0.7168 | 0.7341 | 0.7349 | 0.7219 | - | - | - | - | - |
518
+ | 0.4643 | 500 | 27.1884 | 30.4957 | 0.7469 | 0.7428 | 0.7475 | 0.7547 | 0.7426 | - | - | - | - | - |
519
+ | 0.5571 | 600 | 27.0428 | 29.5877 | 0.7133 | 0.7138 | 0.7380 | 0.7549 | 0.7533 | - | - | - | - | - |
520
+ | 0.6500 | 700 | 26.7957 | 30.3813 | 0.7520 | 0.7430 | 0.7570 | 0.7604 | 0.7647 | - | - | - | - | - |
521
+ | 0.7428 | 800 | 26.2667 | 30.6293 | 0.7323 | 0.7333 | 0.7558 | 0.7609 | 0.7479 | - | - | - | - | - |
522
+ | 0.8357 | 900 | 25.9412 | 29.8621 | 0.7730 | 0.7732 | 0.7913 | 0.8117 | 0.7797 | - | - | - | - | - |
523
+ | 0.9285 | 1000 | 25.7816 | 31.7315 | 0.7856 | 0.7918 | 0.7916 | 0.8025 | 0.8048 | - | - | - | - | - |
524
+ | 1.0214 | 1100 | 25.1666 | 31.6311 | 0.7651 | 0.7668 | 0.7673 | 0.7826 | 0.7846 | - | - | - | - | - |
525
+ | 1.1142 | 1200 | 24.7681 | 32.3005 | 0.7719 | 0.7892 | 0.7941 | 0.8022 | 0.7939 | - | - | - | - | - |
526
+ | 1.2071 | 1300 | 24.8771 | 32.1761 | 0.7660 | 0.7744 | 0.7807 | 0.7884 | 0.7841 | - | - | - | - | - |
527
+ | 1.2999 | 1400 | 24.9063 | 33.2694 | 0.7646 | 0.7644 | 0.7884 | 0.7906 | 0.7886 | - | - | - | - | - |
528
+ | 1.3928 | 1500 | 24.7283 | 32.4350 | 0.7935 | 0.7974 | 0.8071 | 0.8112 | 0.8062 | - | - | - | - | - |
529
+ | 1.4856 | 1600 | 24.4217 | 34.1219 | 0.7781 | 0.7754 | 0.7739 | 0.7916 | 0.7889 | - | - | - | - | - |
530
+ | 1.5785 | 1700 | 24.4923 | 33.1239 | 0.7636 | 0.7709 | 0.7882 | 0.7991 | 0.7913 | - | - | - | - | - |
531
+ | 1.6713 | 1800 | 24.0844 | 33.5233 | 0.7785 | 0.7832 | 0.7880 | 0.7977 | 0.8014 | - | - | - | - | - |
532
+ | 1.7642 | 1900 | 24.1453 | 35.4602 | 0.7795 | 0.7816 | 0.8053 | 0.8115 | 0.7944 | - | - | - | - | - |
533
+ | 1.8570 | 2000 | 24.2271 | 36.2812 | 0.8003 | 0.8009 | 0.8008 | 0.8102 | 0.8009 | - | - | - | - | - |
534
+ | 1.9499 | 2100 | 23.7371 | 37.0276 | 0.7769 | 0.7866 | 0.7918 | 0.7926 | 0.7832 | - | - | - | - | - |
535
+ | 2.0427 | 2200 | 23.3566 | 34.5721 | 0.7931 | 0.8017 | 0.8020 | 0.8159 | 0.8027 | - | - | - | - | - |
536
+ | 2.1356 | 2300 | 23.2523 | 35.5316 | 0.7931 | 0.7981 | 0.7896 | 0.8157 | 0.8142 | - | - | - | - | - |
537
+ | 2.2284 | 2400 | 23.0447 | 36.6811 | 0.7973 | 0.7962 | 0.7935 | 0.8030 | 0.8037 | - | - | - | - | - |
538
+ | 2.3213 | 2500 | 22.9782 | 37.5482 | 0.8121 | 0.8185 | 0.8200 | 0.8293 | 0.8244 | - | - | - | - | - |
539
+ | 2.4141 | 2600 | 22.9119 | 37.2809 | 0.8077 | 0.8116 | 0.8113 | 0.8333 | 0.8151 | - | - | - | - | - |
540
+ | 2.5070 | 2700 | 23.1302 | 37.7993 | 0.8255 | 0.8304 | 0.8310 | 0.8376 | 0.8303 | - | - | - | - | - |
541
+ | 2.5998 | 2800 | 22.9941 | 38.8005 | 0.8182 | 0.8214 | 0.8143 | 0.8193 | 0.8155 | - | - | - | - | - |
542
+ | 2.6927 | 2900 | 22.8876 | 36.2524 | 0.8201 | 0.8222 | 0.8194 | 0.8347 | 0.8260 | - | - | - | - | - |
543
+ | 2.7855 | 3000 | 22.5304 | 38.1523 | 0.8195 | 0.8280 | 0.8356 | 0.8545 | 0.8394 | - | - | - | - | - |
544
+ | 2.8784 | 3100 | 22.446 | 39.4876 | 0.8242 | 0.8246 | 0.8319 | 0.8483 | 0.8397 | - | - | - | - | - |
545
+ | 2.9712 | 3200 | 22.5077 | 39.1910 | 0.8231 | 0.8249 | 0.8334 | 0.8475 | 0.8372 | - | - | - | - | - |
546
+ | **3.0641** | **3300** | **21.9675** | **36.4245** | **0.8408** | **0.8425** | **0.8456** | **0.8619** | **0.8577** | **-** | **-** | **-** | **-** | **-** |
547
+ | 3.1569 | 3400 | 21.9361 | 36.7119 | 0.8344 | 0.8405 | 0.8460 | 0.8656 | 0.8644 | - | - | - | - | - |
548
+ | 3.2498 | 3500 | 21.7747 | 37.7140 | 0.8279 | 0.8353 | 0.8414 | 0.8510 | 0.8446 | - | - | - | - | - |
549
+ | 3.3426 | 3600 | 21.8649 | 38.9102 | 0.8298 | 0.8331 | 0.8456 | 0.8494 | 0.8447 | - | - | - | - | - |
550
+ | 3.4355 | 3700 | 21.794 | 37.4385 | 0.8278 | 0.8328 | 0.8377 | 0.8442 | 0.8373 | - | - | - | - | - |
551
+ | 3.5283 | 3800 | 21.7968 | 37.0225 | 0.8352 | 0.8501 | 0.8540 | 0.8722 | 0.8553 | - | - | - | - | - |
552
+ | 3.6212 | 3900 | 21.5941 | 37.5736 | 0.8344 | 0.8515 | 0.8511 | 0.8643 | 0.8587 | - | - | - | - | - |
553
+ | 3.7140 | 4000 | 21.8181 | 37.4984 | 0.8340 | 0.8440 | 0.8470 | 0.8607 | 0.8484 | - | - | - | - | - |
554
+ | 3.8069 | 4100 | 21.7035 | 37.9701 | 0.8346 | 0.8394 | 0.8436 | 0.8615 | 0.8479 | - | - | - | - | - |
555
+ | 3.8997 | 4200 | 21.398 | 38.1567 | 0.8349 | 0.8365 | 0.8470 | 0.8572 | 0.8405 | - | - | - | - | - |
556
+ | 3.9926 | 4300 | 21.6518 | 38.3515 | 0.8358 | 0.8395 | 0.8470 | 0.8562 | 0.8434 | - | - | - | - | - |
557
+ | 4.0 | 4308 | - | - | - | - | - | - | - | 0.8258 | 0.8227 | 0.8246 | 0.8183 | 0.8077 |
558
+
559
+ * The bold row denotes the saved checkpoint.
560
+
561
+ ### Framework Versions
562
+ - Python: 3.12.7
563
+ - Sentence Transformers: 3.3.1
564
+ - Transformers: 4.51.3
565
+ - PyTorch: 2.6.0+cu124
566
+ - Accelerate: 1.4.0
567
+ - Datasets: 3.3.2
568
+ - Tokenizers: 0.21.1
569
+
570
+ ## Citation
571
+
572
+ ### BibTeX
573
+
574
+ #### Sentence Transformers
575
+ ```bibtex
576
+ @inproceedings{reimers-2019-sentence-bert,
577
+ title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
578
+ author = "Reimers, Nils and Gurevych, Iryna",
579
+ booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
580
+ month = "11",
581
+ year = "2019",
582
+ publisher = "Association for Computational Linguistics",
583
+ url = "https://arxiv.org/abs/1908.10084",
584
+ }
585
+ ```
586
+
587
+ #### MatryoshkaLoss
588
+ ```bibtex
589
+ @misc{kusupati2024matryoshka,
590
+ title={Matryoshka Representation Learning},
591
+ author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
592
+ year={2024},
593
+ eprint={2205.13147},
594
+ archivePrefix={arXiv},
595
+ primaryClass={cs.LG}
596
+ }
597
+ ```
598
+
599
+ #### CoSENTLoss
600
+ ```bibtex
601
+ @online{kexuefm-8847,
602
+ title={CoSENT: A more efficient sentence vector scheme than Sentence-BERT},
603
+ author={Su Jianlin},
604
+ year={2022},
605
+ month={Jan},
606
+ url={https://kexue.fm/archives/8847},
607
+ }
608
+ ```
609
+
610
+ <!--
611
+ ## Glossary
612
+
613
+ *Clearly define terms in order to be accessible across audiences.*
614
+ -->
615
+
616
+ <!--
617
+ ## Model Card Authors
618
+
619
+ *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
620
+ -->
621
+
622
+ <!--
623
+ ## Model Card Contact
624
+
625
+ *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
626
+ -->
config.json ADDED
@@ -0,0 +1,49 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "NewModel"
4
+ ],
5
+ "attention_probs_dropout_prob": 0.0,
6
+ "auto_map": {
7
+ "AutoConfig": "Alibaba-NLP/new-impl--configuration.NewConfig",
8
+ "AutoModel": "Alibaba-NLP/new-impl--modeling.NewModel",
9
+ "AutoModelForMaskedLM": "Alibaba-NLP/new-impl--modeling.NewForMaskedLM",
10
+ "AutoModelForMultipleChoice": "Alibaba-NLP/new-impl--modeling.NewForMultipleChoice",
11
+ "AutoModelForQuestionAnswering": "Alibaba-NLP/new-impl--modeling.NewForQuestionAnswering",
12
+ "AutoModelForSequenceClassification": "Alibaba-NLP/new-impl--modeling.NewForSequenceClassification",
13
+ "AutoModelForTokenClassification": "Alibaba-NLP/new-impl--modeling.NewForTokenClassification"
14
+ },
15
+ "classifier_dropout": 0.0,
16
+ "hidden_act": "gelu",
17
+ "hidden_dropout_prob": 0.1,
18
+ "hidden_size": 768,
19
+ "id2label": {
20
+ "0": "LABEL_0"
21
+ },
22
+ "initializer_range": 0.02,
23
+ "intermediate_size": 3072,
24
+ "label2id": {
25
+ "LABEL_0": 0
26
+ },
27
+ "layer_norm_eps": 1e-12,
28
+ "layer_norm_type": "layer_norm",
29
+ "logn_attention_clip1": false,
30
+ "logn_attention_scale": false,
31
+ "max_position_embeddings": 8192,
32
+ "model_type": "new",
33
+ "num_attention_heads": 12,
34
+ "num_hidden_layers": 12,
35
+ "pack_qkv": true,
36
+ "pad_token_id": 1,
37
+ "position_embedding_type": "rope",
38
+ "rope_scaling": {
39
+ "factor": 8.0,
40
+ "type": "ntk"
41
+ },
42
+ "rope_theta": 20000,
43
+ "torch_dtype": "float32",
44
+ "transformers_version": "4.51.3",
45
+ "type_vocab_size": 1,
46
+ "unpad_inputs": false,
47
+ "use_memory_efficient_attention": false,
48
+ "vocab_size": 250048
49
+ }
config_sentence_transformers.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "__version__": {
3
+ "sentence_transformers": "3.3.1",
4
+ "transformers": "4.51.3",
5
+ "pytorch": "2.6.0+cu124"
6
+ },
7
+ "prompts": {},
8
+ "default_prompt_name": null,
9
+ "similarity_fn_name": "cosine"
10
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f07bd2c8880ab8585d68a59105eee58be37377acf8cc726faf4cf8c1acdfa2d4
3
+ size 1221487872
modules.json ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "idx": 0,
4
+ "name": "0",
5
+ "path": "",
6
+ "type": "sentence_transformers.models.Transformer"
7
+ },
8
+ {
9
+ "idx": 1,
10
+ "name": "1",
11
+ "path": "1_Pooling",
12
+ "type": "sentence_transformers.models.Pooling"
13
+ },
14
+ {
15
+ "idx": 2,
16
+ "name": "2",
17
+ "path": "2_Normalize",
18
+ "type": "sentence_transformers.models.Normalize"
19
+ }
20
+ ]
sentence_bert_config.json ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ {
2
+ "max_seq_length": 8192,
3
+ "do_lower_case": false
4
+ }
special_tokens_map.json ADDED
@@ -0,0 +1,51 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "<s>",
4
+ "lstrip": false,
5
+ "normalized": false,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "cls_token": {
10
+ "content": "<s>",
11
+ "lstrip": false,
12
+ "normalized": false,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "eos_token": {
17
+ "content": "</s>",
18
+ "lstrip": false,
19
+ "normalized": false,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ },
23
+ "mask_token": {
24
+ "content": "<mask>",
25
+ "lstrip": true,
26
+ "normalized": false,
27
+ "rstrip": false,
28
+ "single_word": false
29
+ },
30
+ "pad_token": {
31
+ "content": "<pad>",
32
+ "lstrip": false,
33
+ "normalized": false,
34
+ "rstrip": false,
35
+ "single_word": false
36
+ },
37
+ "sep_token": {
38
+ "content": "</s>",
39
+ "lstrip": false,
40
+ "normalized": false,
41
+ "rstrip": false,
42
+ "single_word": false
43
+ },
44
+ "unk_token": {
45
+ "content": "<unk>",
46
+ "lstrip": false,
47
+ "normalized": false,
48
+ "rstrip": false,
49
+ "single_word": false
50
+ }
51
+ }
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:aa7a6ad87a7ce8fe196787355f6af7d03aee94d19c54a5eb1392ed18c8ef451a
3
+ size 17082988
tokenizer_config.json ADDED
@@ -0,0 +1,62 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "added_tokens_decoder": {
3
+ "0": {
4
+ "content": "<s>",
5
+ "lstrip": false,
6
+ "normalized": false,
7
+ "rstrip": false,
8
+ "single_word": false,
9
+ "special": true
10
+ },
11
+ "1": {
12
+ "content": "<pad>",
13
+ "lstrip": false,
14
+ "normalized": false,
15
+ "rstrip": false,
16
+ "single_word": false,
17
+ "special": true
18
+ },
19
+ "2": {
20
+ "content": "</s>",
21
+ "lstrip": false,
22
+ "normalized": false,
23
+ "rstrip": false,
24
+ "single_word": false,
25
+ "special": true
26
+ },
27
+ "3": {
28
+ "content": "<unk>",
29
+ "lstrip": false,
30
+ "normalized": false,
31
+ "rstrip": false,
32
+ "single_word": false,
33
+ "special": true
34
+ },
35
+ "250001": {
36
+ "content": "<mask>",
37
+ "lstrip": true,
38
+ "normalized": false,
39
+ "rstrip": false,
40
+ "single_word": false,
41
+ "special": true
42
+ }
43
+ },
44
+ "bos_token": "<s>",
45
+ "clean_up_tokenization_spaces": true,
46
+ "cls_token": "<s>",
47
+ "eos_token": "</s>",
48
+ "extra_special_tokens": {},
49
+ "mask_token": "<mask>",
50
+ "max_length": 8192,
51
+ "model_max_length": 8192,
52
+ "pad_to_multiple_of": null,
53
+ "pad_token": "<pad>",
54
+ "pad_token_type_id": 0,
55
+ "padding_side": "right",
56
+ "sep_token": "</s>",
57
+ "stride": 0,
58
+ "tokenizer_class": "XLMRobertaTokenizerFast",
59
+ "truncation_side": "right",
60
+ "truncation_strategy": "longest_first",
61
+ "unk_token": "<unk>"
62
+ }