radoslavralev commited on
Commit
f23e239
·
verified ·
1 Parent(s): e75c423

Add new SentenceTransformer model

Browse files
2_Dense/model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:d5e0d234a36d4cd513a1f95bf413a6eea8972f8c835c5270a3e57d4eabf1b5ed
3
- size 1181344
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bc630dbae3594eeb0a6c8575cfa1de738bc5b246dffca9741b2d4f5851dd7989
3
+ size 2362528
3_Dense/model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:38a849eb316f199a702045d159bf2fd1eec62ad05f2bc051329e42e986c6731a
3
- size 1181344
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:88ac9eb6f9310d3ee402976ca8daab484eaf9e3dfcdd22f7dc49d36f7e30ed38
3
+ size 2362528
README.md CHANGED
@@ -63,6 +63,49 @@ datasets:
63
  - redis/langcache-sentencepairs-v2
64
  pipeline_tag: sentence-similarity
65
  library_name: sentence-transformers
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
66
  ---
67
 
68
  # Redis fine-tuned BiEncoder model for semantic caching on LangCache
@@ -128,9 +171,9 @@ print(embeddings.shape)
128
  # Get the similarity scores for the embeddings
129
  similarities = model.similarity(embeddings, embeddings)
130
  print(similarities)
131
- # tensor([[1.0000, 1.0000, 0.7693],
132
- # [1.0000, 1.0000, 0.7693],
133
- # [0.7693, 0.7693, 1.0000]])
134
  ```
135
 
136
  <!--
@@ -157,6 +200,26 @@ You can finetune this model on your own dataset.
157
  *List how the model may foreseeably be misused and address what users ought not to do with the model.*
158
  -->
159
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
160
  <!--
161
  ## Bias, Risks and Limitations
162
 
@@ -246,6 +309,7 @@ You can finetune this model on your own dataset.
246
  - `dataloader_persistent_workers`: True
247
  - `push_to_hub`: True
248
  - `hub_model_id`: redis/langcache-embed-v3
 
249
  - `batch_sampler`: no_duplicates
250
 
251
  #### All Hyperparameters
@@ -359,7 +423,7 @@ You can finetune this model on your own dataset.
359
  - `neftune_noise_alpha`: None
360
  - `optim_target_modules`: None
361
  - `batch_eval_metrics`: False
362
- - `eval_on_start`: False
363
  - `use_liger_kernel`: False
364
  - `liger_kernel_config`: None
365
  - `eval_use_gather_object`: False
@@ -372,6 +436,12 @@ You can finetune this model on your own dataset.
372
 
373
  </details>
374
 
 
 
 
 
 
 
375
  ### Framework Versions
376
  - Python: 3.12.3
377
  - Sentence Transformers: 5.1.0
 
63
  - redis/langcache-sentencepairs-v2
64
  pipeline_tag: sentence-similarity
65
  library_name: sentence-transformers
66
+ metrics:
67
+ - cosine_accuracy@1
68
+ - cosine_precision@1
69
+ - cosine_recall@1
70
+ - cosine_ndcg@10
71
+ - cosine_mrr@1
72
+ - cosine_map@100
73
+ - cosine_auc_precision_cache_hit_ratio
74
+ - cosine_auc_similarity_distribution
75
+ model-index:
76
+ - name: Redis fine-tuned BiEncoder model for semantic caching on LangCache
77
+ results:
78
+ - task:
79
+ type: custom-information-retrieval
80
+ name: Custom Information Retrieval
81
+ dataset:
82
+ name: test
83
+ type: test
84
+ metrics:
85
+ - type: cosine_accuracy@1
86
+ value: 0.5880219631236443
87
+ name: Cosine Accuracy@1
88
+ - type: cosine_precision@1
89
+ value: 0.5880219631236443
90
+ name: Cosine Precision@1
91
+ - type: cosine_recall@1
92
+ value: 0.5706780985738924
93
+ name: Cosine Recall@1
94
+ - type: cosine_ndcg@10
95
+ value: 0.7717640552650085
96
+ name: Cosine Ndcg@10
97
+ - type: cosine_mrr@1
98
+ value: 0.5880219631236443
99
+ name: Cosine Mrr@1
100
+ - type: cosine_map@100
101
+ value: 0.7213999116625115
102
+ name: Cosine Map@100
103
+ - type: cosine_auc_precision_cache_hit_ratio
104
+ value: 0.35292771304732773
105
+ name: Cosine Auc Precision Cache Hit Ratio
106
+ - type: cosine_auc_similarity_distribution
107
+ value: 0.1674589579463346
108
+ name: Cosine Auc Similarity Distribution
109
  ---
110
 
111
  # Redis fine-tuned BiEncoder model for semantic caching on LangCache
 
171
  # Get the similarity scores for the embeddings
172
  similarities = model.similarity(embeddings, embeddings)
173
  print(similarities)
174
+ # tensor([[1.0000, 1.0000, 0.5313],
175
+ # [1.0000, 1.0000, 0.5313],
176
+ # [0.5313, 0.5313, 1.0000]])
177
  ```
178
 
179
  <!--
 
200
  *List how the model may foreseeably be misused and address what users ought not to do with the model.*
201
  -->
202
 
203
+ ## Evaluation
204
+
205
+ ### Metrics
206
+
207
+ #### Custom Information Retrieval
208
+
209
+ * Dataset: `test`
210
+ * Evaluated with <code>ir_evaluator.CustomInformationRetrievalEvaluator</code>
211
+
212
+ | Metric | Value |
213
+ |:-------------------------------------|:-----------|
214
+ | cosine_accuracy@1 | 0.588 |
215
+ | cosine_precision@1 | 0.588 |
216
+ | cosine_recall@1 | 0.5707 |
217
+ | **cosine_ndcg@10** | **0.7718** |
218
+ | cosine_mrr@1 | 0.588 |
219
+ | cosine_map@100 | 0.7214 |
220
+ | cosine_auc_precision_cache_hit_ratio | 0.3529 |
221
+ | cosine_auc_similarity_distribution | 0.1675 |
222
+
223
  <!--
224
  ## Bias, Risks and Limitations
225
 
 
309
  - `dataloader_persistent_workers`: True
310
  - `push_to_hub`: True
311
  - `hub_model_id`: redis/langcache-embed-v3
312
+ - `eval_on_start`: True
313
  - `batch_sampler`: no_duplicates
314
 
315
  #### All Hyperparameters
 
423
  - `neftune_noise_alpha`: None
424
  - `optim_target_modules`: None
425
  - `batch_eval_metrics`: False
426
+ - `eval_on_start`: True
427
  - `use_liger_kernel`: False
428
  - `liger_kernel_config`: None
429
  - `eval_use_gather_object`: False
 
436
 
437
  </details>
438
 
439
+ ### Training Logs
440
+ | Epoch | Step | Validation Loss | test_cosine_ndcg@10 |
441
+ |:-----:|:----:|:---------------:|:-------------------:|
442
+ | 0 | 0 | 1.0850 | 0.7718 |
443
+
444
+
445
  ### Framework Versions
446
  - Python: 3.12.3
447
  - Sentence Transformers: 5.1.0
config.json CHANGED
@@ -12,7 +12,7 @@
12
  "cls_token_id": 50281,
13
  "decoder_bias": true,
14
  "deterministic_flash_attn": false,
15
- "dtype": "bfloat16",
16
  "embedding_dropout": 0.0,
17
  "eos_token_id": 50282,
18
  "global_attn_every_n_layers": 3,
 
12
  "cls_token_id": 50281,
13
  "decoder_bias": true,
14
  "deterministic_flash_attn": false,
15
+ "dtype": "float32",
16
  "embedding_dropout": 0.0,
17
  "eos_token_id": 50282,
18
  "global_attn_every_n_layers": 3,
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:95d02211c4cca89113f9f3e93ed91f5176bf50170faa2cb835f7bfea15bb9dd2
3
- size 298041696
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:04aa7437b7f98ed3f652e300c1d767d07c1864c10b3055ea63831997faefa8d6
3
+ size 596070136