Add new SentenceTransformer model
Browse files- 2_Dense/model.safetensors +2 -2
- 3_Dense/model.safetensors +2 -2
- README.md +74 -4
- config.json +1 -1
- model.safetensors +2 -2
2_Dense/model.safetensors
CHANGED
@@ -1,3 +1,3 @@
|
|
1 |
version https://git-lfs.github.com/spec/v1
|
2 |
-
oid sha256:
|
3 |
-
size
|
|
|
1 |
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:bc630dbae3594eeb0a6c8575cfa1de738bc5b246dffca9741b2d4f5851dd7989
|
3 |
+
size 2362528
|
3_Dense/model.safetensors
CHANGED
@@ -1,3 +1,3 @@
|
|
1 |
version https://git-lfs.github.com/spec/v1
|
2 |
-
oid sha256:
|
3 |
-
size
|
|
|
1 |
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:88ac9eb6f9310d3ee402976ca8daab484eaf9e3dfcdd22f7dc49d36f7e30ed38
|
3 |
+
size 2362528
|
README.md
CHANGED
@@ -63,6 +63,49 @@ datasets:
|
|
63 |
- redis/langcache-sentencepairs-v2
|
64 |
pipeline_tag: sentence-similarity
|
65 |
library_name: sentence-transformers
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
66 |
---
|
67 |
|
68 |
# Redis fine-tuned BiEncoder model for semantic caching on LangCache
|
@@ -128,9 +171,9 @@ print(embeddings.shape)
|
|
128 |
# Get the similarity scores for the embeddings
|
129 |
similarities = model.similarity(embeddings, embeddings)
|
130 |
print(similarities)
|
131 |
-
# tensor([[1.0000, 1.0000, 0.
|
132 |
-
# [1.0000, 1.0000, 0.
|
133 |
-
# [0.
|
134 |
```
|
135 |
|
136 |
<!--
|
@@ -157,6 +200,26 @@ You can finetune this model on your own dataset.
|
|
157 |
*List how the model may foreseeably be misused and address what users ought not to do with the model.*
|
158 |
-->
|
159 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
160 |
<!--
|
161 |
## Bias, Risks and Limitations
|
162 |
|
@@ -246,6 +309,7 @@ You can finetune this model on your own dataset.
|
|
246 |
- `dataloader_persistent_workers`: True
|
247 |
- `push_to_hub`: True
|
248 |
- `hub_model_id`: redis/langcache-embed-v3
|
|
|
249 |
- `batch_sampler`: no_duplicates
|
250 |
|
251 |
#### All Hyperparameters
|
@@ -359,7 +423,7 @@ You can finetune this model on your own dataset.
|
|
359 |
- `neftune_noise_alpha`: None
|
360 |
- `optim_target_modules`: None
|
361 |
- `batch_eval_metrics`: False
|
362 |
-
- `eval_on_start`:
|
363 |
- `use_liger_kernel`: False
|
364 |
- `liger_kernel_config`: None
|
365 |
- `eval_use_gather_object`: False
|
@@ -372,6 +436,12 @@ You can finetune this model on your own dataset.
|
|
372 |
|
373 |
</details>
|
374 |
|
|
|
|
|
|
|
|
|
|
|
|
|
375 |
### Framework Versions
|
376 |
- Python: 3.12.3
|
377 |
- Sentence Transformers: 5.1.0
|
|
|
63 |
- redis/langcache-sentencepairs-v2
|
64 |
pipeline_tag: sentence-similarity
|
65 |
library_name: sentence-transformers
|
66 |
+
metrics:
|
67 |
+
- cosine_accuracy@1
|
68 |
+
- cosine_precision@1
|
69 |
+
- cosine_recall@1
|
70 |
+
- cosine_ndcg@10
|
71 |
+
- cosine_mrr@1
|
72 |
+
- cosine_map@100
|
73 |
+
- cosine_auc_precision_cache_hit_ratio
|
74 |
+
- cosine_auc_similarity_distribution
|
75 |
+
model-index:
|
76 |
+
- name: Redis fine-tuned BiEncoder model for semantic caching on LangCache
|
77 |
+
results:
|
78 |
+
- task:
|
79 |
+
type: custom-information-retrieval
|
80 |
+
name: Custom Information Retrieval
|
81 |
+
dataset:
|
82 |
+
name: test
|
83 |
+
type: test
|
84 |
+
metrics:
|
85 |
+
- type: cosine_accuracy@1
|
86 |
+
value: 0.5880219631236443
|
87 |
+
name: Cosine Accuracy@1
|
88 |
+
- type: cosine_precision@1
|
89 |
+
value: 0.5880219631236443
|
90 |
+
name: Cosine Precision@1
|
91 |
+
- type: cosine_recall@1
|
92 |
+
value: 0.5706780985738924
|
93 |
+
name: Cosine Recall@1
|
94 |
+
- type: cosine_ndcg@10
|
95 |
+
value: 0.7717640552650085
|
96 |
+
name: Cosine Ndcg@10
|
97 |
+
- type: cosine_mrr@1
|
98 |
+
value: 0.5880219631236443
|
99 |
+
name: Cosine Mrr@1
|
100 |
+
- type: cosine_map@100
|
101 |
+
value: 0.7213999116625115
|
102 |
+
name: Cosine Map@100
|
103 |
+
- type: cosine_auc_precision_cache_hit_ratio
|
104 |
+
value: 0.35292771304732773
|
105 |
+
name: Cosine Auc Precision Cache Hit Ratio
|
106 |
+
- type: cosine_auc_similarity_distribution
|
107 |
+
value: 0.1674589579463346
|
108 |
+
name: Cosine Auc Similarity Distribution
|
109 |
---
|
110 |
|
111 |
# Redis fine-tuned BiEncoder model for semantic caching on LangCache
|
|
|
171 |
# Get the similarity scores for the embeddings
|
172 |
similarities = model.similarity(embeddings, embeddings)
|
173 |
print(similarities)
|
174 |
+
# tensor([[1.0000, 1.0000, 0.5313],
|
175 |
+
# [1.0000, 1.0000, 0.5313],
|
176 |
+
# [0.5313, 0.5313, 1.0000]])
|
177 |
```
|
178 |
|
179 |
<!--
|
|
|
200 |
*List how the model may foreseeably be misused and address what users ought not to do with the model.*
|
201 |
-->
|
202 |
|
203 |
+
## Evaluation
|
204 |
+
|
205 |
+
### Metrics
|
206 |
+
|
207 |
+
#### Custom Information Retrieval
|
208 |
+
|
209 |
+
* Dataset: `test`
|
210 |
+
* Evaluated with <code>ir_evaluator.CustomInformationRetrievalEvaluator</code>
|
211 |
+
|
212 |
+
| Metric | Value |
|
213 |
+
|:-------------------------------------|:-----------|
|
214 |
+
| cosine_accuracy@1 | 0.588 |
|
215 |
+
| cosine_precision@1 | 0.588 |
|
216 |
+
| cosine_recall@1 | 0.5707 |
|
217 |
+
| **cosine_ndcg@10** | **0.7718** |
|
218 |
+
| cosine_mrr@1 | 0.588 |
|
219 |
+
| cosine_map@100 | 0.7214 |
|
220 |
+
| cosine_auc_precision_cache_hit_ratio | 0.3529 |
|
221 |
+
| cosine_auc_similarity_distribution | 0.1675 |
|
222 |
+
|
223 |
<!--
|
224 |
## Bias, Risks and Limitations
|
225 |
|
|
|
309 |
- `dataloader_persistent_workers`: True
|
310 |
- `push_to_hub`: True
|
311 |
- `hub_model_id`: redis/langcache-embed-v3
|
312 |
+
- `eval_on_start`: True
|
313 |
- `batch_sampler`: no_duplicates
|
314 |
|
315 |
#### All Hyperparameters
|
|
|
423 |
- `neftune_noise_alpha`: None
|
424 |
- `optim_target_modules`: None
|
425 |
- `batch_eval_metrics`: False
|
426 |
+
- `eval_on_start`: True
|
427 |
- `use_liger_kernel`: False
|
428 |
- `liger_kernel_config`: None
|
429 |
- `eval_use_gather_object`: False
|
|
|
436 |
|
437 |
</details>
|
438 |
|
439 |
+
### Training Logs
|
440 |
+
| Epoch | Step | Validation Loss | test_cosine_ndcg@10 |
|
441 |
+
|:-----:|:----:|:---------------:|:-------------------:|
|
442 |
+
| 0 | 0 | 1.0850 | 0.7718 |
|
443 |
+
|
444 |
+
|
445 |
### Framework Versions
|
446 |
- Python: 3.12.3
|
447 |
- Sentence Transformers: 5.1.0
|
config.json
CHANGED
@@ -12,7 +12,7 @@
|
|
12 |
"cls_token_id": 50281,
|
13 |
"decoder_bias": true,
|
14 |
"deterministic_flash_attn": false,
|
15 |
-
"dtype": "
|
16 |
"embedding_dropout": 0.0,
|
17 |
"eos_token_id": 50282,
|
18 |
"global_attn_every_n_layers": 3,
|
|
|
12 |
"cls_token_id": 50281,
|
13 |
"decoder_bias": true,
|
14 |
"deterministic_flash_attn": false,
|
15 |
+
"dtype": "float32",
|
16 |
"embedding_dropout": 0.0,
|
17 |
"eos_token_id": 50282,
|
18 |
"global_attn_every_n_layers": 3,
|
model.safetensors
CHANGED
@@ -1,3 +1,3 @@
|
|
1 |
version https://git-lfs.github.com/spec/v1
|
2 |
-
oid sha256:
|
3 |
-
size
|
|
|
1 |
version https://git-lfs.github.com/spec/v1
|
2 |
+
oid sha256:04aa7437b7f98ed3f652e300c1d767d07c1864c10b3055ea63831997faefa8d6
|
3 |
+
size 596070136
|