Commit
·
723775e
1
Parent(s):
58bbcf2
Update README
Browse files
README.md
CHANGED
@@ -95,33 +95,33 @@ Performance of this model compared to the top base models on the [MTEB leaderboa
|
|
95 |
|
96 |
The following datasets were used to evaluate model performance.
|
97 |
|
98 |
-
- [PubMed QA](https://huggingface.co/datasets/
|
99 |
- Subset: pqa_labeled, Split: train, Pair: (question, long_answer)
|
100 |
-
- [PubMed Subset](https://huggingface.co/datasets/
|
101 |
- Split: test, Pair: (title, text)
|
102 |
-
- [PubMed Summary](https://huggingface.co/datasets/scientific_papers)
|
103 |
- Subset: pubmed, Split: validation, Pair: (article, abstract)
|
104 |
|
105 |
Evaluation results from the original model are shown below for reference. The [Pearson correlation coefficient](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient) is used as the evaluation metric.
|
106 |
|
107 |
| Model | PubMed QA | PubMed Subset | PubMed Summary | Average |
|
108 |
| ----------------------------------------------------------------------------- | --------- | ------------- | -------------- | --------- |
|
109 |
-
| [all-MiniLM-L6-v2](https://hf.co/sentence-transformers/all-MiniLM-L6-v2)
|
110 |
-
| [bge-base-en-v1.5](https://hf.co/BAAI/bge-
|
111 |
-
| [gte-base](https://hf.co/thenlper/gte-base)
|
112 |
-
| [**pubmedbert-base-embeddings**](https://hf.co/neuml/pubmedbert-base-embeddings) | **93.27** | **97.
|
113 |
-
| [S-PubMedBert-MS-MARCO](https://hf.co/pritamdeka/S-PubMedBert-MS-MARCO)
|
114 |
|
115 |
See the table below for evaluation results per dimension for `pubmedbert-base-embeddings-matryoshka`.
|
116 |
|
117 |
| Model | PubMed QA | PubMed Subset | PubMed Summary | Average |
|
118 |
| --------------------| --------- | ------------- | -------------- | --------- |
|
119 |
-
| Dimensions = 64 | 92.16 |
|
120 |
-
| Dimensions = 128 | 92.80 | 96.
|
121 |
-
| Dimensions = 256 | 93.11 | 96.
|
122 |
-
| Dimensions = 384 | 93.42 |
|
123 |
-
| Dimensions = 512 | 93.37 |
|
124 |
-
| **Dimensions = 768** | **93.53** | **
|
125 |
|
126 |
This model performs slightly better overall compared to the original model.
|
127 |
|
|
|
95 |
|
96 |
The following datasets were used to evaluate model performance.
|
97 |
|
98 |
+
- [PubMed QA](https://huggingface.co/datasets/qiaojin/PubMedQA)
|
99 |
- Subset: pqa_labeled, Split: train, Pair: (question, long_answer)
|
100 |
+
- [PubMed Subset](https://huggingface.co/datasets/awinml/pubmed_abstract_3_1k)
|
101 |
- Split: test, Pair: (title, text)
|
102 |
+
- [PubMed Summary](https://huggingface.co/datasets/armanc/scientific_papers)
|
103 |
- Subset: pubmed, Split: validation, Pair: (article, abstract)
|
104 |
|
105 |
Evaluation results from the original model are shown below for reference. The [Pearson correlation coefficient](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient) is used as the evaluation metric.
|
106 |
|
107 |
| Model | PubMed QA | PubMed Subset | PubMed Summary | Average |
|
108 |
| ----------------------------------------------------------------------------- | --------- | ------------- | -------------- | --------- |
|
109 |
+
| [all-MiniLM-L6-v2](https://hf.co/sentence-transformers/all-MiniLM-L6-v2) | 90.40 | 95.92 | 94.07 | 93.46 |
|
110 |
+
| [bge-base-en-v1.5](https://hf.co/BAAI/bge-base-en-v1.5) | 91.02 | 95.82 | 94.49 | 93.78 |
|
111 |
+
| [gte-base](https://hf.co/thenlper/gte-base) | 92.97 | 96.90 | 96.24 | 95.37 |
|
112 |
+
| [**pubmedbert-base-embeddings**](https://hf.co/neuml/pubmedbert-base-embeddings) | **93.27** | **97.00** | **96.58** | **95.62** |
|
113 |
+
| [S-PubMedBert-MS-MARCO](https://hf.co/pritamdeka/S-PubMedBert-MS-MARCO) | 90.86 | 93.68 | 93.54 | 92.69 |
|
114 |
|
115 |
See the table below for evaluation results per dimension for `pubmedbert-base-embeddings-matryoshka`.
|
116 |
|
117 |
| Model | PubMed QA | PubMed Subset | PubMed Summary | Average |
|
118 |
| --------------------| --------- | ------------- | -------------- | --------- |
|
119 |
+
| Dimensions = 64 | 92.16 | 96.14 | 95.67 | 94.66 |
|
120 |
+
| Dimensions = 128 | 92.80 | 96.58 | 96.22 | 95.20 |
|
121 |
+
| Dimensions = 256 | 93.11 | 96.82 | 96.53 | 95.49 |
|
122 |
+
| Dimensions = 384 | 93.42 | 97.00 | 96.61 | 95.68 |
|
123 |
+
| Dimensions = 512 | 93.37 | 97.07 | 96.61 | 95.68 |
|
124 |
+
| **Dimensions = 768** | **93.53** | **97.13** | **96.70** | **95.79** |
|
125 |
|
126 |
This model performs slightly better overall compared to the original model.
|
127 |
|