davidmezzetti commited on
Commit
723775e
·
1 Parent(s): 58bbcf2

Update README

Browse files
Files changed (1) hide show
  1. README.md +14 -14
README.md CHANGED
@@ -95,33 +95,33 @@ Performance of this model compared to the top base models on the [MTEB leaderboa
95
 
96
  The following datasets were used to evaluate model performance.
97
 
98
- - [PubMed QA](https://huggingface.co/datasets/pubmed_qa)
99
  - Subset: pqa_labeled, Split: train, Pair: (question, long_answer)
100
- - [PubMed Subset](https://huggingface.co/datasets/zxvix/pubmed_subset_new)
101
  - Split: test, Pair: (title, text)
102
- - [PubMed Summary](https://huggingface.co/datasets/scientific_papers)
103
  - Subset: pubmed, Split: validation, Pair: (article, abstract)
104
 
105
  Evaluation results from the original model are shown below for reference. The [Pearson correlation coefficient](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient) is used as the evaluation metric.
106
 
107
  | Model | PubMed QA | PubMed Subset | PubMed Summary | Average |
108
  | ----------------------------------------------------------------------------- | --------- | ------------- | -------------- | --------- |
109
- | [all-MiniLM-L6-v2](https://hf.co/sentence-transformers/all-MiniLM-L6-v2) | 90.40 | 95.86 | 94.07 | 93.44 |
110
- | [bge-base-en-v1.5](https://hf.co/BAAI/bge-large-en-v1.5) | 91.02 | 95.60 | 94.49 | 93.70 |
111
- | [gte-base](https://hf.co/thenlper/gte-base) | 92.97 | 96.83 | 96.24 | 95.35 |
112
- | [**pubmedbert-base-embeddings**](https://hf.co/neuml/pubmedbert-base-embeddings) | **93.27** | **97.07** | **96.58** | **95.64** |
113
- | [S-PubMedBert-MS-MARCO](https://hf.co/pritamdeka/S-PubMedBert-MS-MARCO) | 90.86 | 93.33 | 93.54 | 92.58 |
114
 
115
  See the table below for evaluation results per dimension for `pubmedbert-base-embeddings-matryoshka`.
116
 
117
  | Model | PubMed QA | PubMed Subset | PubMed Summary | Average |
118
  | --------------------| --------- | ------------- | -------------- | --------- |
119
- | Dimensions = 64 | 92.16 | 95.85 | 95.67 | 94.56 |
120
- | Dimensions = 128 | 92.80 | 96.44 | 96.22 | 95.15 |
121
- | Dimensions = 256 | 93.11 | 96.68 | 96.53 | 95.44 |
122
- | Dimensions = 384 | 93.42 | 96.79 | 96.61 | 95.61 |
123
- | Dimensions = 512 | 93.37 | 96.87 | 96.61 | 95.62 |
124
- | **Dimensions = 768** | **93.53** | **96.95** | **96.70** | **95.73** |
125
 
126
  This model performs slightly better overall compared to the original model.
127
 
 
95
 
96
  The following datasets were used to evaluate model performance.
97
 
98
+ - [PubMed QA](https://huggingface.co/datasets/qiaojin/PubMedQA)
99
  - Subset: pqa_labeled, Split: train, Pair: (question, long_answer)
100
+ - [PubMed Subset](https://huggingface.co/datasets/awinml/pubmed_abstract_3_1k)
101
  - Split: test, Pair: (title, text)
102
+ - [PubMed Summary](https://huggingface.co/datasets/armanc/scientific_papers)
103
  - Subset: pubmed, Split: validation, Pair: (article, abstract)
104
 
105
  Evaluation results from the original model are shown below for reference. The [Pearson correlation coefficient](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient) is used as the evaluation metric.
106
 
107
  | Model | PubMed QA | PubMed Subset | PubMed Summary | Average |
108
  | ----------------------------------------------------------------------------- | --------- | ------------- | -------------- | --------- |
109
+ | [all-MiniLM-L6-v2](https://hf.co/sentence-transformers/all-MiniLM-L6-v2) | 90.40 | 95.92 | 94.07 | 93.46 |
110
+ | [bge-base-en-v1.5](https://hf.co/BAAI/bge-base-en-v1.5) | 91.02 | 95.82 | 94.49 | 93.78 |
111
+ | [gte-base](https://hf.co/thenlper/gte-base) | 92.97 | 96.90 | 96.24 | 95.37 |
112
+ | [**pubmedbert-base-embeddings**](https://hf.co/neuml/pubmedbert-base-embeddings) | **93.27** | **97.00** | **96.58** | **95.62** |
113
+ | [S-PubMedBert-MS-MARCO](https://hf.co/pritamdeka/S-PubMedBert-MS-MARCO) | 90.86 | 93.68 | 93.54 | 92.69 |
114
 
115
  See the table below for evaluation results per dimension for `pubmedbert-base-embeddings-matryoshka`.
116
 
117
  | Model | PubMed QA | PubMed Subset | PubMed Summary | Average |
118
  | --------------------| --------- | ------------- | -------------- | --------- |
119
+ | Dimensions = 64 | 92.16 | 96.14 | 95.67 | 94.66 |
120
+ | Dimensions = 128 | 92.80 | 96.58 | 96.22 | 95.20 |
121
+ | Dimensions = 256 | 93.11 | 96.82 | 96.53 | 95.49 |
122
+ | Dimensions = 384 | 93.42 | 97.00 | 96.61 | 95.68 |
123
+ | Dimensions = 512 | 93.37 | 97.07 | 96.61 | 95.68 |
124
+ | **Dimensions = 768** | **93.53** | **97.13** | **96.70** | **95.79** |
125
 
126
  This model performs slightly better overall compared to the original model.
127