Ozan Oktay
commited on
Commit
·
250e296
1
Parent(s):
1d1737c
Update README.md
Browse filesRemove the AKA text to avoid confusion - Instead add a description saying that the general model is suitable for other clinical applications as well.
README.md
CHANGED
@@ -14,11 +14,11 @@ widget:
|
|
14 |
example_title: "Medication"
|
15 |
---
|
16 |
|
17 |
-
# CXR-BERT-general
|
18 |
|
19 |
[CXR-BERT](https://arxiv.org/abs/2204.09817) is a chest X-ray (CXR) domain-specific language model that makes use of an improved vocabulary, novel pretraining procedure, weight regularization, and text augmentations. The resulting model demonstrates improved performance on radiology natural language inference, radiology masked language model token prediction, and downstream vision-language processing tasks such as zero-shot phrase grounding and image classification.
|
20 |
|
21 |
-
First, we pretrain **CXR-BERT-general** from a randomly initialized BERT model via Masked Language Modeling (MLM) on abstracts [PubMed](https://pubmed.ncbi.nlm.nih.gov/) and clinical notes from the publicly-available [MIMIC-III](https://physionet.org/content/mimiciii/1.4/) and [MIMIC-CXR](https://physionet.org/content/mimic-cxr/).
|
22 |
|
23 |
**CXR-BERT-specialized** is continually pretrained from CXR-BERT-general to further specialize in the chest X-ray domain. At the final stage, CXR-BERT is trained in a multi-modal contrastive learning framework, similar to the [CLIP](https://arxiv.org/abs/2103.00020) framework. The latent representation of [CLS] token is utilized to align text/image embeddings.
|
24 |
|
@@ -26,7 +26,7 @@ First, we pretrain **CXR-BERT-general** from a randomly initialized BERT model v
|
|
26 |
|
27 |
| Model | Model identifier on HuggingFace | Vocabulary | Note |
|
28 |
|---------------------------------------------------|------------------------------------------|----------------|-----------------------------------------------------------|
|
29 |
-
| CXR-BERT-general
|
30 |
| CXR-BERT-specialized (after multi-modal training) | microsoft/BiomedVLP-CXR-BERT-specialized | PubMed & MIMIC | Pretrained for chest X-ray domain |
|
31 |
|
32 |
## Citation
|
|
|
14 |
example_title: "Medication"
|
15 |
---
|
16 |
|
17 |
+
# CXR-BERT-general
|
18 |
|
19 |
[CXR-BERT](https://arxiv.org/abs/2204.09817) is a chest X-ray (CXR) domain-specific language model that makes use of an improved vocabulary, novel pretraining procedure, weight regularization, and text augmentations. The resulting model demonstrates improved performance on radiology natural language inference, radiology masked language model token prediction, and downstream vision-language processing tasks such as zero-shot phrase grounding and image classification.
|
20 |
|
21 |
+
First, we pretrain **CXR-BERT-general** from a randomly initialized BERT model via Masked Language Modeling (MLM) on abstracts [PubMed](https://pubmed.ncbi.nlm.nih.gov/) and clinical notes from the publicly-available [MIMIC-III](https://physionet.org/content/mimiciii/1.4/) and [MIMIC-CXR](https://physionet.org/content/mimic-cxr/). In that regard, the general model is expected to be suitable to be used in other clinical domains outside the chest radiology.
|
22 |
|
23 |
**CXR-BERT-specialized** is continually pretrained from CXR-BERT-general to further specialize in the chest X-ray domain. At the final stage, CXR-BERT is trained in a multi-modal contrastive learning framework, similar to the [CLIP](https://arxiv.org/abs/2103.00020) framework. The latent representation of [CLS] token is utilized to align text/image embeddings.
|
24 |
|
|
|
26 |
|
27 |
| Model | Model identifier on HuggingFace | Vocabulary | Note |
|
28 |
|---------------------------------------------------|------------------------------------------|----------------|-----------------------------------------------------------|
|
29 |
+
| CXR-BERT-general | microsoft/BiomedVLP-CXR-BERT-general | PubMed & MIMIC | Pretrained for biomedical literature and clinical domains |
|
30 |
| CXR-BERT-specialized (after multi-modal training) | microsoft/BiomedVLP-CXR-BERT-specialized | PubMed & MIMIC | Pretrained for chest X-ray domain |
|
31 |
|
32 |
## Citation
|