Fill-Mask
Transformers
PyTorch
English
bert
exbert
Ozan Oktay commited on
Commit
250e296
·
1 Parent(s): 1d1737c

Update README.md

Browse files

Remove the AKA text to avoid confusion - Instead add a description saying that the general model is suitable for other clinical applications as well.

Files changed (1) hide show
  1. README.md +3 -3
README.md CHANGED
@@ -14,11 +14,11 @@ widget:
14
  example_title: "Medication"
15
  ---
16
 
17
- # CXR-BERT-general (aka PubMed-MIMIC-BERT)
18
 
19
  [CXR-BERT](https://arxiv.org/abs/2204.09817) is a chest X-ray (CXR) domain-specific language model that makes use of an improved vocabulary, novel pretraining procedure, weight regularization, and text augmentations. The resulting model demonstrates improved performance on radiology natural language inference, radiology masked language model token prediction, and downstream vision-language processing tasks such as zero-shot phrase grounding and image classification.
20
 
21
- First, we pretrain **CXR-BERT-general** from a randomly initialized BERT model via Masked Language Modeling (MLM) on abstracts [PubMed](https://pubmed.ncbi.nlm.nih.gov/) and clinical notes from the publicly-available [MIMIC-III](https://physionet.org/content/mimiciii/1.4/) and [MIMIC-CXR](https://physionet.org/content/mimic-cxr/).
22
 
23
  **CXR-BERT-specialized** is continually pretrained from CXR-BERT-general to further specialize in the chest X-ray domain. At the final stage, CXR-BERT is trained in a multi-modal contrastive learning framework, similar to the [CLIP](https://arxiv.org/abs/2103.00020) framework. The latent representation of [CLS] token is utilized to align text/image embeddings.
24
 
@@ -26,7 +26,7 @@ First, we pretrain **CXR-BERT-general** from a randomly initialized BERT model v
26
 
27
  | Model | Model identifier on HuggingFace | Vocabulary | Note |
28
  |---------------------------------------------------|------------------------------------------|----------------|-----------------------------------------------------------|
29
- | CXR-BERT-general (aka PubMed-MIMIC-BERT) | microsoft/BiomedVLP-CXR-BERT-general | PubMed & MIMIC | Pretrained for biomedical literature and clinical domains |
30
  | CXR-BERT-specialized (after multi-modal training) | microsoft/BiomedVLP-CXR-BERT-specialized | PubMed & MIMIC | Pretrained for chest X-ray domain |
31
 
32
  ## Citation
 
14
  example_title: "Medication"
15
  ---
16
 
17
+ # CXR-BERT-general
18
 
19
  [CXR-BERT](https://arxiv.org/abs/2204.09817) is a chest X-ray (CXR) domain-specific language model that makes use of an improved vocabulary, novel pretraining procedure, weight regularization, and text augmentations. The resulting model demonstrates improved performance on radiology natural language inference, radiology masked language model token prediction, and downstream vision-language processing tasks such as zero-shot phrase grounding and image classification.
20
 
21
+ First, we pretrain **CXR-BERT-general** from a randomly initialized BERT model via Masked Language Modeling (MLM) on abstracts [PubMed](https://pubmed.ncbi.nlm.nih.gov/) and clinical notes from the publicly-available [MIMIC-III](https://physionet.org/content/mimiciii/1.4/) and [MIMIC-CXR](https://physionet.org/content/mimic-cxr/). In that regard, the general model is expected to be suitable to be used in other clinical domains outside the chest radiology.
22
 
23
  **CXR-BERT-specialized** is continually pretrained from CXR-BERT-general to further specialize in the chest X-ray domain. At the final stage, CXR-BERT is trained in a multi-modal contrastive learning framework, similar to the [CLIP](https://arxiv.org/abs/2103.00020) framework. The latent representation of [CLS] token is utilized to align text/image embeddings.
24
 
 
26
 
27
  | Model | Model identifier on HuggingFace | Vocabulary | Note |
28
  |---------------------------------------------------|------------------------------------------|----------------|-----------------------------------------------------------|
29
+ | CXR-BERT-general | microsoft/BiomedVLP-CXR-BERT-general | PubMed & MIMIC | Pretrained for biomedical literature and clinical domains |
30
  | CXR-BERT-specialized (after multi-modal training) | microsoft/BiomedVLP-CXR-BERT-specialized | PubMed & MIMIC | Pretrained for chest X-ray domain |
31
 
32
  ## Citation