|
--- |
|
license: apache-2.0 |
|
pipeline_tag: image-to-text |
|
--- |
|
|
|
## generate-cxr |
|
|
|
This BlipForConditionalGeneration model generates realistic radiology reports given an chest X-ray and a clinical indication (e.g. 'RLL crackles, eval for pneumonia'). |
|
|
|
- **Developed by:** Nathan Sutton |
|
- **Model type:** BLIP |
|
- **Language(s) (NLP):** English |
|
- **License:** Apache 2.0 |
|
- **Finetuned from model:** Salesforce/blip-image-captioning-large |
|
|
|
## Model Sources |
|
|
|
- **Repository:** https://github.com/nathansutton/prerad |
|
- **Paper:** https://medium.com/@nasutton/a-new-generative-model-for-radiology-b687a993cbb |
|
- **Demo:** https://nathansutton-prerad.hf.space/ |
|
|
|
|
|
## Out-of-Scope Use |
|
|
|
Any medical application. |
|
|
|
## How to Get Started with the Model |
|
|
|
``` |
|
from PIL import Image |
|
from transformers import BlipForConditionalGeneration, BlipProcessor |
|
|
|
# read in the model |
|
processor = BlipProcessor.from_pretrained("nathansutton/generate-cxr") |
|
model = BlipForConditionalGeneration.from_pretrained("nathansutton/generate-cxr") |
|
|
|
# your data |
|
my_image = 'my-chest-x-ray.jpg' |
|
my_indication = 'RLL crackles, eval for pneumonia' |
|
|
|
# process the inputs |
|
inputs = processor( |
|
images=Image.open(my_image), |
|
text='indication:' + my_indication, |
|
return_tensors="pt" |
|
) |
|
|
|
# generate an entire radiology report |
|
output = model.generate(**inputs,max_length=512) |
|
report = processor.decode(output[0], skip_special_tokens=True) |
|
|
|
``` |
|
|
|
# Training Details |
|
|
|
This model was trained by cross-referencing the radiology reports in MIMIC-CXR with the images in the MIMIC-CXR-JPG. None are available here and require a data usage agreement with physionet. |
|
|
