VisualHeist - figure, scheme and table segmentation from PDFs (with captions, headers & footnotes)
Model Summary
VisualHeist is an object detection model finetuned to extract tables and figures from PDFs. VisualHeist has two versions:
The base model is recommended if you are running it on low-RAM systems
The models are finetuned from microsoft/Florence-2 checkpoints. VisualHeist is inspired by and adapted from yifeihu/TF-ID
- The models were finetuned with 3435 figures and 1716 tables from 110 PDF articles across various publishers. All bounding boxes are manually annotated using CoCo Annotator.
- TF-ID models take an image of a single paper page as the input, and return image files for all figures, schemes and tables in the given page.
Training Code and Dataset
- Dataset: Zenodo repository
- Code: github.com/aspuru-guzik-group/MERMaid
Benchmarks
We manually curated a diverse evaluation dataset consisting of 121 literature articles covering a range of topics, including organic and inorganic chemistry, atmospheric science, batteries, materials science, metal-organic frameworks (MOFs), biology, and science education. These PDFs, published between 1949 and 2025, include both main articles and supplementary materials.
We also additionally curated another collection of 98 literature articles (MERMaid-100) reporting novel reaction methodologies that spans three distinct chemical domains: organic electrosynthesis, photocatalysis, and organic synthesis.
Additional performance discussion can be found from our preprint article
The full DOI lists can be downloaded from ourZenodo repository.
The evaluation results for visualheist-large are:
| Total Images | F1 score | |
|---|---|---|
| All | 1935 | 93% |
| Main | 423 | 96% |
| pre-2000 | 260 | 93% |
| Supplementary Materials | 1252 | 92% |
| MERMaid-100 | 100 | 99% |
Running the Model
Refer to our github repository for detailed instructions on how to run the model
BibTex and citation info
<To be updated with our archive citation>
- Downloads last month
- 1