|
--- |
|
license: apache-2.0 |
|
--- |
|
This is the official repo for paper [Supervised Fine-tuning *in turn* Improves Visual Foundation Models](https://arxiv.org/abs/2401.10222). |
|
|
|
<div align="center"> |
|
|
|
📃[**Paper (ArXiv)**](https://arxiv.org/abs/2401.10222) **|** [**Code**](https://github.com/TencentARC/ViSFT) **|** 🤗[**Huggingface**](https://huggingface.co/TencentARC/ViSFT) |
|
|
|
|
|
|
|
</div> |
|
|
|
## News |
|
* [2024/01/19] We open source the [ViSFT]() including training scripts and weights. Evaluation codes will be released soon. |
|
|
|
## Introduction |
|
Image-text training like CLIP has dominated the pretraining of vision foundation models in recent years. Subsequent efforts have been made to introduce region-level visual learning into CLIP’s pretraining but face scalability challenges due to the lack of large-scale region-level datasets. Drawing inspiration from supervised fine-tuning (SFT) in natural language processing such as instruction tuning, we explore the potential of fine-grained SFT in enhancing the generation of vision foundation models after their pretraining. Thus a two-stage method **ViSFT** (**Vi**sion **SFT**) is proposed to unleash the fine-grained knowledge of vision foundation models. In ViSFT, the vision foundation model is enhanced by performing visual joint learning on some in-domain tasks and then tested on out-of-domain benchmarks. With updating using ViSFT on 8 V100 GPUs in less than 2 days, a vision transformer with over 4.4B parameters shows improvements across various out-of-domain benchmarks including vision and vision-linguistic scenarios. |
|
|
|
|
|
## Installation |
|
|
|
### creating a conda environment |
|
``` |
|
conda create -n ViSFT python=3.8 |
|
|
|
conda activate ViSFT |
|
``` |
|
### Install pytorch |
|
we use torch1.12 with CUDA11.3 on 8 NVIDIA Volta V100- SXM2-32GB GPUs |
|
``` |
|
pip install --extra-index-url https://download.pytorch.org/whl/cu113 torch==1.12.0 |
|
|
|
pip install --extra-index-url https://download.pytorch.org/whl/cu113 torchvision==0.13.0 |
|
|
|
pip install --extra-index-url https://download.pytorch.org/whl/cu113 torchaudio==0.12.0 |
|
``` |
|
|
|
|
|
### xformers installation |
|
|
|
Flash attention is required for running EVA-ViT-E. |
|
please refer to [xformers](https://github.com/facebookresearch/xformers) |
|
|
|
### loralib installation |
|
|
|
``` |
|
pip install --user git+https://github.com/microsoft/LoRA |
|
``` |
|
|
|
### compile MSDeform for Mask2former head |
|
``` |
|
cd ./mmf/models/visft/ops |
|
sudo sh make.sh |
|
# back to root dir |
|
cd ../../../../ |
|
``` |
|
|
|
### Other packages installation |
|
``` |
|
pip install -r requirements.txt |
|
``` |
|
|
|
## Dataset Preparation |
|
|
|
export DATA_PATH=your_data_path |
|
|
|
### image caption |
|
Generating hdf5 files for image caption following [hdf5](https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Image-Captioning/blob/master/create_input_files.py) |
|
|
|
file strcture: |
|
|
|
``` |
|
DATA_PATH/ |
|
└── processed_datasets/ |
|
└─── coco_caption_hdf5_files |
|
├──TEST_CAPLENS_coco_5_cap_per_img_5_min_word_freq.json |
|
├──TEST_CAPTIONS_coco_5_cap_per_img_5_min_word_freq.json |
|
├──TEST_IMAGES_coco_5_cap_per_img_5_min_word_freq.hdf5 |
|
├──TRAIN_CAPLENS_coco_5_cap_per_img_5_min_word_freq.json |
|
├──TRAIN_CAPTIONS_coco_5_cap_per_img_5_min_word_freq.json |
|
├──TRAIN_IMAGES_coco_5_cap_per_img_5_min_word_freq.hdf5 |
|
├──VAL_CAPLENS_coco_5_cap_per_img_5_min_word_freq.json |
|
├──VAL_CAPTIONS_coco_5_cap_per_img_5_min_word_freq.json |
|
├──VAL_IMAGES_coco_5_cap_per_img_5_min_word_freq.hdf5 |
|
└───WORDMAP_coco_5_cap_per_img_5_min_word_freq.json |
|
``` |
|
### Detection & Segmentation |
|
|
|
file strcture: |
|
|
|
``` |
|
DATA_PATH/ |
|
└── public_datasets/ |
|
└─── coco |
|
├──train2017 |
|
├──val2017 |
|
├──test2017 |
|
└───annotations |
|
├──instances_train2017.json |
|
├──instances_val2017.json |
|
└───image_info_test-dev2017.json |
|
``` |
|
|
|
## Training |
|
### Stage1 |
|
To get compatible in-domain task heads. Using 8 NVIDIA Volta V100-SXM2-32GB GPUs for every in-domain task head. |
|
|
|
**For eva-vit-g** |
|
|
|
Preparing weights from [LAVIS](https://github.com/salesforce/LAVIS) |
|
``` |
|
wget https://storage.googleapis.com/sfr-vision-language-research/LAVIS/models/BLIP2/eva_vit_g.pth |
|
``` |
|
Adding your weights path to configs under dir:./projects/visft/configs/stage1/eva_g/ |
|
``` |
|
backbone_dir: path/eva_vit_g.pth |
|
``` |
|
Implementing training |
|
``` |
|
bash ./scripts/stage1_train/eva_g/caption.sh |
|
bash ./scripts/stage1_train/eva_g/detection.sh |
|
bash ./scripts/stage1_train/eva_g/segment.sh |
|
``` |
|
|
|
**For eva-vit-e** |
|
|
|
Preparing EVA-CLIP weights from [EVA](https://huggingface.co/QuanSun/EVA-CLIP/blob/main/EVA02_CLIP_E_psz14_plus_s9B.pt) |
|
|
|
Extact ViT weights |
|
``` |
|
python ./scripts/preprocess/extract_eva_e_vit.py |
|
``` |
|
Adding your weights path to configs under dir:./projects/visft/configs/stage1/eva_e/ |
|
``` |
|
backbone_dir: path/EVA02_CLIP_E_psz14_plus_s9B_Visual.pt |
|
``` |
|
Implementing training |
|
``` |
|
# can be executed in parallel |
|
bash ./scripts/stage1_train/eva_e/caption.sh |
|
bash ./scripts/stage1_train/eva_e/detection.sh |
|
bash ./scripts/stage1_train/eva_e/segment.sh |
|
``` |
|
|
|
Or you can use the weights we provided. |
|
|
|
| In-domain Heads | | | |
|
|----------|:-------------:|:-------------:| |
|
| | EVA-G | EVA-E| |
|
| Caption Head | [weights](https://huggingface.co/TencentARC/ViSFT/blob/main/eva_g_caption_heads.ckpt) | [weights](https://huggingface.co/TencentARC/ViSFT/blob/main/eva_e_caption_heads.ckpt)| |
|
| Segment Head | [weights](https://huggingface.co/TencentARC/ViSFT/blob/main/eva_g_segment_heads.ckpt) |[weights](https://huggingface.co/TencentARC/ViSFT/blob/main/eva_e_segment_heads.ckpt)| |
|
| Detection Head | [weights](https://huggingface.co/TencentARC/ViSFT/blob/main/eva_g_detection_heads.ckpt) |[weights](https://huggingface.co/TencentARC/ViSFT/blob/main/eva_e_detection_heads.ckpt)| |
|
|
|
|
|
### Stage2 |
|
|
|
**For eva-vit-g** |
|
|
|
Adding your weights path to configs under dir:./projects/visft/configs/stage2/eva_g/stage2.yaml |
|
``` |
|
backbone_dir: path/eva_vit_g.pth |
|
caption_ckpt_path: 'path/eva_g_caption_heads.ckpt' |
|
segment_ckpt_path:'path/eva_g_segment_heads.ckpt' |
|
detection_ckpt_path: 'path/eva_g_detection_heads.ckpt' |
|
``` |
|
Implementing training |
|
``` |
|
bash ./scripts/stage2_train/eva_g/stage2.sh |
|
``` |
|
|
|
**For eva-vit-e** |
|
|
|
Adding your weights path to configs under dir:./projects/visft/configs/stage2/eva_e/stage2.yaml |
|
``` |
|
backbone_dir: path/EVA02_CLIP_E_psz14_plus_s9B_Visual.pt |
|
caption_ckpt_path: 'path/eva_e_caption_heads.ckpt' |
|
segment_ckpt_path:'path/eva_e_segment_heads.ckpt' |
|
detection_ckpt_path: 'path/eva_e_detection_heads.ckpt' |
|
``` |
|
Implementing training |
|
``` |
|
bash ./scripts/stage2_train/eva_e/stage2.sh |
|
``` |
|
### Get LoRA Weights |
|
You can extract expected LoRA weights by |
|
|
|
``` |
|
python ./scripts/postprocess/extract_lora_weights.py |
|
``` |
|
|
|
Or use the LoRA weights we provide: |
|
| LoRA weights | | | |
|
|----------|:-------------:|:-------------:| |
|
| Iters| EVA-G | EVA-E| |
|
| 5k | [weights](https://huggingface.co/TencentARC/ViSFT/blob/main/eva_g_lora_5000.pt) | [weights](https://huggingface.co/TencentARC/ViSFT/blob/main/eva_e_lora_5000.pt)| |
|
| 10k | [weights](https://huggingface.co/TencentARC/ViSFT/blob/main/eva_g_lora_10000.pt) |[weights](https://huggingface.co/TencentARC/ViSFT/blob/main/eva_e_lora_10000.pt)| |
|
| 15k | [weights](https://huggingface.co/TencentARC/ViSFT/blob/main/eva_g_lora_15000.pt) |[weights](https://huggingface.co/TencentARC/ViSFT/blob/main/eva_e_lora_15000.pt)| |
|
| 20k | [weights](https://huggingface.co/TencentARC/ViSFT/blob/main/eva_g_lora_20000.pt) |[weights](https://huggingface.co/TencentARC/ViSFT/blob/main/eva_e_lora_20000.pt)| |
|
| 50k | [weights](https://huggingface.co/TencentARC/ViSFT/blob/main/eva_g_lora_50000.pt) |[weights](https://huggingface.co/TencentARC/ViSFT/blob/main/eva_e_lora_50000.pt)| |
|
## Evaluation Benchmarks |
|
- [] Zero-shot Image Classification |
|
- [] Zero-shot Image-text Retrieval |
|
- [] OCR |
|
- [] Grounded Object Indentification |
|
- [] VQA |
|
- [] Image Captioning on NoCaps |
|
|
|
## Acknowledgement |
|
The code of ViSFT is based on the official implementation of [mmf](https://github.com/facebookresearch/mmf), [EVA](https://github.com/baaivision/EVA/tree/master) and [LAVIS](https://github.com/salesforce/LAVIS/tree/main) |
|
|
|
## Citation |
|
If you found our work valuable, please cite: |
|
``` |
|
@misc{jiang2024supervised, |
|
title={Supervised Fine-tuning in turn Improves Visual Foundation Models}, |
|
author={Xiaohu Jiang and Yixiao Ge and Yuying Ge and Chun Yuan and Ying Shan}, |
|
year={2024}, |
|
eprint={2401.10222}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CV} |
|
} |
|
``` |
|
|
|
|
|
|
