File size: 3,255 Bytes
2f60e94 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 |
> # Cloned from https://github.com/amazon-science/mm-cot
# Multimodal Chain-of-Thought Reasoning in Language Models
<h5 align="center"><i>"Imagine learning a textbook without figures or tables."</i></h5>
Multimodal-CoT incorporates vision features in a decoupled training framework. The framework consists of two training stages: (i) rationale generation and (ii) answer inference. Both stages share the same model architecture but differ in the input and output.
## Requirements
Install all required python dependencies:
```
pip install -r requirements.txt
```
## Datasets
Download the dataset from the following repository:
```
https://github.com/lupantech/ScienceQA/tree/main/data
```
Download the extracted vision features from [vision_features](https://drive.google.com/file/d/13B0hc_F_45-UlqPLKSgRz-ALtFQ8kIJr/view?usp=share_link) and unzip the files under `vision_features`
## Instructions
### Training
```
# rationale generation
CUDA_VISIBLE_DEVICES=0,1 python main.py \
--model allenai/unifiedqa-t5-base \
--user_msg rationale --img_type detr \
--bs 8 --eval_bs 4 --eval_acc 10 --output_len 512 \
--final_eval --prompt_format QCM-LE
# answer inference
CUDA_VISIBLE_DEVICES=0,1 python main.py \
--model allenai/unifiedqa-t5-base \
--user_msg answer --img_type detr \
--bs 8 --eval_bs 4 --eval_acc 10 --output_len 64 \
--final_eval --prompt_format QCMG-A \
--eval_le experiments/rationale_allenai-unifiedqa-t5-base_detr_QCM-LE_lr5e-05_bs16_op512_ep20/predictions_ans_eval.json \
--test_le experiments/rationale_allenai-unifiedqa-t5-base_detr_QCM-LE_lr5e-05_bs16_op512_ep20/predictions_ans_test.json
```
### Inference
Our trained models are available at [models](https://drive.google.com/file/d/1FtTYOJPHnWnFfCxNC6M3gar4RAX5E21b/view?usp=share_link). To use our trained models, please put the them under the ```models``` folder.
```
# rationale generation
CUDA_VISIBLE_DEVICES=0,1 python main.py \
--model allenai/unifiedqa-t5-base \
--user_msg rationale --img_type detr \
--bs 8 --eval_bs 4 --eval_acc 10 --output_len 512 \
--final_eval --prompt_format QCM-LE \
--evaluate_dir models/MM-CoT-UnifiedQA-base-Rationale
# answer inference
CUDA_VISIBLE_DEVICES=0,1 python main.py \
--model allenai/unifiedqa-t5-base \
--user_msg answer --img_type detr \
--bs 8 --eval_bs 4 --eval_acc 10 --output_len 64 \
--final_eval --prompt_format QCMG-A \
--eval_le models/rationale/predictions_ans_eval.json \
--test_le models/rationale/predictions_ans_test.json \
--evaluate_dir models/MM-CoT-UnifiedQA-base-Answer
```
## Citing MM-CoT
```
@article{zhang2023multicot,
title={Multimodal Chain-of-Thought Reasoning in Language Models},
author={Zhang, Zhuosheng and Zhang, Aston and Li, Mu and Zhao, Hai and Karypis, George and Smola, Alex},
journal={arXiv preprint arXiv:2302.00923},
year={2023}
}
```
## License
This project is licensed under the Apache-2.0 License.
## Acknowledgement
Part of our codes are adapted from [ScienceQA](https://github.com/lupantech/ScienceQA) and [Transformers](https://github.com/huggingface/transformers).
We thank Pan Lu for providing parameter size for ScienceQA baselines.
|