pytorch vit image-classification transformers license:apache-2.0
---
license: apache-2.0
---

# Taiyi-vit-87M-D (base-sized model)

Based on pre-trained clip-vit-base **(patch 16, resolution 224x224)**, we introduce multimodal information. For multimodal pre-training tasks, we design several special training objectives in our paper. Our code and details of pre-training tasks will be made publicly available upon paper acceptance.

The pre-training datasets are MSCOCO and VG. "D" implies a special training method.

# Taiyi (太乙)

Taiyi models are a branch of the Fengshenbang (封神榜) series of models. The models in Taiyi are pre-trained with multimodal pre-training strategies.

# Usage

```python
from transformers import ViTFeatureExtractor, ViTForImageClassification
from PIL import Image
import requests

url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)

feature_extractor = ViTFeatureExtractor.from_pretrained('IDEA-CCNL/Taiyi-vit-87M-D')
model = ViTForImageClassification.from_pretrained('IDEA-CCNL/Taiyi-vit-87M-D')

inputs = feature_extractor(images=image, return_tensors="pt")
outputs = model(**inputs)
logits = outputs.logits
# model predicts one of the 1000 ImageNet classes
predicted_class_idx = logits.argmax(-1).item()
print("Predicted class:", model.config.id2label[predicted_class_idx])
# Predicted class: Egyptian cat
```

# Benchmark

|                                      | CIFAR10 | ImageNet1k |
|--------------------------------------|:-------:|:----------:|
| clip-vit-base-patch16-224 (official) |   96.2  |    80.2    |
| Taiyi-vit-87M-D (local)              |   98.7  |    82.4    |

The local test settings are:

learning_rate=2e-5, 
batch_size=128, 
num_train_epochs=5, 
weight_decay=0.01

# Citation

If you find the resource is useful, please cite the following website in your paper.

```
@misc{Fengshenbang-LM,
  title={Fengshenbang-LM},
  author={IDEA-CCNL},
  year={2022},
  howpublished={\url{https://github.com/IDEA-CCNL/Fengshenbang-LM}},
}
```