File size: 3,305 Bytes
1c22e42 207b277 1c22e42 6e834e1 7e6a59d 1c22e42 4912c50 9c46368 4912c50 ff5d2ae 4912c50 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 |
---
base_model:
- timm/ViT-L-16-SigLIP-256
datasets:
- mlfoundations/datacomp_1b
license: mit
tags:
- pytorch_model_hub_mixin
- model_hub_mixin
pipeline_tag: zero-shot-image-classification
library_name: prolip
---
## Official implementation of fine-tuned ViT-L/16 ProLIP on DataComp 1B
- This weight is a fine-tuned version of ViT-L/16 by Probabilistic Language-Image Pre-Training (ProLIP)
- Pre-trained weight
- https://huggingface.co/timm/ViT-L-16-SigLIP-256
- Fine-tuning dataset
- DataComp 1B / Seen samples 1.28B
- Architectural difference
- The original uses 256x256 resolution, while ProLIP uses 224x224
- ProLIP uses `[CLS]` token for pooling, while the original SIGLIP model uses attention pooling.
- ProLIP text encoder uses the `[CLS]` token, while the original model does not.
### Overview
- Paper: https://arxiv.org/abs/2410.18857
- GitHub: https://github.com/naver-ai/prolip
- More models are available at https://huggingface.co/collections/SanghyukChun/prolip-6712595dfc87fd8597350291
### Performance overview
- Zero-shot ImageNet-1k top-1 accuracy: 79.4%
- Zero-shot ImageNet distribution shifts: 68.6%
- Zero-shot VTAB performance: 64.0%
- Zero-shot retrieval performance: 61.3%
- Average zero-shot performance on 38 tasks: 65.9%
```python
import requests
from PIL import Image
import torch
from prolip.model import ProLIPHF
from transformers import CLIPProcessor
from prolip.tokenizer import HFTokenizer
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch16")
model = ProLIPHF.from_pretrained("SanghyukChun/ProLIP-ViT-L-16-FT-DC-1B-1_28M")
tokenizer = HFTokenizer("timm/ViT-B-16-SigLIP", context_length=64, clean="canonicalize")
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
inputs = processor(images=image, return_tensors="pt", padding=True)
texts = ["A couple of cats laying on top of a pink blanket.", "A man walks through a flooded road during a rainstorm", "photo"]
texts = tokenizer(texts)
outputs = model(image=inputs["pixel_values"], text=texts)
l2_logit = outputs["image_features"]["mean"] @ outputs["text_features"]["mean"].T
i_unc = torch.exp(outputs["image_features"]["std"]).sum(dim=-1)
t_unc = torch.exp(outputs["text_features"]["std"]).sum(dim=-1)
csd_logit = l2_logit - 0.5 * t_unc
csd_logit2 = l2_logit.T - 0.5 * i_unc
print("Mean-only image-to-text logits (by L2 distance):", l2_logit)
print("Uncertainty-aware image-to-text logits (by CSD):", csd_logit)
print("Uncertainty-aware text-to-image logits (by CSD):", csd_logit2.T)
print("Image uncertainty: ", i_unc)
print("Text uncertainty: ", t_unc)
```
## Citation
```bibtex
@inproceedings{chun2025prolip,
title={Probabilistic Language-Image Pre-Training},
author={Chun, Sanghyuk and Kim, Wonjae and Park, Song and Yun, Sangdoo},
year={2025},
booktitle={International Conference on Learning Representations (ICLR)},
}
@inproceedings{chun2025longprolip,
title={LongProLIP: A Probabilistic Vision-Language Model with Long Context Text},
author={Chun, Sanghyuk and Yun, Sangdoo},
year={2025},
booktitle={ICLR Workshop on Quantify Uncertainty and Hallucination in Foundation Models},
}
``` |