File size: 2,914 Bytes
1a43751 6837c3d 1a43751 03eb332 1a43751 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 |
---
license: mit
base_model:
- meta-llama/Meta-Llama-3-8B
- facebook/dinov2-base
---
# ShareLock: Ultra-Lightweight CLIP-like Vision-Language Model
Welcome to the Hugging Face repository for **ShareLock**, an ultra-lightweight CLIP-like vision-language model. This repository hosts pretrained checkpoints for ShareLock, enabling easy integration into your projects.
ShareLock is introduced in the paper:
**"Do Better Language Models Have Crisper Vision?"**
*[Jona Ruthardt](https://jonaruthardt.github.io), [Gertjan J. Burghouts](https://gertjanburghouts.github.io), [Serge Belongie](https://sergebelongie.github.io), [Yuki M. Asano](yukimasano.github.io)*
🌐 **[Project Page](https://jonaruthardt.github.io/project/ShareLock/)**
⌨️ **[GitHub Repository](https://github.com/JonaRuthardt/ShareLock)**
📄 **[Read the Paper on arXiv](https://arxiv.org/abs/2410.07173)**
---
## 🧠 Model Overview
**ShareLock** combines strong frozen features from unimodal vision and language models to achieve competitive multimodal performance with minimal resources.
### Key Highlights:
- **Ultra-Lightweight:** ShareLock is trained on only 563k image-caption pairs, requiring just 1 GPU hour.
- **Efficient Performance:** Achieves 51% zero-shot accuracy on ImageNet.
- **Plug-and-Play:** Easily integrates into downstream vision-language tasks.
---
## 📂 Available Checkpoints
### Model Variants:
1. **ShareLock trained on [CC3M](https://huggingface.co/FunAILab/ShareLock/blob/main/ShareLock-CC3M.ckpt)**
2. **ShareLock trained on [CC12M](https://huggingface.co/FunAILab/ShareLock/blob/main/ShareLock-CC12M.ckpt)**
---
## 🚀 Usage
You can load ShareLock models using the `ShareLock` class directly for inference or fine-tuning:
### Example: Zero-shot Classification
```python
from sharelock.models.model import ShareLock
# Path to the checkpoint
checkpoint_path = "path/to/checkpoint.ckpt"
config = {
# Add your configuration for model hyperparameters etc. here
}
# Load the ShareLock model
model = ShareLock.load_from_checkpoint(checkpoint_path, config=config)
# Encode text and images for multimodal tasks
image_embeddings = model.encode_image(your_image_tensor)
text_embeddings = model.encode_text(["a cat", "a dog"])
# Perform multimodal operations
```
---
## 🛠️ Details
For training scripts, evaluation, or further implementation details, visit our [GitHub repository](https://github.com/JonaRuthardt/ShareLock)
---
## 📜 Citation
If you use ShareLock in your research, please cite:
```bibtex
@article{ruthardt2024sharelock,
title={Do Better Language Models Have Crisper Vision?},
author={Jona Ruthardt and Gertjan J. Burghouts and Serge Belongie and Yuki M. Asano},
journal={arXiv preprint arXiv:2410.07173},
year={2024}
}
```
---
## 📧 Contact
For any questions or collaborations, feel free to reach out to [Jona Ruthardt](mailto:[email protected]).
|