---
license: openrail
datasets:
- openbmb/VisRAG-Ret-Train-In-domain-data
base_model:
- openai/clip-vit-large-patch14
tags:
- Embeddings
- Multi-modal
- text2image
- text2text
---

# OmniEmb-v1: Multi-Modal Embeddings for Unified Retrieval

A compact multi-modal embedding model that creates unified embeddings for text and images, enabling efficient retrieval across modalities without intermediate VLM transformations.

## Features

* 1536d unified embedding space
* Text2Text, Text2Image, and Image2Image retrieval support
* Direct embedding without VLM conversion steps
* Layout preservation for image data

## Performance

### Cross-Modal Retrieval (vs CLIP-ViT-B/32)
* Hits@1: 0.428 (+60.8%)
* Hits@5: 0.651 (+38.9%)

### Correlation Metrics (vs LaBSE)
* STS-B Pearson: 0.800 (+9.7%)
* STS-B Spearman: 0.795 (+7.3%)
* SICK Pearson: 0.782 (+6.3%)

### Error Metrics (vs LaBSE)
* STS-B MSE: 3.222 (-19.6%)
* SICK MSE: 0.750 (-41.5%)

## Installation & Usage

Install package:
```bash
pip install sportsvision
```

Basic usage:
```python
import torch
from sportsvision.research.configs import UnifiedEmbedderConfig
from sportsvision.research.models import UnifiedEmbedderModel
from transformers import AutoConfig, AutoModel
from PIL import Image

# Register the custom configuration and model
AutoConfig.register("unified_embedder", UnifiedEmbedderConfig)
AutoModel.register(UnifiedEmbedderConfig, UnifiedEmbedderModel)

# Initialize the model from the pretrained repository
emb_model = AutoModel.from_pretrained("sportsvision/omniemb-v1")

# Determine the device
device = "cuda" if torch.cuda.is_available() else "cpu"

# Move the model to the device
emb_model = emb_model.to(device)

# Set the model to evaluation mode
emb_model.eval()

# Sample texts
texts = [
    "Playoff season is exciting!",
    "Injury updates for the team."
]

# Encode texts to obtain embeddings
text_embeddings = emb_model.encode_texts(texts)
print("Text Embeddings:", text_embeddings)

# Sample images
image_paths = [
    "path_to_image1.jpg",
    "path_to_image2.jpg"
]

# Load images using PIL
images = [Image.open(img_path).convert('RGB') for img_path in image_paths]

# Encode images to obtain embeddings
image_embeddings = emb_model.encode_images(images)
print("Image Embeddings:", image_embeddings)
```

## Training

* Fine-tuned CLIP architecture
* Trained on VisRAG dataset using contrastive loss
* Evaluation scripts and detailed methodology documentation coming soon

## Limitations

* Currently being benchmarked against ImageBind and other similar models
* Working on model extensions

## Citation

If you use this model in your research, please cite:

```bibtex
@misc{kodathala2024omniemb,
  author = {Kodathala, Varun},
  title = {OmniEmb-v1: Multi-Modal Embeddings for Unified Retrieval},
  year = {2024},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/sportsvision/omniemb-v1}}
}
```