
## Multilingual-clip: XLM-Roberta-Large-Vit-B-16Plus

Multilingual-CLIP extends OpenAI's English text encoders to multiple other languages. This model *only* contains the multilingual text encoder. The corresponding image model `Vit-B-16Plus` can be retrieved via instructions found on `mlfoundations` [open_clip repository on Github](https://github.com/mlfoundations/open_clip). We provide a usage example below. 

## Requirements

To use both the multilingual text encoder and corresponding image encoder, we need to install the packages [`multilingual-clip`](https://github.com/FreddeFrallan/Multilingual-CLIP) and [`open_clip_torch`](https://github.com/mlfoundations/open_clip). 

```
pip install multilingual-clip
pip install open_clip_torch
```

## Usage

Extracting embeddings from the text encoder can be done in the following way:

```python
from multilingual_clip import pt_multilingual_clip
import transformers

texts = [
    'Three blind horses listening to Mozart.',
    'Älgen är skogens konung!',
    'Wie leben Eisbären in der Antarktis?',
    'Вы знали, что все белые медведи левши?'
]
model_name = 'M-CLIP/XLM-Roberta-Large-Vit-B-16Plus'

# Load Model & Tokenizer
model = pt_multilingual_clip.MultilingualCLIP.from_pretrained(model_name)
tokenizer = transformers.AutoTokenizer.from_pretrained(model_name)

embeddings = model.forward(texts, tokenizer)
print("Text features shape:", embeddings.shape)
```

Extracting embeddings from the corresponding image encoder:

```python
import torch
import open_clip
import requests
from PIL import Image

device = "cuda" if torch.cuda.is_available() else "cpu"
model, _, preprocess = open_clip.create_model_and_transforms('ViT-B-16-plus-240', pretrained="laion400m_e32")
model.to(device)

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
image = preprocess(image).unsqueeze(0).to(device)

with torch.no_grad():
    image_features = model.encode_image(image)

print("Image features shape:", image_features.shape) 
```

## Evaluation results

None of the M-CLIP models have been extensivly evaluated, but testing them on Txt2Img retrieval on the humanly translated MS-COCO dataset, we see the following **R@10** results:

| Name | En | De | Es | Fr | Zh | It | Pl | Ko | Ru | Tr | Jp |
| ----------------------------------|:-----: |:-----: |:-----: |:-----: | :-----: |:-----: |:-----: |:-----: |:-----: |:-----: |:-----: |
| [OpenAI CLIP Vit-B/32](https://github.com/openai/CLIP)| 90.3 | - | - | - | - | - | - | - | - | - | - |
| [OpenAI CLIP Vit-L/14](https://github.com/openai/CLIP)| 91.8 | - | - | - | - | - | - | - | - | - | - |
| [OpenCLIP ViT-B-16+-](https://github.com/openai/CLIP)| 94.3 | - | - | - | - | - | - | - | - | - | - |
| [LABSE Vit-L/14](https://huggingface.co/M-CLIP/LABSE-Vit-L-14)| 91.6 | 89.6 | 89.5 | 89.9 | 88.9 | 90.1 | 89.8 | 80.8 | 85.5 | 89.8 | 73.9 |
| [XLM-R Large Vit-B/32](https://huggingface.co/M-CLIP/XLM-Roberta-Large-Vit-B-32)| 91.8 | 88.7 | 89.1 | 89.4 | 89.3 | 89.8| 91.4 | 82.1 | 86.1 | 88.8 | 81.0 |
| [XLM-R Vit-L/14](https://huggingface.co/M-CLIP/XLM-Roberta-Large-Vit-L-14)| 92.4 | 90.6 | 91.0 | 90.0 | 89.7 | 91.1 | 91.3 | 85.2 | 85.8 | 90.3 | 81.9 |
| [XLM-R Large Vit-B/16+](https://huggingface.co/M-CLIP/XLM-Roberta-Large-Vit-B-16Plus)| **95.0** | **93.0** | **93.6** | **93.1** | **94.0** | **93.1** | **94.4** | **89.0** | **90.0** | **93.0** | **84.2** |


## Training/Model details

Further details about the model training and data can be found in the [model card](https://github.com/FreddeFrallan/Multilingual-CLIP/blob/main/larger_mclip.md).
{"language": ["multilingual", "af", "sq", "am", "ar", "az", "bn", "bs", "bg", "ca", "zh", "hr", "cs", "da", "nl", "en", "et", "fr", "de", "el", "hi", "hu", "is", "id", "it", "ja", "mk", "ml", "mr", "pl", "pt", "ro", "ru", "sr", "sl", "es", "sw", "sv", "tl", "te", "tr", "tk", "uk", "ur", "ug", "uz", "vi", "xh"]}
transformers pytorch tf multilingual af sq am ar az bn bs bg ca zh hr cs da nl en et fr de el hi hu is id it ja mk ml mr pl pt ro ru sr sl es sw sv tl te tr tk uk ur ug uz vi xh endpoints_compatible has_space region:us
TAGS
#transformers #pytorch #tf #multilingual #af #sq #am #ar #az #bn #bs #bg #ca #zh #hr #cs #da #nl #en #et #fr #de #el #hi #hu #is #id #it #ja #mk #ml #mr #pl #pt #ro #ru #sr #sl #es #sw #sv #tl #te #tr #tk #uk #ur #ug #uz #vi #xh #endpoints_compatible #has_space #region-us 

Multilingual-clip: XLM-Roberta-Large-Vit-B-16Plus
-------------------------------------------------


Multilingual-CLIP extends OpenAI's English text encoders to multiple other languages. This model *only* contains the multilingual text encoder. The corresponding image model 'Vit-B-16Plus' can be retrieved via instructions found on 'mlfoundations' open\_clip repository on Github. We provide a usage example below.


Requirements
------------


To use both the multilingual text encoder and corresponding image encoder, we need to install the packages 'multilingual-clip' and 'open\_clip\_torch'.


Usage
-----


Extracting embeddings from the text encoder can be done in the following way:


Extracting embeddings from the corresponding image encoder:


Evaluation results
------------------


None of the M-CLIP models have been extensivly evaluated, but testing them on Txt2Img retrieval on the humanly translated MS-COCO dataset, we see the following R@10 results:



Training/Model details
----------------------


Further details about the model training and data can be found in the model card.



TAGS
#transformers #pytorch #tf #multilingual #af #sq #am #ar #az #bn #bs #bg #ca #zh #hr #cs #da #nl #en #et #fr #de #el #hi #hu #is #id #it #ja #mk #ml #mr #pl #pt #ro #ru #sr #sl #es #sw #sv #tl #te #tr #tk #uk #ur #ug #uz #vi #xh #endpoints_compatible #has_space #region-us 

passage: TAGS
#transformers #pytorch #tf #multilingual #af #sq #am #ar #az #bn #bs #bg #ca #zh #hr #cs #da #nl #en #et #fr #de #el #hi #hu #is #id #it #ja #mk #ml #mr #pl #pt #ro #ru #sr #sl #es #sw #sv #tl #te #tr #tk #uk #ur #ug #uz #vi #xh #endpoints_compatible #has_space #region-us 
