Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,109 @@
----
-license: apache-2.0
----
+### Ops-MM-embedding-v1-7B
+
+**Ops-MM-embedding-v1-7B** is a dense, large-scale multimodal embedding model developed and open-sourced by the Alibaba Cloud OpenSearch-AI team, fine-tuned from Qwen2-VL.
+
+---
+
+### **Key Features**
+
+#### Unified Multimodal Embeddings
+- Encodes text, images, text-image pairs, visual documents, and videos (by treating video frames as multiple image inputs) into a unified embedding space for cross-modal retrieval.
+
+#### High Performance on MMEB
+- Achieves **SOTA results** among models of similar scale on **MMEB-V2** and **MMEB-Image** benchmark (until 2025-07-03).
+
+#### Multilingual Capabilities
+- **Ops-MM-embedding-v1-7B** achieves SOTA performance among dense models on the ViDoRe-v2 benchmark, demonstrating strong cross-lingual generalization.
+
+
+
+### Training data
+
+MMEB-train, CC-3M, colpali training set.
+
+
+### Performance
+
+#### MMEB-V2
+
+| Model                    | Model Size (B) | Overall | Image-Overall | Video-Overall | Visdoc-Overall |
+| ------------------------ | -------------- | ------- | ------------- | ------------- | -------------- |
+| seed-1.6-embedding       | unknown        | 71.57   | 77.78         | 55.34         | 74.41          |
+| Ops-MM-embedding-v1-7B   | 8.29           | 67.79   | 72.72         | 53.76         | 70.91          |
+| Ops-MM-embedding-v1-2B   | 2.21           | 63.62   | 69.03         | 47.56         | 67.55          |
+| VLM2Vec-V2.0-Qwen2VL-2B  | 2.21           | 58.39   | 64.85         | 34.85         | 66.34          |
+| gme-Qwen2-VL-2B-Instruct | 2.21           | 54.37   | 51.89         | 33.86         | 73.47          |
+
+---
+
+#### MMEB-Image
+
+The table below compares performance on MMEB-Image benchmark among models of similar size.
+
+| Models                                | Model Size(B) | Image-Overall | I-CLS | I-QA  | I-RET  | I-VG   |
+| ------------------------------------- | ------------- | ------------- | ----- | ----- | ------ | ------ |
+| Ops-MM-embedding-v1-7B                | 8.29          | **72.72**     | 69.65 | 69.58 | 73.09  | 87.15  |
+| QQMM-embed                            | 8.297         | 72.175        | 70.07 | 69.52 | 71.175 | 87.075 |
+| B3_Qwen2_7B                           | 8.29          | 72            | 70    | 66.5  | 74.1   | 84.6   |
+| UniME(LLaVA-OneVision-7B-LoRA-Res336) | 8.03          | 70.7          | 66.8  | 66.6  | 70.5   | 90.9   |
+| LLaVE-7B                              | 8.03          | 70.3          | 65.7  | 65.4  | 70.9   | 91.9   |
+| UNITE-Instruct-7B                     | 8.29          | 70.3          | 68.3  | 65.1  | 71.6   | 84.8   |
+
+---
+
+#### ViDoRe-v2
+
+| Model                  | Avg      | ESG Restaurant Human | MIT Bio | Econ. Macro | ESG Restaurant Synth. | MIT Bio Multi. | Econ Macro Multi. | ESG Restaurant Synth. Multi. |
+| ---------------------- | -------- | -------------------- | ------- | ----------- | --------------------- | -------------- | ----------------- | ---------------------------- |
+| gme-7B                 | 59.3     | 65.8                 | 64      | 62.9        | 54.3                  | 55.1           | 56.2              | 56.7                         |
+| seed 1.6 embedding     | 58.9     | 63.3                 | 63.9    | 64.0        | 58.4                  | 57.1           | 53.8              | 52.0                         |
+| Ops-MM-embedding-v1-7B | **60.6** | 66.3                 | 58.4    | 67.4        | 60.0                  | 54.3           | 60.9              | 56.8                         |
+| Ops-MM-embedding-v1-2B | 54.4     | 58.6                 | 56.0    | 56.4        | 55.8                  | 52.9           | 47.9              | 53.4                         |
+
+
+
+
+## Usage
+
+```python
+from ops_mm_embedding_v1 import OpsMMEmbeddingV1, fetch_image
+
+
+model = OpsMMEmbeddingV1(
+    "OpenSearch-AI/Ops-MM-embedding-v1-7B",
+    device="cuda",
+    attn_implementation="flash_attention_2"
+)
+
+t2i_prompt = "Find an image that matches the given text."
+texts = [
+    "The Tesla Cybertruck is a battery electric pickup truck built by Tesla, Inc. since 2023.",
+    "Alibaba office.",
+    "Alibaba office.",
+]
+images = [
+    "https://upload.wikimedia.org/wikipedia/commons/e/e9/Tesla_Cybertruck_damaged_window.jpg",
+    "https://upload.wikimedia.org/wikipedia/commons/e/e0/TaobaoCity_Alibaba_Xixi_Park.jpg",
+    "https://upload.wikimedia.org/wikipedia/commons/thumb/b/b0/Alibaba_Binjiang_Park.jpg/1024px-Alibaba_Binjiang_Park.jpg"
+]
+
+images = [fetch_image(image) for image in images]
+
+# Text and image embedding
+text_embeddings = model.get_text_embeddings(texts)
+image_embeddings = model.get_image_embeddings(images)
+print('Text and image embeddings', (text_embeddings @ image_embeddings.T).tolist())
+
+# Fused Embedding
+text_with_image_embeddings = model.get_fused_embeddings(texts=texts, images=images, instruction=t2i_prompt)
+print('Text and image embeddings', (text_embeddings @ image_embeddings.T).tolist())
+
+# Multi-image embeddings
+multi_images = [
+    [images[0]],
+    [images[1], images[2]],
+]
+multi_image_embeddings = model.get_image_embeddings(multi_images)
+print('Multi-image embeddings', (multi_image_embeddings @ multi_image_embeddings.T).tolist())
+
+```

added_tokens.json

@@ -0,0 +1,16 @@
+{
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

chat_template.json

@@ -0,0 +1,3 @@
+{
+  "chat_template": "{% set image_count = namespace(value=0) %}{% set video_count = namespace(value=0) %}{% for message in messages %}{% if loop.first and message['role'] != 'system' %}<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n{% endif %}<|im_start|>{{ message['role'] }}\n{% if message['content'] is string %}{{ message['content'] }}<|im_end|>\n{% else %}{% for content in message['content'] %}{% if content['type'] == 'image' or 'image' in content or 'image_url' in content %}{% set image_count.value = image_count.value + 1 %}{% if add_vision_id %}Picture {{ image_count.value }}: {% endif %}<|vision_start|><|image_pad|><|vision_end|>{% elif content['type'] == 'video' or 'video' in content %}{% set video_count.value = video_count.value + 1 %}{% if add_vision_id %}Video {{ video_count.value }}: {% endif %}<|vision_start|><|video_pad|><|vision_end|>{% elif 'text' in content %}{{ content['text'] }}{% endif %}{% endfor %}<|im_end|>\n{% endif %}{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant\n{% endif %}"
+}

config.json

@@ -0,0 +1,56 @@
+{
+  "architectures": [
+    "Qwen2VLForConditionalGeneration"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 151643,
+  "eos_token_id": 151645,
+  "hidden_act": "silu",
+  "hidden_size": 3584,
+  "image_token_id": 151655,
+  "initializer_range": 0.02,
+  "intermediate_size": 18944,
+  "max_position_embeddings": 32768,
+  "max_window_layers": 28,
+  "model_type": "qwen2_vl",
+  "num_attention_heads": 28,
+  "num_hidden_layers": 28,
+  "num_key_value_heads": 4,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": {
+    "mrope_section": [
+      16,
+      24,
+      24
+    ],
+    "rope_type": "default",
+    "type": "default"
+  },
+  "rope_theta": 1000000.0,
+  "sliding_window": 32768,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.51.1",
+  "use_cache": true,
+  "use_sliding_window": false,
+  "video_token_id": 151656,
+  "vision_config": {
+    "depth": 32,
+    "embed_dim": 1280,
+    "hidden_act": "quick_gelu",
+    "hidden_size": 3584,
+    "in_channels": 3,
+    "in_chans": 3,
+    "mlp_ratio": 4,
+    "model_type": "qwen2_vl",
+    "num_heads": 16,
+    "patch_size": 14,
+    "spatial_merge_size": 2,
+    "spatial_patch_size": 14,
+    "temporal_patch_size": 2
+  },
+  "vision_end_token_id": 151653,
+  "vision_start_token_id": 151652,
+  "vision_token_id": 151654,
+  "vocab_size": 152064
+}

demo.py

@@ -0,0 +1,39 @@
+from ops_mm_embedding_v1 import OpsMMEmbeddingV1, fetch_image
+
+
+model = OpsMMEmbeddingV1(
+    "OpenSearch-AI/Ops-MM-embedding-v1-7B",
+    device="cuda",
+    attn_implementation="flash_attention_2"
+)
+
+t2i_prompt = "Find an image that matches the given text."
+texts = [
+    "The Tesla Cybertruck is a battery electric pickup truck built by Tesla, Inc. since 2023.",
+    "Alibaba office.",
+    "Alibaba office.",
+]
+images = [
+    "https://upload.wikimedia.org/wikipedia/commons/e/e9/Tesla_Cybertruck_damaged_window.jpg",
+    "https://upload.wikimedia.org/wikipedia/commons/e/e0/TaobaoCity_Alibaba_Xixi_Park.jpg",
+    "https://upload.wikimedia.org/wikipedia/commons/thumb/b/b0/Alibaba_Binjiang_Park.jpg/1024px-Alibaba_Binjiang_Park.jpg"
+]
+
+images = [fetch_image(image) for image in images]
+
+# Text and image embedding
+text_embeddings = model.get_text_embeddings(texts)
+image_embeddings = model.get_image_embeddings(images)
+print('Text and image embeddings', (text_embeddings @ image_embeddings.T).tolist())
+
+# Fused Embedding
+text_with_image_embeddings = model.get_fused_embeddings(texts=texts, images=images, instruction=t2i_prompt)
+print('Text and image embeddings', (text_embeddings @ image_embeddings.T).tolist())
+
+# Multi-image embeddings
+multi_images = [
+    [images[0]],
+    [images[1], images[2]],
+]
+multi_image_embeddings = model.get_image_embeddings(multi_images)
+print('Multi-image embeddings', (multi_image_embeddings @ multi_image_embeddings.T).tolist())

generation_config.json

@@ -0,0 +1,13 @@
+{
+  "bos_token_id": 151643,
+  "do_sample": true,
+  "eos_token_id": [
+    151645,
+    151643
+  ],
+  "pad_token_id": 151643,
+  "temperature": 0.01,
+  "top_k": 1,
+  "top_p": 0.001,
+  "transformers_version": "4.51.1"
+}

model-00001-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4185606d530d99b1bec0f01060de12fe659e47f2f85fa2a391e3119f7141283a
+size 4966659944

model-00002-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:239f228a09c8812e92b21d663830f471127dc6718e70ee66a9b3c6199c07fb37
+size 4991495816

model-00003-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e8d8f54412ec8ab7d2674226038a15166891a44672612af950cf184c9f3ee3dd
+size 4932751040

model-00004-of-00004.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b4b6498a60113a3998a8a7f8e86a9df5501dd8b0d203d0b557f8a6a6851cff17
+size 1691924384

model.safetensors.index.json

@@ -0,0 +1,737 @@
+{
+  "metadata": {
+    "total_size": 16582751232
+  },
+  "weight_map": {
+    "lm_head.weight": "model-00004-of-00004.safetensors",
+    "model.embed_tokens.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.10.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.10.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.11.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.12.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.13.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.14.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.15.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.16.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.16.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.16.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.16.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.16.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.16.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.17.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.17.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.17.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.17.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.17.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.17.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.17.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.17.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.17.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.17.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.17.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.17.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.18.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.18.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.18.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.18.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.18.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.18.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.18.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.18.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.18.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.18.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.18.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.18.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.19.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.19.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.19.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.19.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.19.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.19.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.19.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.19.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.19.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.19.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.19.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.19.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.2.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.2.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.20.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.20.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.20.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.20.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.20.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.20.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.20.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.20.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.20.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.20.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.20.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.20.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.21.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.21.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.21.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.21.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.22.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.22.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.22.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.22.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.23.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.24.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.input_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.25.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.input_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.26.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.26.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.26.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
+    "model.layers.26.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
+    "model.layers.27.input_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.27.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.27.mlp.gate_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.27.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.27.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
+    "model.layers.27.self_attn.k_proj.bias": "model-00004-of-00004.safetensors",
+    "model.layers.27.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.27.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.27.self_attn.q_proj.bias": "model-00004-of-00004.safetensors",
+    "model.layers.27.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.27.self_attn.v_proj.bias": "model-00004-of-00004.safetensors",
+    "model.layers.27.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
+    "model.layers.3.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.3.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.input_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.4.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.5.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.5.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.5.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.5.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
+    "model.layers.5.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
+    "model.layers.6.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.6.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.6.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.6.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.6.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.6.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.6.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.6.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.6.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.6.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.6.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.6.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.7.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.7.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.7.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.7.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.7.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.7.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.7.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.7.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.7.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.7.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.7.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.7.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.8.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.8.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.8.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.8.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.8.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.8.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.8.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.8.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.8.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.8.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.8.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.8.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.input_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.9.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.9.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
+    "model.layers.9.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
+    "model.layers.9.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
+    "model.norm.weight": "model-00004-of-00004.safetensors",
+    "visual.blocks.0.attn.proj.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.0.attn.proj.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.0.attn.qkv.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.0.attn.qkv.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.0.mlp.fc1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.0.mlp.fc1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.0.mlp.fc2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.0.mlp.fc2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.0.norm1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.0.norm1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.0.norm2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.0.norm2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.1.attn.proj.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.1.attn.proj.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.1.attn.qkv.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.1.attn.qkv.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.1.mlp.fc1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.1.mlp.fc1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.1.mlp.fc2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.1.mlp.fc2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.1.norm1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.1.norm1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.1.norm2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.1.norm2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.10.attn.proj.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.10.attn.proj.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.10.attn.qkv.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.10.attn.qkv.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.10.mlp.fc1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.10.mlp.fc1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.10.mlp.fc2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.10.mlp.fc2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.10.norm1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.10.norm1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.10.norm2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.10.norm2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.11.attn.proj.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.11.attn.proj.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.11.attn.qkv.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.11.attn.qkv.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.11.mlp.fc1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.11.mlp.fc1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.11.mlp.fc2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.11.mlp.fc2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.11.norm1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.11.norm1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.11.norm2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.11.norm2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.12.attn.proj.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.12.attn.proj.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.12.attn.qkv.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.12.attn.qkv.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.12.mlp.fc1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.12.mlp.fc1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.12.mlp.fc2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.12.mlp.fc2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.12.norm1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.12.norm1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.12.norm2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.12.norm2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.13.attn.proj.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.13.attn.proj.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.13.attn.qkv.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.13.attn.qkv.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.13.mlp.fc1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.13.mlp.fc1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.13.mlp.fc2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.13.mlp.fc2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.13.norm1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.13.norm1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.13.norm2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.13.norm2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.14.attn.proj.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.14.attn.proj.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.14.attn.qkv.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.14.attn.qkv.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.14.mlp.fc1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.14.mlp.fc1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.14.mlp.fc2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.14.mlp.fc2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.14.norm1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.14.norm1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.14.norm2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.14.norm2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.15.attn.proj.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.15.attn.proj.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.15.attn.qkv.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.15.attn.qkv.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.15.mlp.fc1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.15.mlp.fc1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.15.mlp.fc2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.15.mlp.fc2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.15.norm1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.15.norm1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.15.norm2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.15.norm2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.16.attn.proj.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.16.attn.proj.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.16.attn.qkv.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.16.attn.qkv.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.16.mlp.fc1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.16.mlp.fc1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.16.mlp.fc2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.16.mlp.fc2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.16.norm1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.16.norm1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.16.norm2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.16.norm2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.17.attn.proj.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.17.attn.proj.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.17.attn.qkv.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.17.attn.qkv.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.17.mlp.fc1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.17.mlp.fc1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.17.mlp.fc2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.17.mlp.fc2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.17.norm1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.17.norm1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.17.norm2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.17.norm2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.18.attn.proj.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.18.attn.proj.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.18.attn.qkv.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.18.attn.qkv.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.18.mlp.fc1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.18.mlp.fc1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.18.mlp.fc2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.18.mlp.fc2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.18.norm1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.18.norm1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.18.norm2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.18.norm2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.19.attn.proj.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.19.attn.proj.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.19.attn.qkv.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.19.attn.qkv.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.19.mlp.fc1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.19.mlp.fc1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.19.mlp.fc2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.19.mlp.fc2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.19.norm1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.19.norm1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.19.norm2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.19.norm2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.2.attn.proj.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.2.attn.proj.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.2.attn.qkv.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.2.attn.qkv.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.2.mlp.fc1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.2.mlp.fc1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.2.mlp.fc2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.2.mlp.fc2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.2.norm1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.2.norm1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.2.norm2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.2.norm2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.20.attn.proj.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.20.attn.proj.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.20.attn.qkv.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.20.attn.qkv.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.20.mlp.fc1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.20.mlp.fc1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.20.mlp.fc2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.20.mlp.fc2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.20.norm1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.20.norm1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.20.norm2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.20.norm2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.21.attn.proj.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.21.attn.proj.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.21.attn.qkv.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.21.attn.qkv.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.21.mlp.fc1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.21.mlp.fc1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.21.mlp.fc2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.21.mlp.fc2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.21.norm1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.21.norm1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.21.norm2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.21.norm2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.22.attn.proj.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.22.attn.proj.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.22.attn.qkv.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.22.attn.qkv.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.22.mlp.fc1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.22.mlp.fc1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.22.mlp.fc2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.22.mlp.fc2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.22.norm1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.22.norm1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.22.norm2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.22.norm2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.23.attn.proj.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.23.attn.proj.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.23.attn.qkv.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.23.attn.qkv.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.23.mlp.fc1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.23.mlp.fc1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.23.mlp.fc2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.23.mlp.fc2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.23.norm1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.23.norm1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.23.norm2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.23.norm2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.24.attn.proj.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.24.attn.proj.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.24.attn.qkv.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.24.attn.qkv.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.24.mlp.fc1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.24.mlp.fc1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.24.mlp.fc2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.24.mlp.fc2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.24.norm1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.24.norm1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.24.norm2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.24.norm2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.25.attn.proj.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.25.attn.proj.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.25.attn.qkv.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.25.attn.qkv.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.25.mlp.fc1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.25.mlp.fc1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.25.mlp.fc2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.25.mlp.fc2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.25.norm1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.25.norm1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.25.norm2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.25.norm2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.26.attn.proj.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.26.attn.proj.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.26.attn.qkv.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.26.attn.qkv.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.26.mlp.fc1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.26.mlp.fc1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.26.mlp.fc2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.26.mlp.fc2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.26.norm1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.26.norm1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.26.norm2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.26.norm2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.27.attn.proj.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.27.attn.proj.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.27.attn.qkv.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.27.attn.qkv.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.27.mlp.fc1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.27.mlp.fc1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.27.mlp.fc2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.27.mlp.fc2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.27.norm1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.27.norm1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.27.norm2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.27.norm2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.28.attn.proj.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.28.attn.proj.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.28.attn.qkv.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.28.attn.qkv.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.28.mlp.fc1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.28.mlp.fc1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.28.mlp.fc2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.28.mlp.fc2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.28.norm1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.28.norm1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.28.norm2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.28.norm2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.29.attn.proj.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.29.attn.proj.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.29.attn.qkv.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.29.attn.qkv.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.29.mlp.fc1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.29.mlp.fc1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.29.mlp.fc2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.29.mlp.fc2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.29.norm1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.29.norm1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.29.norm2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.29.norm2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.3.attn.proj.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.3.attn.proj.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.3.attn.qkv.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.3.attn.qkv.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.3.mlp.fc1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.3.mlp.fc1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.3.mlp.fc2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.3.mlp.fc2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.3.norm1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.3.norm1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.3.norm2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.3.norm2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.30.attn.proj.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.30.attn.proj.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.30.attn.qkv.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.30.attn.qkv.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.30.mlp.fc1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.30.mlp.fc1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.30.mlp.fc2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.30.mlp.fc2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.30.norm1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.30.norm1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.30.norm2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.30.norm2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.31.attn.proj.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.31.attn.proj.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.31.attn.qkv.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.31.attn.qkv.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.31.mlp.fc1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.31.mlp.fc1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.31.mlp.fc2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.31.mlp.fc2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.31.norm1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.31.norm1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.31.norm2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.31.norm2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.4.attn.proj.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.4.attn.proj.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.4.attn.qkv.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.4.attn.qkv.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.4.mlp.fc1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.4.mlp.fc1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.4.mlp.fc2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.4.mlp.fc2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.4.norm1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.4.norm1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.4.norm2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.4.norm2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.5.attn.proj.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.5.attn.proj.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.5.attn.qkv.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.5.attn.qkv.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.5.mlp.fc1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.5.mlp.fc1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.5.mlp.fc2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.5.mlp.fc2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.5.norm1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.5.norm1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.5.norm2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.5.norm2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.6.attn.proj.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.6.attn.proj.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.6.attn.qkv.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.6.attn.qkv.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.6.mlp.fc1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.6.mlp.fc1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.6.mlp.fc2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.6.mlp.fc2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.6.norm1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.6.norm1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.6.norm2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.6.norm2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.7.attn.proj.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.7.attn.proj.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.7.attn.qkv.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.7.attn.qkv.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.7.mlp.fc1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.7.mlp.fc1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.7.mlp.fc2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.7.mlp.fc2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.7.norm1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.7.norm1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.7.norm2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.7.norm2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.8.attn.proj.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.8.attn.proj.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.8.attn.qkv.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.8.attn.qkv.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.8.mlp.fc1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.8.mlp.fc1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.8.mlp.fc2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.8.mlp.fc2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.8.norm1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.8.norm1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.8.norm2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.8.norm2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.9.attn.proj.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.9.attn.proj.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.9.attn.qkv.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.9.attn.qkv.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.9.mlp.fc1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.9.mlp.fc1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.9.mlp.fc2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.9.mlp.fc2.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.9.norm1.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.9.norm1.weight": "model-00001-of-00004.safetensors",
+    "visual.blocks.9.norm2.bias": "model-00001-of-00004.safetensors",
+    "visual.blocks.9.norm2.weight": "model-00001-of-00004.safetensors",
+    "visual.merger.ln_q.bias": "model-00001-of-00004.safetensors",
+    "visual.merger.ln_q.weight": "model-00001-of-00004.safetensors",
+    "visual.merger.mlp.0.bias": "model-00001-of-00004.safetensors",
+    "visual.merger.mlp.0.weight": "model-00001-of-00004.safetensors",
+    "visual.merger.mlp.2.bias": "model-00001-of-00004.safetensors",
+    "visual.merger.mlp.2.weight": "model-00001-of-00004.safetensors",
+    "visual.patch_embed.proj.weight": "model-00001-of-00004.safetensors"
+  }
+}

ops_mm_embedding_v1.py

@@ -0,0 +1,309 @@
+import math
+from typing import List, Optional, TypeAlias, Union
+
+import torch
+import torch.nn as nn
+from PIL import Image
+from tqdm import tqdm
+from transformers import AutoModelForImageTextToText, AutoProcessor
+
+ImageInput: TypeAlias = Union[Image.Image, List[Image.Image]]
+BatchImageInput: TypeAlias = Union[List[Image.Image], List[List[Image.Image]]]
+
+
+class OpsMMEmbeddingV1(nn.Module):
+    def __init__(
+        self,
+        model_name: str,
+        device: str = "cuda",
+        max_length: Optional[int] = None,
+        attn_implementation: Optional[str] = None,
+    ):
+        super().__init__()
+        self.device = device
+        self.max_length = max_length
+        self.default_instruction = "You are a helpful assistant."
+        self.base_model = AutoModelForImageTextToText.from_pretrained(
+            model_name,
+            torch_dtype=torch.bfloat16,
+            low_cpu_mem_usage=True,
+            attn_implementation=attn_implementation,
+        ).to(self.device)
+
+        self.processor = AutoProcessor.from_pretrained(model_name, min_pixels=256 * 28 * 28, max_pixels=1280 * 28 * 28)
+        self.processor.tokenizer.padding_side = "left"
+        self.eval()
+
+    def encode_input(self, input):
+        hidden_states = self.base_model(**input, return_dict=True, output_hidden_states=True)
+        hidden_states = hidden_states.hidden_states[-1]
+        pooled_output = self._pooling(hidden_states)
+        return pooled_output
+
+    def _pooling(self, last_hidden_state):
+        batch_size = last_hidden_state.shape[0]
+        reps = last_hidden_state[torch.arange(batch_size), -1, :]
+        reps = torch.nn.functional.normalize(reps, p=2, dim=-1)
+        return reps
+
+    def _validate_instructions(
+        self,
+        texts: Optional[List[str]],
+        images: Optional[BatchImageInput],
+        instruction: Optional[Union[str, List[str]]],
+    ) -> List[str]:
+        """Validate and format instructions to match batch size"""
+        batch_size = max(len(x) if x is not None else 0 for x in [texts, images])
+
+        if instruction is None:
+            return [self.default_instruction] * batch_size
+
+        if isinstance(instruction, str):
+            return [instruction] * batch_size
+
+        if isinstance(instruction, list):
+            if len(instruction) != batch_size:
+                raise ValueError(f"Length of instruction list ({len(instruction)}) must match batch size ({batch_size}) when texts/images are provided")
+            return instruction
+
+        raise TypeError("instruction must be str, List[str] or None")
+
+    def _process_images(self, images: ImageInput) -> List[Image.Image]:
+        """Convert single image or list of images to processed format"""
+        if isinstance(images, Image.Image) or isinstance(images, str):
+            return [fetch_image(images)]
+        return [fetch_image(i) for i in images]
+
+    def embed(
+        self,
+        texts: Optional[List[str]] = None,
+        images: Optional[BatchImageInput] = None,
+        instruction: Optional[Union[str, List[str]]] = None,
+        **kwargs,
+    ) -> torch.Tensor:
+        """Generate embeddings for text, images, or combined inputs.
+
+        Args:
+            texts: List of text inputs (optional)
+            images: Can be:
+                - List[Image.Image]: Single image per input
+                - List[List[Image.Image]]: Multiple images per input
+            instruction: Instruction(s) for the model. Can be:
+                - None: use default instruction
+                - str: use same instruction for all inputs
+                - List[str]: per-input instructions (must match batch size)
+        """
+        if texts is None and images is None:
+            raise ValueError("Either texts or images must be provided")
+
+        instructions = self._validate_instructions(texts, images, instruction)
+
+        # Determine batch size
+        batch_size = len(texts) if texts is not None else len(images)  # type: ignore
+
+        input_texts, input_images = [], []
+        for i in range(batch_size):
+            text = texts[i] if texts is not None else None
+            image = images[i] if images is not None else None
+
+            input_str = ""
+            processed_image = None
+            if image is not None:
+                processed_image = self._process_images(image)
+                input_str += "<|vision_start|><|image_pad|><|vision_end|>" * len(processed_image)
+
+            if text is not None:
+                input_str += text
+
+            msg = f"<|im_start|>system\n{instructions[i]}<|im_end|>\n<|im_start|>user\n{input_str}<|im_end|>\n<|im_start|>assistant\n<|endoftext|>"
+
+            input_texts.append(msg)
+            input_images.append(processed_image)
+
+        # Only pass to processor if we actually have images
+        processed_images = input_images if any(img is not None for img in input_images) else None
+
+        inputs = self.processor(
+            text=input_texts,
+            images=processed_images,
+            padding=True,
+            truncation=True,
+            max_length=self.max_length,
+            return_tensors="pt",
+        )
+        inputs = {k: v.to(self.device) for k, v in inputs.items()}
+
+        with torch.inference_mode():
+            embeddings = self.encode_input(inputs)
+
+        return embeddings
+
+    def get_text_embeddings(
+        self,
+        texts: List[str],
+        instruction: Optional[Union[str, List[str]]] = None,
+        **kwargs,
+    ) -> torch.Tensor:
+        """Convenience method for text-only embeddings"""
+        return self.get_fused_embeddings(texts=texts, instruction=instruction, **kwargs)
+
+    def get_image_embeddings(
+        self,
+        images: BatchImageInput,
+        instruction: Optional[Union[str, List[str]]] = None,
+        **kwargs,
+    ) -> torch.Tensor:
+        """Convenience method for image-only embeddings.
+
+        Args:
+            images: Can be:
+                - List[Image.Image]: Single image per input
+                - List[List[Image.Image]]: Multiple images per input
+        """
+        return self.get_fused_embeddings(images=images, instruction=instruction, **kwargs)
+
+    def get_fused_embeddings(
+        self,
+        texts: Optional[List[str]] = None,
+        images: Optional[BatchImageInput] = None,
+        instruction: Optional[Union[str, List[str]]] = None,
+        batch_size: int = 8,
+        show_progress: bool = True,
+        **kwargs,
+    ) -> torch.Tensor:
+        """Batch processing for large collections of texts/images.
+
+        Args:
+            texts: List of text inputs (optional)
+            images: Can be:
+                - List[Image.Image]: Single image per input
+                - List[List[Image.Image]]: Multiple images per input
+            instruction: Instruction(s) for the model
+            batch_size: Number of items to process at once
+            show_progress: Whether to display progress bar
+        """
+
+        if texts is None and images is None:
+            raise ValueError("Either texts or images must be provided")
+
+        total_items = len(texts) if texts is not None else len(images)  # type: ignore
+        num_batches = math.ceil(total_items / batch_size)
+
+        all_embeddings = []
+        progress = tqdm(total=num_batches, disable=not show_progress, desc="Processing")
+
+        for i in range(0, total_items, batch_size):
+            batch_texts = texts[i : i + batch_size] if texts is not None else None
+            batch_images = images[i : i + batch_size] if images is not None else None
+            batch_emb = self.embed(texts=batch_texts, images=batch_images, instruction=instruction)
+
+            all_embeddings.append(batch_emb.cpu())
+            progress.update(1)
+
+        progress.close()
+        return torch.cat(all_embeddings, dim=0).to(self.device)
+
+    def forward(self, **inputs) -> torch.Tensor:
+        """Alias for encode_input"""
+        return self.encode_input(inputs)
+
+
+### Modified from qwen_vl_utils.vision_process.py
+import base64
+import logging
+import math
+from io import BytesIO
+
+import requests
+
+IMAGE_FACTOR = 28
+MIN_PIXELS = 256 * 28 * 28
+MAX_PIXELS = 1280 * 28 * 28
+MAX_RATIO = 200
+
+
+def round_by_factor(number: int, factor: int) -> int:
+    """Returns the closest integer to 'number' that is divisible by 'factor'."""
+    return round(number / factor) * factor
+
+
+def ceil_by_factor(number: int | float, factor: int) -> int:
+    """Returns the smallest integer greater than or equal to 'number' that is divisible by 'factor'."""
+    return math.ceil(number / factor) * factor
+
+
+def floor_by_factor(number: int | float, factor: int) -> int:
+    """Returns the largest integer less than or equal to 'number' that is divisible by 'factor'."""
+    return math.floor(number / factor) * factor
+
+
+def smart_resize(
+    height: int,
+    width: int,
+    factor: int = IMAGE_FACTOR,
+    min_pixels: int = MIN_PIXELS,
+    max_pixels: int = MAX_PIXELS,
+) -> tuple[int, int]:
+    """
+    Rescales the image so that the following conditions are met:
+    1. Both dimensions (height and width) are divisible by 'factor'.
+    2. The total number of pixels is within the range ['min_pixels', 'max_pixels'].
+    3. The aspect ratio of the image is maintained as closely as possible.
+    """
+    h_bar = max(factor, round_by_factor(height, factor))
+    w_bar = max(factor, round_by_factor(width, factor))
+    if h_bar * w_bar > max_pixels:
+        beta = math.sqrt((height * width) / max_pixels)
+        h_bar = floor_by_factor(height / beta, factor)
+        w_bar = floor_by_factor(width / beta, factor)
+    elif h_bar * w_bar < min_pixels:
+        beta = math.sqrt(min_pixels / (height * width))
+        h_bar = ceil_by_factor(height * beta, factor)
+        w_bar = ceil_by_factor(width * beta, factor)
+
+    if max(h_bar, w_bar) / min(h_bar, w_bar) > MAX_RATIO:
+        logging.warning(f"Absolute aspect ratio must be smaller than {MAX_RATIO}, got {max(h_bar, w_bar) / min(h_bar, w_bar)}")
+        if h_bar > w_bar:
+            h_bar = w_bar * MAX_RATIO
+        else:
+            w_bar = h_bar * MAX_RATIO
+    return h_bar, w_bar
+
+
+def fetch_image(
+    image: str | Image.Image,
+    size_factor: int = IMAGE_FACTOR,
+    min_pixels: int = MIN_PIXELS,
+    max_pixels: int = MAX_PIXELS,
+) -> Image.Image:
+    image_obj = None
+    if isinstance(image, Image.Image):
+        image_obj = image
+    elif image.startswith("http://") or image.startswith("https://"):
+        image_obj = Image.open(requests.get(image, stream=True).raw)  # type: ignore
+    elif image.startswith("file://"):
+        image_obj = Image.open(image[7:])
+    elif image.startswith("data:image"):
+        if "base64," in image:
+            _, base64_data = image.split("base64,", 1)
+            data = base64.b64decode(base64_data)
+            image_obj = Image.open(BytesIO(data))
+    else:
+        image_obj = Image.open(image)
+    if image_obj is None:
+        raise ValueError(f"Unrecognized image input, support local path, http url, base64 and PIL.Image, got {image}")
+    image = image_obj.convert("RGB")
+    width, height = image.size
+    resized_height, resized_width = smart_resize(
+        height,
+        width,
+        factor=size_factor,
+        min_pixels=min_pixels,
+        max_pixels=max_pixels,
+    )
+    image = image.resize((resized_width, resized_height))
+
+    return image
+
+
+###

preprocessor_config.json

@@ -0,0 +1,29 @@
+{
+  "do_convert_rgb": true,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.48145466,
+    0.4578275,
+    0.40821073
+  ],
+  "image_processor_type": "Qwen2VLImageProcessor",
+  "image_std": [
+    0.26862954,
+    0.26130258,
+    0.27577711
+  ],
+  "max_pixels": 12845056,
+  "merge_size": 2,
+  "min_pixels": 3136,
+  "patch_size": 14,
+  "processor_class": "Qwen2VLProcessor",
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098,
+  "size": {
+    "longest_edge": 12845056,
+    "shortest_edge": 3136
+  },
+  "temporal_patch_size": 2
+}

score/Ops-MM-embedding-v1-7B.json

@@ -0,0 +1,2289 @@
+{
+    "metadata": {
+        "model_name": "Ops-MM-embedding-v1-7B",
+        "model_backbone": "Qwen2-VL-7B",
+        "model_size": 8.29,
+        "embedding_dimension": 3584,
+        "max_length_tokens": null,
+        "model_release_date": "2025-07-03",
+        "data_source": "OpenSearch-AI",
+        "url": "https://huggingface.co/OpenSearch-AI/Ops-MM-embedding-v1-7B",
+        "report_generated_date": "2025-07-02T23:11:24.116227"
+    },
+    "metrics": {
+        "image": {
+            "ImageNet-1K": {
+                "hit@1": 0.811,
+                "hit@5": 0.964,
+                "hit@10": 0.982,
+                "ndcg_linear@1": 0.811,
+                "ndcg_linear@5": 0.8972089841991893,
+                "ndcg_linear@10": 0.903203094426922,
+                "ndcg_exponential@1": 0.811,
+                "ndcg_exponential@5": 0.8972089841991893,
+                "ndcg_exponential@10": 0.903203094426922,
+                "precision@1": 0.811,
+                "precision@5": 0.1928,
+                "precision@10": 0.09820000000000002,
+                "recall@1": 0.811,
+                "recall@5": 0.964,
+                "recall@10": 0.982,
+                "f1@1": 0.811,
+                "f1@5": 0.32133333333333336,
+                "f1@10": 0.17854545454545453,
+                "map@1": 0.811,
+                "map@5": 0.8745833333333333,
+                "map@10": 0.8771603174603174,
+                "mrr@1": 0.811,
+                "mrr@5": 0.8745833333333333,
+                "mrr@10": 0.8771603174603174,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "N24News": {
+                "hit@1": 0.821,
+                "hit@5": 0.974,
+                "hit@10": 0.998,
+                "ndcg_linear@1": 0.821,
+                "ndcg_linear@5": 0.908003904347735,
+                "ndcg_linear@10": 0.9159637568629064,
+                "ndcg_exponential@1": 0.821,
+                "ndcg_exponential@5": 0.908003904347735,
+                "ndcg_exponential@10": 0.9159637568629064,
+                "precision@1": 0.821,
+                "precision@5": 0.1948,
+                "precision@10": 0.09980000000000001,
+                "recall@1": 0.821,
+                "recall@5": 0.974,
+                "recall@10": 0.998,
+                "f1@1": 0.821,
+                "f1@5": 0.3246666666666666,
+                "f1@10": 0.18145454545454545,
+                "map@1": 0.821,
+                "map@5": 0.8855333333333333,
+                "map@10": 0.8889365079365079,
+                "mrr@1": 0.821,
+                "mrr@5": 0.8855333333333333,
+                "mrr@10": 0.8889365079365079,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "HatefulMemes": {
+                "hit@1": 0.757,
+                "hit@5": 1.0,
+                "hit@10": 1.0,
+                "ndcg_linear@1": 0.757,
+                "ndcg_linear@5": 0.9103159301178643,
+                "ndcg_linear@10": 0.9103159301178643,
+                "ndcg_exponential@1": 0.757,
+                "ndcg_exponential@5": 0.9103159301178643,
+                "ndcg_exponential@10": 0.9103159301178643,
+                "precision@1": 0.757,
+                "precision@5": 0.20000000000000004,
+                "precision@10": 0.10000000000000002,
+                "recall@1": 0.757,
+                "recall@5": 1.0,
+                "recall@10": 1.0,
+                "f1@1": 0.757,
+                "f1@5": 0.3333333333333333,
+                "f1@10": 0.18181818181818182,
+                "map@1": 0.757,
+                "map@5": 0.8785,
+                "map@10": 0.8785,
+                "mrr@1": 0.757,
+                "mrr@5": 0.8785,
+                "mrr@10": 0.8785,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "VOC2007": {
+                "hit@1": 0.848,
+                "hit@5": 0.977,
+                "hit@10": 0.992,
+                "ndcg_linear@1": 0.848,
+                "ndcg_linear@5": 0.9241752272291508,
+                "ndcg_linear@10": 0.9290490649008084,
+                "ndcg_exponential@1": 0.848,
+                "ndcg_exponential@5": 0.9241752272291508,
+                "ndcg_exponential@10": 0.9290490649008084,
+                "precision@1": 0.848,
+                "precision@5": 0.19540000000000005,
+                "precision@10": 0.09920000000000001,
+                "recall@1": 0.848,
+                "recall@5": 0.977,
+                "recall@10": 0.992,
+                "f1@1": 0.848,
+                "f1@5": 0.3256666666666666,
+                "f1@10": 0.18036363636363634,
+                "map@1": 0.848,
+                "map@5": 0.9059333333333334,
+                "map@10": 0.9079559523809524,
+                "mrr@1": 0.848,
+                "mrr@5": 0.9059333333333334,
+                "mrr@10": 0.9079559523809524,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "SUN397": {
+                "hit@1": 0.81,
+                "hit@5": 0.975,
+                "hit@10": 0.993,
+                "ndcg_linear@1": 0.81,
+                "ndcg_linear@5": 0.9044714381294062,
+                "ndcg_linear@10": 0.9104367712161627,
+                "ndcg_exponential@1": 0.81,
+                "ndcg_exponential@5": 0.9044714381294062,
+                "ndcg_exponential@10": 0.9104367712161627,
+                "precision@1": 0.81,
+                "precision@5": 0.19500000000000003,
+                "precision@10": 0.09930000000000001,
+                "recall@1": 0.81,
+                "recall@5": 0.975,
+                "recall@10": 0.993,
+                "f1@1": 0.81,
+                "f1@5": 0.325,
+                "f1@10": 0.1805454545454545,
+                "map@1": 0.81,
+                "map@5": 0.8804500000000001,
+                "map@10": 0.8829964285714286,
+                "mrr@1": 0.81,
+                "mrr@5": 0.8804500000000001,
+                "mrr@10": 0.8829964285714286,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "Place365": {
+                "hit@1": 0.456,
+                "hit@5": 0.742,
+                "hit@10": 0.832,
+                "ndcg_linear@1": 0.456,
+                "ndcg_linear@5": 0.612464933944354,
+                "ndcg_linear@10": 0.6421212605535749,
+                "ndcg_exponential@1": 0.456,
+                "ndcg_exponential@5": 0.612464933944354,
+                "ndcg_exponential@10": 0.6421212605535749,
+                "precision@1": 0.456,
+                "precision@5": 0.14839999999999998,
+                "precision@10": 0.08320000000000002,
+                "recall@1": 0.456,
+                "recall@5": 0.742,
+                "recall@10": 0.832,
+                "f1@1": 0.456,
+                "f1@5": 0.24733333333333338,
+                "f1@10": 0.15127272727272728,
+                "map@1": 0.456,
+                "map@5": 0.5689833333333334,
+                "map@10": 0.5815511904761905,
+                "mrr@1": 0.456,
+                "mrr@5": 0.5689833333333334,
+                "mrr@10": 0.5815511904761905,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "ImageNet-A": {
+                "hit@1": 0.578,
+                "hit@5": 0.814,
+                "hit@10": 0.873,
+                "ndcg_linear@1": 0.578,
+                "ndcg_linear@5": 0.7067527294684098,
+                "ndcg_linear@10": 0.7260494542589042,
+                "ndcg_exponential@1": 0.578,
+                "ndcg_exponential@5": 0.7067527294684098,
+                "ndcg_exponential@10": 0.7260494542589042,
+                "precision@1": 0.578,
+                "precision@5": 0.1628,
+                "precision@10": 0.08730000000000002,
+                "recall@1": 0.578,
+                "recall@5": 0.814,
+                "recall@10": 0.873,
+                "f1@1": 0.578,
+                "f1@5": 0.27133333333333337,
+                "f1@10": 0.15872727272727272,
+                "map@1": 0.578,
+                "map@5": 0.67075,
+                "map@10": 0.6788373015873016,
+                "mrr@1": 0.578,
+                "mrr@5": 0.67075,
+                "mrr@10": 0.6788373015873016,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "ImageNet-R": {
+                "hit@1": 0.902,
+                "hit@5": 0.981,
+                "hit@10": 0.992,
+                "ndcg_linear@1": 0.902,
+                "ndcg_linear@5": 0.9490524502565482,
+                "ndcg_linear@10": 0.9525844230009524,
+                "ndcg_exponential@1": 0.902,
+                "ndcg_exponential@5": 0.9490524502565482,
+                "ndcg_exponential@10": 0.9525844230009524,
+                "precision@1": 0.902,
+                "precision@5": 0.19620000000000004,
+                "precision@10": 0.09920000000000001,
+                "recall@1": 0.902,
+                "recall@5": 0.981,
+                "recall@10": 0.992,
+                "f1@1": 0.902,
+                "f1@5": 0.327,
+                "f1@10": 0.18036363636363634,
+                "map@1": 0.902,
+                "map@5": 0.9379833333333334,
+                "map@10": 0.9394273809523809,
+                "mrr@1": 0.902,
+                "mrr@5": 0.9379833333333334,
+                "mrr@10": 0.9394273809523809,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "ObjectNet": {
+                "hit@1": 0.697,
+                "hit@5": 0.876,
+                "hit@10": 0.912,
+                "ndcg_linear@1": 0.697,
+                "ndcg_linear@5": 0.7998091251553962,
+                "ndcg_linear@10": 0.8114293964429173,
+                "ndcg_exponential@1": 0.697,
+                "ndcg_exponential@5": 0.7998091251553962,
+                "ndcg_exponential@10": 0.8114293964429173,
+                "precision@1": 0.697,
+                "precision@5": 0.17520000000000002,
+                "precision@10": 0.09120000000000002,
+                "recall@1": 0.697,
+                "recall@5": 0.876,
+                "recall@10": 0.912,
+                "f1@1": 0.697,
+                "f1@5": 0.292,
+                "f1@10": 0.1658181818181818,
+                "map@1": 0.697,
+                "map@5": 0.7738333333333334,
+                "map@10": 0.778622619047619,
+                "mrr@1": 0.697,
+                "mrr@5": 0.7738333333333334,
+                "mrr@10": 0.778622619047619,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "Country211": {
+                "hit@1": 0.285,
+                "hit@5": 0.506,
+                "hit@10": 0.609,
+                "ndcg_linear@1": 0.285,
+                "ndcg_linear@5": 0.40328589269051235,
+                "ndcg_linear@10": 0.4360567158341879,
+                "ndcg_exponential@1": 0.285,
+                "ndcg_exponential@5": 0.40328589269051235,
+                "ndcg_exponential@10": 0.4360567158341879,
+                "precision@1": 0.285,
+                "precision@5": 0.10119999999999998,
+                "precision@10": 0.06089999999999999,
+                "recall@1": 0.285,
+                "recall@5": 0.506,
+                "recall@10": 0.609,
+                "f1@1": 0.285,
+                "f1@5": 0.1686666666666667,
+                "f1@10": 0.11072727272727274,
+                "map@1": 0.285,
+                "map@5": 0.36905,
+                "map@10": 0.3822527777777777,
+                "mrr@1": 0.285,
+                "mrr@5": 0.36905,
+                "mrr@10": 0.3822527777777777,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "OK-VQA": {
+                "hit@1": 0.706,
+                "hit@5": 0.935,
+                "hit@10": 0.97,
+                "ndcg_linear@1": 0.706,
+                "ndcg_linear@5": 0.8344075880155006,
+                "ndcg_linear@10": 0.8459473006000523,
+                "ndcg_exponential@1": 0.706,
+                "ndcg_exponential@5": 0.8344075880155006,
+                "ndcg_exponential@10": 0.8459473006000523,
+                "precision@1": 0.706,
+                "precision@5": 0.18700000000000003,
+                "precision@10": 0.097,
+                "recall@1": 0.706,
+                "recall@5": 0.935,
+                "recall@10": 0.97,
+                "f1@1": 0.706,
+                "f1@5": 0.3116666666666667,
+                "f1@10": 0.17636363636363633,
+                "map@1": 0.706,
+                "map@5": 0.8004,
+                "map@10": 0.805288492063492,
+                "mrr@1": 0.706,
+                "mrr@5": 0.8004,
+                "mrr@10": 0.805288492063492,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "A-OKVQA": {
+                "hit@1": 0.6,
+                "hit@5": 0.885,
+                "hit@10": 0.941,
+                "ndcg_linear@1": 0.6,
+                "ndcg_linear@5": 0.7567001853131614,
+                "ndcg_linear@10": 0.774920654838386,
+                "ndcg_exponential@1": 0.6,
+                "ndcg_exponential@5": 0.7567001853131614,
+                "ndcg_exponential@10": 0.774920654838386,
+                "precision@1": 0.6,
+                "precision@5": 0.177,
+                "precision@10": 0.09410000000000002,
+                "recall@1": 0.6,
+                "recall@5": 0.885,
+                "recall@10": 0.941,
+                "f1@1": 0.6,
+                "f1@5": 0.295,
+                "f1@10": 0.17109090909090907,
+                "map@1": 0.6,
+                "map@5": 0.7134833333333334,
+                "map@10": 0.7210642857142857,
+                "mrr@1": 0.6,
+                "mrr@5": 0.7134833333333334,
+                "mrr@10": 0.7210642857142857,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "DocVQA": {
+                "hit@1": 0.947,
+                "hit@5": 0.995,
+                "hit@10": 0.996,
+                "ndcg_linear@1": 0.947,
+                "ndcg_linear@5": 0.9755232833635987,
+                "ndcg_linear@10": 0.9758794905507068,
+                "ndcg_exponential@1": 0.947,
+                "ndcg_exponential@5": 0.9755232833635987,
+                "ndcg_exponential@10": 0.9758794905507068,
+                "precision@1": 0.947,
+                "precision@5": 0.19900000000000004,
+                "precision@10": 0.09960000000000002,
+                "recall@1": 0.947,
+                "recall@5": 0.995,
+                "recall@10": 0.996,
+                "f1@1": 0.947,
+                "f1@5": 0.3316666666666666,
+                "f1@10": 0.18109090909090905,
+                "map@1": 0.947,
+                "map@5": 0.9687833333333334,
+                "map@10": 0.9689500000000001,
+                "mrr@1": 0.947,
+                "mrr@5": 0.9687833333333334,
+                "mrr@10": 0.9689500000000001,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "InfographicsVQA": {
+                "hit@1": 0.736,
+                "hit@5": 0.906,
+                "hit@10": 0.932,
+                "ndcg_linear@1": 0.736,
+                "ndcg_linear@5": 0.8277407513883303,
+                "ndcg_linear@10": 0.8362488999302439,
+                "ndcg_exponential@1": 0.736,
+                "ndcg_exponential@5": 0.8277407513883303,
+                "ndcg_exponential@10": 0.8362488999302439,
+                "precision@1": 0.736,
+                "precision@5": 0.18120000000000003,
+                "precision@10": 0.09320000000000002,
+                "recall@1": 0.736,
+                "recall@5": 0.906,
+                "recall@10": 0.932,
+                "f1@1": 0.736,
+                "f1@5": 0.302,
+                "f1@10": 0.16945454545454544,
+                "map@1": 0.736,
+                "map@5": 0.8015666666666666,
+                "map@10": 0.805140873015873,
+                "mrr@1": 0.736,
+                "mrr@5": 0.8015666666666666,
+                "mrr@10": 0.805140873015873,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "ChartQA": {
+                "hit@1": 0.652,
+                "hit@5": 0.81,
+                "hit@10": 0.873,
+                "ndcg_linear@1": 0.652,
+                "ndcg_linear@5": 0.7383237284140184,
+                "ndcg_linear@10": 0.758789100394747,
+                "ndcg_exponential@1": 0.652,
+                "ndcg_exponential@5": 0.7383237284140184,
+                "ndcg_exponential@10": 0.758789100394747,
+                "precision@1": 0.652,
+                "precision@5": 0.162,
+                "precision@10": 0.08730000000000002,
+                "recall@1": 0.652,
+                "recall@5": 0.81,
+                "recall@10": 0.873,
+                "f1@1": 0.652,
+                "f1@5": 0.27,
+                "f1@10": 0.15872727272727274,
+                "map@1": 0.652,
+                "map@5": 0.7142833333333333,
+                "map@10": 0.7227880952380953,
+                "mrr@1": 0.652,
+                "mrr@5": 0.7142833333333333,
+                "mrr@10": 0.7227880952380953,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "Visual7W": {
+                "hit@1": 0.584,
+                "hit@5": 0.904,
+                "hit@10": 0.951,
+                "ndcg_linear@1": 0.584,
+                "ndcg_linear@5": 0.7607038779643807,
+                "ndcg_linear@10": 0.7762948323430009,
+                "ndcg_exponential@1": 0.584,
+                "ndcg_exponential@5": 0.7607038779643807,
+                "ndcg_exponential@10": 0.7762948323430009,
+                "precision@1": 0.584,
+                "precision@5": 0.18080000000000002,
+                "precision@10": 0.09510000000000002,
+                "recall@1": 0.584,
+                "recall@5": 0.904,
+                "recall@10": 0.951,
+                "f1@1": 0.584,
+                "f1@5": 0.3013333333333334,
+                "f1@10": 0.1729090909090909,
+                "map@1": 0.584,
+                "map@5": 0.7124833333333334,
+                "map@10": 0.7191464285714286,
+                "mrr@1": 0.584,
+                "mrr@5": 0.7124833333333334,
+                "mrr@10": 0.7191464285714286,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "ScienceQA": {
+                "hit@1": 0.499,
+                "hit@5": 0.809,
+                "hit@10": 0.911,
+                "ndcg_linear@1": 0.499,
+                "ndcg_linear@5": 0.6739528358410788,
+                "ndcg_linear@10": 0.7062005471589817,
+                "ndcg_exponential@1": 0.499,
+                "ndcg_exponential@5": 0.6739528358410788,
+                "ndcg_exponential@10": 0.7062005471589817,
+                "precision@1": 0.499,
+                "precision@5": 0.1618,
+                "precision@10": 0.09110000000000001,
+                "recall@1": 0.499,
+                "recall@5": 0.809,
+                "recall@10": 0.911,
+                "f1@1": 0.499,
+                "f1@5": 0.26966666666666667,
+                "f1@10": 0.16563636363636364,
+                "map@1": 0.499,
+                "map@5": 0.6281666666666667,
+                "map@10": 0.641040873015873,
+                "mrr@1": 0.499,
+                "mrr@5": 0.6281666666666667,
+                "mrr@10": 0.641040873015873,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "VizWiz": {
+                "hit@1": 0.566,
+                "hit@5": 0.719,
+                "hit@10": 0.751,
+                "ndcg_linear@1": 0.566,
+                "ndcg_linear@5": 0.6514422712364567,
+                "ndcg_linear@10": 0.6616436437103389,
+                "ndcg_exponential@1": 0.566,
+                "ndcg_exponential@5": 0.6514422712364567,
+                "ndcg_exponential@10": 0.6616436437103389,
+                "precision@1": 0.566,
+                "precision@5": 0.14379999999999998,
+                "precision@10": 0.07510000000000001,
+                "recall@1": 0.566,
+                "recall@5": 0.719,
+                "recall@10": 0.751,
+                "f1@1": 0.566,
+                "f1@5": 0.23966666666666672,
+                "f1@10": 0.13654545454545455,
+                "map@1": 0.566,
+                "map@5": 0.6285833333333333,
+                "map@10": 0.6327095238095238,
+                "mrr@1": 0.566,
+                "mrr@5": 0.6285833333333333,
+                "mrr@10": 0.6327095238095238,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "GQA": {
+                "hit@1": 0.799,
+                "hit@5": 0.946,
+                "hit@10": 0.972,
+                "ndcg_linear@1": 0.799,
+                "ndcg_linear@5": 0.8811134365876296,
+                "ndcg_linear@10": 0.8897748874138558,
+                "ndcg_exponential@1": 0.799,
+                "ndcg_exponential@5": 0.8811134365876296,
+                "ndcg_exponential@10": 0.8897748874138558,
+                "precision@1": 0.799,
+                "precision@5": 0.1892,
+                "precision@10": 0.09720000000000002,
+                "recall@1": 0.799,
+                "recall@5": 0.946,
+                "recall@10": 0.972,
+                "f1@1": 0.799,
+                "f1@5": 0.31533333333333335,
+                "f1@10": 0.17672727272727268,
+                "map@1": 0.799,
+                "map@5": 0.8592000000000001,
+                "map@10": 0.8629210317460316,
+                "mrr@1": 0.799,
+                "mrr@5": 0.8592000000000001,
+                "mrr@10": 0.8629210317460316,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "TextVQA": {
+                "hit@1": 0.869,
+                "hit@5": 0.926,
+                "hit@10": 0.939,
+                "ndcg_linear@1": 0.869,
+                "ndcg_linear@5": 0.9007951815016322,
+                "ndcg_linear@10": 0.9049302047486061,
+                "ndcg_exponential@1": 0.869,
+                "ndcg_exponential@5": 0.9007951815016322,
+                "ndcg_exponential@10": 0.9049302047486061,
+                "precision@1": 0.869,
+                "precision@5": 0.18520000000000006,
+                "precision@10": 0.09390000000000001,
+                "recall@1": 0.869,
+                "recall@5": 0.926,
+                "recall@10": 0.939,
+                "f1@1": 0.869,
+                "f1@5": 0.3086666666666667,
+                "f1@10": 0.1707272727272727,
+                "map@1": 0.869,
+                "map@5": 0.8922833333333333,
+                "map@10": 0.8939476190476191,
+                "mrr@1": 0.869,
+                "mrr@5": 0.8922833333333333,
+                "mrr@10": 0.8939476190476191,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "VisDial": {
+                "hit@1": 0.818,
+                "hit@5": 0.96,
+                "hit@10": 0.981,
+                "ndcg_linear@1": 0.818,
+                "ndcg_linear@5": 0.8981579586925184,
+                "ndcg_linear@10": 0.9050024514275745,
+                "ndcg_exponential@1": 0.818,
+                "ndcg_exponential@5": 0.8981579586925184,
+                "ndcg_exponential@10": 0.9050024514275745,
+                "precision@1": 0.818,
+                "precision@5": 0.192,
+                "precision@10": 0.09810000000000002,
+                "recall@1": 0.818,
+                "recall@5": 0.96,
+                "recall@10": 0.981,
+                "f1@1": 0.818,
+                "f1@5": 0.32,
+                "f1@10": 0.17836363636363634,
+                "map@1": 0.818,
+                "map@5": 0.8771666666666667,
+                "map@10": 0.8800242063492063,
+                "mrr@1": 0.818,
+                "mrr@5": 0.8771666666666667,
+                "mrr@10": 0.8800242063492063,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "CIRR": {
+                "hit@1": 0.552,
+                "hit@5": 0.911,
+                "hit@10": 0.955,
+                "ndcg_linear@1": 0.552,
+                "ndcg_linear@5": 0.7544132207910468,
+                "ndcg_linear@10": 0.7687461843165859,
+                "ndcg_exponential@1": 0.552,
+                "ndcg_exponential@5": 0.7544132207910468,
+                "ndcg_exponential@10": 0.7687461843165859,
+                "precision@1": 0.552,
+                "precision@5": 0.18220000000000003,
+                "precision@10": 0.09550000000000002,
+                "recall@1": 0.552,
+                "recall@5": 0.911,
+                "recall@10": 0.955,
+                "f1@1": 0.552,
+                "f1@5": 0.30366666666666664,
+                "f1@10": 0.17363636363636362,
+                "map@1": 0.552,
+                "map@5": 0.7012666666666667,
+                "map@10": 0.7072380952380952,
+                "mrr@1": 0.552,
+                "mrr@5": 0.7012666666666667,
+                "mrr@10": 0.7072380952380952,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "VisualNews_t2i": {
+                "hit@1": 0.801,
+                "hit@5": 0.907,
+                "hit@10": 0.938,
+                "ndcg_linear@1": 0.801,
+                "ndcg_linear@5": 0.8594919255925174,
+                "ndcg_linear@10": 0.8692591058077485,
+                "ndcg_exponential@1": 0.801,
+                "ndcg_exponential@5": 0.8594919255925174,
+                "ndcg_exponential@10": 0.8692591058077485,
+                "precision@1": 0.801,
+                "precision@5": 0.18140000000000003,
+                "precision@10": 0.09380000000000001,
+                "recall@1": 0.801,
+                "recall@5": 0.907,
+                "recall@10": 0.938,
+                "f1@1": 0.801,
+                "f1@5": 0.3023333333333334,
+                "f1@10": 0.17054545454545453,
+                "map@1": 0.801,
+                "map@5": 0.8435,
+                "map@10": 0.8473805555555555,
+                "mrr@1": 0.801,
+                "mrr@5": 0.8435,
+                "mrr@10": 0.8473805555555555,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "VisualNews_i2t": {
+                "hit@1": 0.843,
+                "hit@5": 0.936,
+                "hit@10": 0.953,
+                "ndcg_linear@1": 0.843,
+                "ndcg_linear@5": 0.8943970763350075,
+                "ndcg_linear@10": 0.8998863523941975,
+                "ndcg_exponential@1": 0.843,
+                "ndcg_exponential@5": 0.8943970763350075,
+                "ndcg_exponential@10": 0.8998863523941975,
+                "precision@1": 0.843,
+                "precision@5": 0.18720000000000003,
+                "precision@10": 0.09530000000000001,
+                "recall@1": 0.843,
+                "recall@5": 0.936,
+                "recall@10": 0.953,
+                "f1@1": 0.843,
+                "f1@5": 0.312,
+                "f1@10": 0.17327272727272724,
+                "map@1": 0.843,
+                "map@5": 0.8803666666666666,
+                "map@10": 0.8826269841269841,
+                "mrr@1": 0.843,
+                "mrr@5": 0.8803666666666666,
+                "mrr@10": 0.8826269841269841,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "MSCOCO_t2i": {
+                "hit@1": 0.793,
+                "hit@5": 0.949,
+                "hit@10": 0.978,
+                "ndcg_linear@1": 0.793,
+                "ndcg_linear@5": 0.8803368672984685,
+                "ndcg_linear@10": 0.8896818613565582,
+                "ndcg_exponential@1": 0.793,
+                "ndcg_exponential@5": 0.8803368672984685,
+                "ndcg_exponential@10": 0.8896818613565582,
+                "precision@1": 0.793,
+                "precision@5": 0.18980000000000005,
+                "precision@10": 0.09780000000000001,
+                "recall@1": 0.793,
+                "recall@5": 0.949,
+                "recall@10": 0.978,
+                "f1@1": 0.793,
+                "f1@5": 0.31633333333333336,
+                "f1@10": 0.17781818181818182,
+                "map@1": 0.793,
+                "map@5": 0.8571333333333333,
+                "map@10": 0.860975,
+                "mrr@1": 0.793,
+                "mrr@5": 0.8571333333333333,
+                "mrr@10": 0.860975,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "MSCOCO_i2t": {
+                "hit@1": 0.721,
+                "hit@5": 0.943,
+                "hit@10": 0.98,
+                "ndcg_linear@1": 0.721,
+                "ndcg_linear@5": 0.8402135232694316,
+                "ndcg_linear@10": 0.8526142772096339,
+                "ndcg_exponential@1": 0.721,
+                "ndcg_exponential@5": 0.8402135232694316,
+                "ndcg_exponential@10": 0.8526142772096339,
+                "precision@1": 0.721,
+                "precision@5": 0.18860000000000002,
+                "precision@10": 0.09800000000000002,
+                "recall@1": 0.721,
+                "recall@5": 0.943,
+                "recall@10": 0.98,
+                "f1@1": 0.721,
+                "f1@5": 0.3143333333333333,
+                "f1@10": 0.17818181818181816,
+                "map@1": 0.721,
+                "map@5": 0.8058666666666666,
+                "map@10": 0.811238492063492,
+                "mrr@1": 0.721,
+                "mrr@5": 0.8058666666666666,
+                "mrr@10": 0.811238492063492,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "NIGHTS": {
+                "hit@1": 0.662,
+                "hit@5": 0.977,
+                "hit@10": 0.997,
+                "ndcg_linear@1": 0.662,
+                "ndcg_linear@5": 0.8428332649702934,
+                "ndcg_linear@10": 0.8493732941342598,
+                "ndcg_exponential@1": 0.662,
+                "ndcg_exponential@5": 0.8428332649702934,
+                "ndcg_exponential@10": 0.8493732941342598,
+                "precision@1": 0.662,
+                "precision@5": 0.19540000000000005,
+                "precision@10": 0.09970000000000001,
+                "recall@1": 0.662,
+                "recall@5": 0.977,
+                "recall@10": 0.997,
+                "f1@1": 0.662,
+                "f1@5": 0.3256666666666666,
+                "f1@10": 0.18127272727272725,
+                "map@1": 0.662,
+                "map@5": 0.797,
+                "map@10": 0.7997424603174603,
+                "mrr@1": 0.662,
+                "mrr@5": 0.797,
+                "mrr@10": 0.7997424603174603,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "WebQA": {
+                "hit@1": 0.919,
+                "hit@5": 0.994,
+                "hit@10": 0.998,
+                "ndcg_linear@1": 0.919,
+                "ndcg_linear@5": 0.9619522053693657,
+                "ndcg_linear@10": 0.963298983072579,
+                "ndcg_exponential@1": 0.919,
+                "ndcg_exponential@5": 0.9619522053693657,
+                "ndcg_exponential@10": 0.963298983072579,
+                "precision@1": 0.919,
+                "precision@5": 0.1988,
+                "precision@10": 0.09980000000000001,
+                "recall@1": 0.919,
+                "recall@5": 0.994,
+                "recall@10": 0.998,
+                "f1@1": 0.919,
+                "f1@5": 0.3313333333333333,
+                "f1@10": 0.18145454545454545,
+                "map@1": 0.919,
+                "map@5": 0.9510166666666666,
+                "map@10": 0.9516039682539683,
+                "mrr@1": 0.919,
+                "mrr@5": 0.9510166666666666,
+                "mrr@10": 0.9516039682539683,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "FashionIQ": {
+                "hit@1": 0.243,
+                "hit@5": 0.511,
+                "hit@10": 0.592,
+                "ndcg_linear@1": 0.243,
+                "ndcg_linear@5": 0.38383730216588574,
+                "ndcg_linear@10": 0.41007610225616226,
+                "ndcg_exponential@1": 0.243,
+                "ndcg_exponential@5": 0.38383730216588574,
+                "ndcg_exponential@10": 0.41007610225616226,
+                "precision@1": 0.243,
+                "precision@5": 0.10219999999999999,
+                "precision@10": 0.05919999999999999,
+                "recall@1": 0.243,
+                "recall@5": 0.511,
+                "recall@10": 0.592,
+                "f1@1": 0.243,
+                "f1@5": 0.17033333333333336,
+                "f1@10": 0.10763636363636366,
+                "map@1": 0.243,
+                "map@5": 0.3416666666666666,
+                "map@10": 0.3525230158730159,
+                "mrr@1": 0.243,
+                "mrr@5": 0.3416666666666666,
+                "mrr@10": 0.3525230158730159,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "Wiki-SS-NQ": {
+                "hit@1": 0.743,
+                "hit@5": 0.909,
+                "hit@10": 0.942,
+                "ndcg_linear@1": 0.743,
+                "ndcg_linear@5": 0.836321433713043,
+                "ndcg_linear@10": 0.8472817749175067,
+                "ndcg_exponential@1": 0.743,
+                "ndcg_exponential@5": 0.836321433713043,
+                "ndcg_exponential@10": 0.8472817749175067,
+                "precision@1": 0.743,
+                "precision@5": 0.18180000000000002,
+                "precision@10": 0.09420000000000002,
+                "recall@1": 0.743,
+                "recall@5": 0.909,
+                "recall@10": 0.942,
+                "f1@1": 0.743,
+                "f1@5": 0.303,
+                "f1@10": 0.17127272727272724,
+                "map@1": 0.743,
+                "map@5": 0.81165,
+                "map@10": 0.8163456349206349,
+                "mrr@1": 0.743,
+                "mrr@5": 0.81165,
+                "mrr@10": 0.8163456349206349,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "OVEN": {
+                "hit@1": 0.732,
+                "hit@5": 0.88,
+                "hit@10": 0.925,
+                "ndcg_linear@1": 0.732,
+                "ndcg_linear@5": 0.813023413986076,
+                "ndcg_linear@10": 0.8277770061315273,
+                "ndcg_exponential@1": 0.732,
+                "ndcg_exponential@5": 0.813023413986076,
+                "ndcg_exponential@10": 0.8277770061315273,
+                "precision@1": 0.732,
+                "precision@5": 0.176,
+                "precision@10": 0.0925,
+                "recall@1": 0.732,
+                "recall@5": 0.88,
+                "recall@10": 0.925,
+                "f1@1": 0.732,
+                "f1@5": 0.2933333333333334,
+                "f1@10": 0.16818181818181818,
+                "map@1": 0.732,
+                "map@5": 0.7905333333333333,
+                "map@10": 0.796740476190476,
+                "mrr@1": 0.732,
+                "mrr@5": 0.7905333333333333,
+                "mrr@10": 0.796740476190476,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "EDIS": {
+                "hit@1": 0.944,
+                "hit@5": 0.997,
+                "hit@10": 1.0,
+                "ndcg_linear@1": 0.944,
+                "ndcg_linear@5": 0.9750516728538741,
+                "ndcg_linear@10": 0.9760124097440858,
+                "ndcg_exponential@1": 0.944,
+                "ndcg_exponential@5": 0.9750516728538741,
+                "ndcg_exponential@10": 0.9760124097440858,
+                "precision@1": 0.944,
+                "precision@5": 0.19940000000000002,
+                "precision@10": 0.10000000000000002,
+                "recall@1": 0.944,
+                "recall@5": 0.997,
+                "recall@10": 1.0,
+                "f1@1": 0.944,
+                "f1@5": 0.3323333333333333,
+                "f1@10": 0.18181818181818182,
+                "map@1": 0.944,
+                "map@5": 0.9675,
+                "map@10": 0.9678916666666667,
+                "mrr@1": 0.944,
+                "mrr@5": 0.9675,
+                "mrr@10": 0.9678916666666667,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "MSCOCO": {
+                "hit@1": 0.739,
+                "hit@5": 0.879,
+                "hit@10": 0.912,
+                "ndcg_linear@1": 0.739,
+                "ndcg_linear@5": 0.8170358283919806,
+                "ndcg_linear@10": 0.8278094286114062,
+                "ndcg_exponential@1": 0.739,
+                "ndcg_exponential@5": 0.8170358283919806,
+                "ndcg_exponential@10": 0.8278094286114062,
+                "precision@1": 0.739,
+                "precision@5": 0.1758,
+                "precision@10": 0.09120000000000002,
+                "recall@1": 0.739,
+                "recall@5": 0.879,
+                "recall@10": 0.912,
+                "f1@1": 0.739,
+                "f1@5": 0.293,
+                "f1@10": 0.1658181818181818,
+                "map@1": 0.739,
+                "map@5": 0.7961166666666667,
+                "map@10": 0.8006210317460317,
+                "mrr@1": 0.739,
+                "mrr@5": 0.7961166666666667,
+                "mrr@10": 0.8006210317460317,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "RefCOCO": {
+                "hit@1": 0.904,
+                "hit@5": 0.995,
+                "hit@10": 0.997,
+                "ndcg_linear@1": 0.904,
+                "ndcg_linear@5": 0.9580666714637504,
+                "ndcg_linear@10": 0.9587154696738694,
+                "ndcg_exponential@1": 0.904,
+                "ndcg_exponential@5": 0.9580666714637504,
+                "ndcg_exponential@10": 0.9587154696738694,
+                "precision@1": 0.904,
+                "precision@5": 0.19900000000000004,
+                "precision@10": 0.09970000000000001,
+                "recall@1": 0.904,
+                "recall@5": 0.995,
+                "recall@10": 0.997,
+                "f1@1": 0.904,
+                "f1@5": 0.3316666666666666,
+                "f1@10": 0.18127272727272725,
+                "map@1": 0.904,
+                "map@5": 0.9452833333333333,
+                "map@10": 0.9455511904761904,
+                "mrr@1": 0.904,
+                "mrr@5": 0.9452833333333333,
+                "mrr@10": 0.9455511904761904,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "RefCOCO-Matching": {
+                "hit@1": 0.927,
+                "hit@5": 1.0,
+                "hit@10": 1.0,
+                "ndcg_linear@1": 0.927,
+                "ndcg_linear@5": 0.9748197315178593,
+                "ndcg_linear@10": 0.9748197315178593,
+                "ndcg_exponential@1": 0.927,
+                "ndcg_exponential@5": 0.9748197315178593,
+                "ndcg_exponential@10": 0.9748197315178593,
+                "precision@1": 0.927,
+                "precision@5": 0.20000000000000004,
+                "precision@10": 0.10000000000000002,
+                "recall@1": 0.927,
+                "recall@5": 1.0,
+                "recall@10": 1.0,
+                "f1@1": 0.927,
+                "f1@5": 0.3333333333333333,
+                "f1@10": 0.18181818181818182,
+                "map@1": 0.927,
+                "map@5": 0.9663333333333334,
+                "map@10": 0.9663333333333334,
+                "mrr@1": 0.927,
+                "mrr@5": 0.9633333333333334,
+                "mrr@10": 0.9633333333333334,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "Visual7W-Pointing": {
+                "hit@1": 0.916,
+                "hit@5": 0.971,
+                "hit@10": 0.985,
+                "ndcg_linear@1": 0.916,
+                "ndcg_linear@5": 0.9457068095214094,
+                "ndcg_linear@10": 0.9502811890192044,
+                "ndcg_exponential@1": 0.916,
+                "ndcg_exponential@5": 0.9457068095214094,
+                "ndcg_exponential@10": 0.9502811890192044,
+                "precision@1": 0.916,
+                "precision@5": 0.19420000000000004,
+                "precision@10": 0.0985,
+                "recall@1": 0.916,
+                "recall@5": 0.971,
+                "recall@10": 0.985,
+                "f1@1": 0.916,
+                "f1@5": 0.32366666666666666,
+                "f1@10": 0.17909090909090908,
+                "map@1": 0.916,
+                "map@5": 0.93725,
+                "map@10": 0.939163492063492,
+                "mrr@1": 0.916,
+                "mrr@5": 0.93725,
+                "mrr@10": 0.939163492063492,
+                "num_pred": 1000,
+                "num_data": 1000
+            }
+        },
+        "visdoc": {
+            "ViDoRe_arxivqa": {
+                "hit@1": 0.724,
+                "hit@5": 0.832,
+                "hit@10": 0.87,
+                "ndcg_linear@1": 0.724,
+                "ndcg_linear@5": 0.7816980877667238,
+                "ndcg_linear@10": 0.794154083491937,
+                "ndcg_exponential@1": 0.724,
+                "ndcg_exponential@5": 0.7816980877667238,
+                "ndcg_exponential@10": 0.794154083491937,
+                "precision@1": 0.724,
+                "precision@5": 0.16639999999999996,
+                "precision@10": 0.087,
+                "recall@1": 0.724,
+                "recall@5": 0.832,
+                "recall@10": 0.87,
+                "f1@1": 0.724,
+                "f1@5": 0.2773333333333334,
+                "f1@10": 0.15818181818181817,
+                "map@1": 0.724,
+                "map@5": 0.765,
+                "map@10": 0.7702428571428571,
+                "mrr@1": 0.724,
+                "mrr@5": 0.765,
+                "mrr@10": 0.7702428571428571,
+                "num_pred": 500,
+                "num_data": 500
+            },
+            "ViDoRe_docvqa": {
+                "hit@1": 0.4035476718403548,
+                "hit@5": 0.5809312638580931,
+                "hit@10": 0.656319290465632,
+                "ndcg_linear@1": 0.4035476718403548,
+                "ndcg_linear@5": 0.49108570677106456,
+                "ndcg_linear@10": 0.5147397758959128,
+                "ndcg_exponential@1": 0.4035476718403548,
+                "ndcg_exponential@5": 0.49108570677106456,
+                "ndcg_exponential@10": 0.5147397758959128,
+                "precision@1": 0.4035476718403548,
+                "precision@5": 0.11707317073170731,
+                "precision@10": 0.06651884700665188,
+                "recall@1": 0.3976164079822616,
+                "recall@5": 0.5727827050997782,
+                "recall@10": 0.644660014781966,
+                "f1@1": 0.39886671593988665,
+                "f1@5": 0.19345167028093863,
+                "f1@10": 0.1199606503375905,
+                "map@1": 0.4035476718403548,
+                "map@5": 0.4620349839862035,
+                "map@10": 0.4718157181571816,
+                "mrr@1": 0.4035476718403548,
+                "mrr@5": 0.46855136733185515,
+                "mrr@10": 0.47878523915109283,
+                "num_pred": 451,
+                "num_data": 451
+            },
+            "ViDoRe_infovqa": {
+                "hit@1": 0.7995951417004049,
+                "hit@5": 0.9230769230769231,
+                "hit@10": 0.9473684210526315,
+                "ndcg_linear@1": 0.7995951417004049,
+                "ndcg_linear@5": 0.8658530746439195,
+                "ndcg_linear@10": 0.8742677543682404,
+                "ndcg_exponential@1": 0.7995951417004049,
+                "ndcg_exponential@5": 0.8658530746439195,
+                "ndcg_exponential@10": 0.8742677543682404,
+                "precision@1": 0.7995951417004049,
+                "precision@5": 0.18461538461538463,
+                "precision@10": 0.09493927125506073,
+                "recall@1": 0.7995951417004049,
+                "recall@5": 0.9220647773279352,
+                "recall@10": 0.9473684210526315,
+                "f1@1": 0.7995951417004049,
+                "f1@5": 0.3075959128590708,
+                "f1@10": 0.17255551466077784,
+                "map@1": 0.7995951417004049,
+                "map@5": 0.8468960863697707,
+                "map@10": 0.8505888117730224,
+                "mrr@1": 0.7995951417004049,
+                "mrr@5": 0.8472334682860998,
+                "mrr@10": 0.8505888117730224,
+                "num_pred": 494,
+                "num_data": 494
+            },
+            "ViDoRe_tabfquad": {
+                "hit@1": 0.8714285714285714,
+                "hit@5": 0.95,
+                "hit@10": 0.975,
+                "ndcg_linear@1": 0.8714285714285714,
+                "ndcg_linear@5": 0.9139664212357709,
+                "ndcg_linear@10": 0.9218222423868873,
+                "ndcg_exponential@1": 0.8714285714285714,
+                "ndcg_exponential@5": 0.9139664212357709,
+                "ndcg_exponential@10": 0.9218222423868873,
+                "precision@1": 0.8714285714285714,
+                "precision@5": 0.18999999999999997,
+                "precision@10": 0.09749999999999999,
+                "recall@1": 0.8714285714285714,
+                "recall@5": 0.95,
+                "recall@10": 0.975,
+                "f1@1": 0.8714285714285714,
+                "f1@5": 0.3166666666666667,
+                "f1@10": 0.1772727272727273,
+                "map@1": 0.8714285714285714,
+                "map@5": 0.9019642857142858,
+                "map@10": 0.9050736961451247,
+                "mrr@1": 0.8714285714285714,
+                "mrr@5": 0.9019642857142858,
+                "mrr@10": 0.9050736961451247,
+                "num_pred": 280,
+                "num_data": 280
+            },
+            "ViDoRe_tatdqa": {
+                "hit@1": 0.41433778857837184,
+                "hit@5": 0.6834750911300121,
+                "hit@10": 0.7885783718104495,
+                "ndcg_linear@1": 0.41433778857837184,
+                "ndcg_linear@5": 0.5558704694199323,
+                "ndcg_linear@10": 0.5903675001014173,
+                "ndcg_exponential@1": 0.41433778857837184,
+                "ndcg_exponential@5": 0.5558704694199323,
+                "ndcg_exponential@10": 0.5903675001014173,
+                "precision@1": 0.41433778857837184,
+                "precision@5": 0.13681652490886997,
+                "precision@10": 0.0789793438639125,
+                "recall@1": 0.413730255164034,
+                "recall@5": 0.6825637910085055,
+                "recall@10": 0.7879708383961118,
+                "f1@1": 0.4139327663021466,
+                "f1@5": 0.2278828907018458,
+                "f1@10": 0.14354357671490117,
+                "map@1": 0.41433778857837184,
+                "map@5": 0.5136644390441474,
+                "map@10": 0.5281519412139096,
+                "mrr@1": 0.41433778857837184,
+                "mrr@5": 0.5141454029971648,
+                "mrr@10": 0.5285721518254932,
+                "num_pred": 1646,
+                "num_data": 1646
+            },
+            "ViDoRe_shiftproject": {
+                "hit@1": 0.61,
+                "hit@5": 0.89,
+                "hit@10": 0.97,
+                "ndcg_linear@1": 0.61,
+                "ndcg_linear@5": 0.7617245745315876,
+                "ndcg_linear@10": 0.7890774568114949,
+                "ndcg_exponential@1": 0.61,
+                "ndcg_exponential@5": 0.7617245745315876,
+                "ndcg_exponential@10": 0.7890774568114949,
+                "precision@1": 0.61,
+                "precision@5": 0.17799999999999996,
+                "precision@10": 0.09699999999999998,
+                "recall@1": 0.61,
+                "recall@5": 0.89,
+                "recall@10": 0.97,
+                "f1@1": 0.61,
+                "f1@5": 0.2966666666666667,
+                "f1@10": 0.17636363636363647,
+                "map@1": 0.61,
+                "map@5": 0.7188333333333334,
+                "map@10": 0.7309761904761904,
+                "mrr@1": 0.61,
+                "mrr@5": 0.7188333333333334,
+                "mrr@10": 0.7309761904761904,
+                "num_pred": 100,
+                "num_data": 100
+            },
+            "ViDoRe_syntheticDocQA_artificial_intelligence": {
+                "hit@1": 0.8,
+                "hit@5": 0.99,
+                "hit@10": 1.0,
+                "ndcg_linear@1": 0.8,
+                "ndcg_linear@5": 0.9099357497170402,
+                "ndcg_linear@10": 0.912826397980219,
+                "ndcg_exponential@1": 0.8,
+                "ndcg_exponential@5": 0.9099357497170402,
+                "ndcg_exponential@10": 0.912826397980219,
+                "precision@1": 0.8,
+                "precision@5": 0.19799999999999998,
+                "precision@10": 0.09999999999999998,
+                "recall@1": 0.8,
+                "recall@5": 0.99,
+                "recall@10": 1.0,
+                "f1@1": 0.8,
+                "f1@5": 0.3300000000000001,
+                "f1@10": 0.1818181818181819,
+                "map@1": 0.8,
+                "map@5": 0.8826666666666667,
+                "map@10": 0.8836666666666666,
+                "mrr@1": 0.8,
+                "mrr@5": 0.8826666666666667,
+                "mrr@10": 0.8836666666666666,
+                "num_pred": 100,
+                "num_data": 100
+            },
+            "ViDoRe_syntheticDocQA_energy": {
+                "hit@1": 0.81,
+                "hit@5": 0.93,
+                "hit@10": 0.96,
+                "ndcg_linear@1": 0.81,
+                "ndcg_linear@5": 0.8753378444481007,
+                "ndcg_linear@10": 0.8856166369581185,
+                "ndcg_exponential@1": 0.81,
+                "ndcg_exponential@5": 0.8753378444481007,
+                "ndcg_exponential@10": 0.8856166369581185,
+                "precision@1": 0.81,
+                "precision@5": 0.18599999999999994,
+                "precision@10": 0.09599999999999997,
+                "recall@1": 0.81,
+                "recall@5": 0.93,
+                "recall@10": 0.96,
+                "f1@1": 0.81,
+                "f1@5": 0.31000000000000005,
+                "f1@10": 0.17454545454545464,
+                "map@1": 0.81,
+                "map@5": 0.8569999999999999,
+                "map@10": 0.8615833333333334,
+                "mrr@1": 0.81,
+                "mrr@5": 0.8569999999999999,
+                "mrr@10": 0.8615833333333334,
+                "num_pred": 100,
+                "num_data": 100
+            },
+            "ViDoRe_syntheticDocQA_government_reports": {
+                "hit@1": 0.83,
+                "hit@5": 0.98,
+                "hit@10": 0.99,
+                "ndcg_linear@1": 0.83,
+                "ndcg_linear@5": 0.9155750345909591,
+                "ndcg_linear@10": 0.9187296833588163,
+                "ndcg_exponential@1": 0.83,
+                "ndcg_exponential@5": 0.9155750345909591,
+                "ndcg_exponential@10": 0.9187296833588163,
+                "precision@1": 0.83,
+                "precision@5": 0.19599999999999998,
+                "precision@10": 0.09899999999999999,
+                "recall@1": 0.83,
+                "recall@5": 0.98,
+                "recall@10": 0.99,
+                "f1@1": 0.83,
+                "f1@5": 0.32666666666666677,
+                "f1@10": 0.18000000000000008,
+                "map@1": 0.83,
+                "map@5": 0.8936666666666667,
+                "map@10": 0.8949166666666667,
+                "mrr@1": 0.83,
+                "mrr@5": 0.8936666666666667,
+                "mrr@10": 0.8949166666666667,
+                "num_pred": 100,
+                "num_data": 100
+            },
+            "ViDoRe_syntheticDocQA_healthcare_industry": {
+                "hit@1": 0.85,
+                "hit@5": 0.99,
+                "hit@10": 1.0,
+                "ndcg_linear@1": 0.85,
+                "ndcg_linear@5": 0.9338868640816542,
+                "ndcg_linear@10": 0.9372201974149877,
+                "ndcg_exponential@1": 0.85,
+                "ndcg_exponential@5": 0.9338868640816542,
+                "ndcg_exponential@10": 0.9372201974149877,
+                "precision@1": 0.85,
+                "precision@5": 0.19799999999999998,
+                "precision@10": 0.09999999999999998,
+                "recall@1": 0.85,
+                "recall@5": 0.99,
+                "recall@10": 1.0,
+                "f1@1": 0.85,
+                "f1@5": 0.3300000000000001,
+                "f1@10": 0.1818181818181819,
+                "map@1": 0.85,
+                "map@5": 0.9145,
+                "map@10": 0.9159285714285714,
+                "mrr@1": 0.85,
+                "mrr@5": 0.9145,
+                "mrr@10": 0.9159285714285714,
+                "num_pred": 100,
+                "num_data": 100
+            },
+            "ViDoRe_esg_reports_human_labeled_v2": {
+                "hit@1": 0.6346153846153846,
+                "hit@5": 0.9038461538461539,
+                "hit@10": 0.9423076923076923,
+                "ndcg_linear@1": 0.6121794871794872,
+                "ndcg_linear@5": 0.662739973618026,
+                "ndcg_linear@10": 0.6801813086670259,
+                "ndcg_exponential@1": 0.6053113553113553,
+                "ndcg_exponential@5": 0.6592882828242518,
+                "ndcg_exponential@10": 0.6769628635131526,
+                "precision@1": 0.6346153846153846,
+                "precision@5": 0.25769230769230766,
+                "precision@10": 0.15192307692307688,
+                "recall@1": 0.4403846153846154,
+                "recall@5": 0.7235431235431237,
+                "recall@10": 0.7684232434232434,
+                "f1@1": 0.4834530853761623,
+                "f1@5": 0.34303453899042136,
+                "f1@10": 0.2309966899616221,
+                "map@1": 0.6346153846153846,
+                "map@5": 0.6022649572649572,
+                "map@10": 0.6106303418803418,
+                "mrr@1": 0.6346153846153846,
+                "mrr@5": 0.7346153846153847,
+                "mrr@10": 0.7389423076923076,
+                "num_pred": 52,
+                "num_data": 52
+            },
+            "ViDoRe_biomedical_lectures_v2": {
+                "hit@1": 0.5375,
+                "hit@5": 0.83125,
+                "hit@10": 0.90625,
+                "ndcg_linear@1": 0.5375,
+                "ndcg_linear@5": 0.5841363021080512,
+                "ndcg_linear@10": 0.6249174529201162,
+                "ndcg_exponential@1": 0.5375,
+                "ndcg_exponential@5": 0.5841363021080511,
+                "ndcg_exponential@10": 0.6249174529201162,
+                "precision@1": 0.5375,
+                "precision@5": 0.26875,
+                "precision@10": 0.17625000000000002,
+                "recall@1": 0.3166321699134199,
+                "recall@5": 0.6228635886172651,
+                "recall@10": 0.7383215161064426,
+                "f1@1": 0.36593299062049056,
+                "f1@5": 0.33357090132090134,
+                "f1@10": 0.25719967294415824,
+                "map@1": 0.5375,
+                "map@5": 0.5106197916666666,
+                "map@10": 0.5325731233465609,
+                "mrr@1": 0.5375,
+                "mrr@5": 0.6533333333333333,
+                "mrr@10": 0.6637103174603174,
+                "num_pred": 160,
+                "num_data": 160
+            },
+            "ViDoRe_biomedical_lectures_v2_multilingual": {
+                "hit@1": 0.490625,
+                "hit@5": 0.8,
+                "hit@10": 0.890625,
+                "ndcg_linear@1": 0.490625,
+                "ndcg_linear@5": 0.5434414494513081,
+                "ndcg_linear@10": 0.582249778808178,
+                "ndcg_exponential@1": 0.490625,
+                "ndcg_exponential@5": 0.5434414494513081,
+                "ndcg_exponential@10": 0.5822497788081779,
+                "precision@1": 0.490625,
+                "precision@5": 0.254375,
+                "precision@10": 0.16359375000000004,
+                "recall@1": 0.2916960107906799,
+                "recall@5": 0.58253104508849,
+                "recall@10": 0.6977005104059648,
+                "f1@1": 0.33520032918470416,
+                "f1@5": 0.3138615559787435,
+                "f1@10": 0.23995111426223562,
+                "map@1": 0.490625,
+                "map@5": 0.47009505208333324,
+                "map@10": 0.48930132895171957,
+                "mrr@1": 0.490625,
+                "mrr@5": 0.613671875,
+                "mrr@10": 0.6260515873015873,
+                "num_pred": 640,
+                "num_data": 640
+            },
+            "ViDoRe_economics_reports_v2": {
+                "hit@1": 0.7586206896551724,
+                "hit@5": 0.9655172413793104,
+                "hit@10": 0.9655172413793104,
+                "ndcg_linear@1": 0.7586206896551724,
+                "ndcg_linear@5": 0.6741969787919236,
+                "ndcg_linear@10": 0.6190122003564679,
+                "ndcg_exponential@1": 0.7586206896551724,
+                "ndcg_exponential@5": 0.6741969787919236,
+                "ndcg_exponential@10": 0.6190122003564678,
+                "precision@1": 0.7586206896551724,
+                "precision@5": 0.5758620689655172,
+                "precision@10": 0.42241379310344823,
+                "recall@1": 0.13585704413718444,
+                "recall@5": 0.35288134780268793,
+                "recall@10": 0.45080357139107263,
+                "f1@1": 0.1922091657419343,
+                "f1@5": 0.356547764866831,
+                "f1@10": 0.3541634848772098,
+                "map@1": 0.7586206896551724,
+                "map@5": 0.5760680076628353,
+                "map@10": 0.48829512775736095,
+                "mrr@1": 0.7586206896551724,
+                "mrr@5": 0.8482758620689655,
+                "mrr@10": 0.8482758620689655,
+                "num_pred": 58,
+                "num_data": 58
+            },
+            "ViDoRe_economics_reports_v2_multilingual": {
+                "hit@1": 0.7025862068965517,
+                "hit@5": 0.9267241379310345,
+                "hit@10": 0.9655172413793104,
+                "ndcg_linear@1": 0.7025862068965517,
+                "ndcg_linear@5": 0.6092499057967938,
+                "ndcg_linear@10": 0.5716310478677091,
+                "ndcg_exponential@1": 0.7025862068965517,
+                "ndcg_exponential@5": 0.6092499057967938,
+                "ndcg_exponential@10": 0.571631047867709,
+                "precision@1": 0.7025862068965517,
+                "precision@5": 0.5137931034482759,
+                "precision@10": 0.39525862068965517,
+                "recall@1": 0.11976302485978202,
+                "recall@5": 0.3134997052445495,
+                "recall@10": 0.4232996166582797,
+                "f1@1": 0.1708464717573616,
+                "f1@5": 0.3146703201409769,
+                "f1@10": 0.330190439964704,
+                "map@1": 0.7025862068965517,
+                "map@5": 0.5082686781609195,
+                "map@10": 0.43586978908741014,
+                "mrr@1": 0.7025862068965517,
+                "mrr@5": 0.7956896551724137,
+                "mrr@10": 0.8007714148877941,
+                "num_pred": 232,
+                "num_data": 232
+            },
+            "ViDoRe_esg_reports_v2": {
+                "hit@1": 0.6491228070175439,
+                "hit@5": 0.8596491228070176,
+                "hit@10": 0.8947368421052632,
+                "ndcg_linear@1": 0.6491228070175439,
+                "ndcg_linear@5": 0.5997641652064848,
+                "ndcg_linear@10": 0.634484891188961,
+                "ndcg_exponential@1": 0.6491228070175439,
+                "ndcg_exponential@5": 0.5997641652064848,
+                "ndcg_exponential@10": 0.634484891188961,
+                "precision@1": 0.6491228070175439,
+                "precision@5": 0.2771929824561403,
+                "precision@10": 0.18771929824561406,
+                "recall@1": 0.31802422723475354,
+                "recall@5": 0.5807435254803677,
+                "recall@10": 0.7037802840434418,
+                "f1@1": 0.3830497579317905,
+                "f1@5": 0.3241369172563792,
+                "f1@10": 0.2614094743270236,
+                "map@1": 0.6491228070175439,
+                "map@5": 0.5182261208576998,
+                "map@10": 0.5330643158822983,
+                "mrr@1": 0.6491228070175439,
+                "mrr@5": 0.7368421052631579,
+                "mrr@10": 0.7407407407407407,
+                "num_pred": 57,
+                "num_data": 57
+            },
+            "ViDoRe_esg_reports_v2_multilingual": {
+                "hit@1": 0.5614035087719298,
+                "hit@5": 0.8552631578947368,
+                "hit@10": 0.8903508771929824,
+                "ndcg_linear@1": 0.5614035087719298,
+                "ndcg_linear@5": 0.5682060214897536,
+                "ndcg_linear@10": 0.5985226385169606,
+                "ndcg_exponential@1": 0.5614035087719298,
+                "ndcg_exponential@5": 0.5682060214897536,
+                "ndcg_exponential@10": 0.5985226385169606,
+                "precision@1": 0.5614035087719298,
+                "precision@5": 0.2754385964912281,
+                "precision@10": 0.18026315789473688,
+                "recall@1": 0.27056182121971595,
+                "recall@5": 0.5850093984962406,
+                "recall@10": 0.6978070175438598,
+                "f1@1": 0.32801009939848413,
+                "f1@5": 0.32522762720398035,
+                "f1@10": 0.25474615009098467,
+                "map@1": 0.5614035087719298,
+                "map@5": 0.47951510721247564,
+                "map@10": 0.49127398937157707,
+                "mrr@1": 0.5614035087719298,
+                "mrr@5": 0.6827485380116959,
+                "mrr@10": 0.6871031746031746,
+                "num_pred": 228,
+                "num_data": 228
+            },
+            "VisRAG_ArxivQA": {
+                "hit@1": 0.7120098039215687,
+                "hit@5": 0.8382352941176471,
+                "hit@10": 0.8700980392156863,
+                "ndcg_linear@1": 0.7120098039215687,
+                "ndcg_linear@5": 0.7817991444138218,
+                "ndcg_linear@10": 0.7919346368535662,
+                "ndcg_exponential@1": 0.7120098039215687,
+                "ndcg_exponential@5": 0.7817991444138218,
+                "ndcg_exponential@10": 0.7919346368535662,
+                "precision@1": 0.7120098039215687,
+                "precision@5": 0.1676470588235294,
+                "precision@10": 0.08700980392156861,
+                "recall@1": 0.7120098039215687,
+                "recall@5": 0.8382352941176471,
+                "recall@10": 0.8700980392156863,
+                "f1@1": 0.7120098039215687,
+                "f1@5": 0.2794117647058824,
+                "f1@10": 0.15819964349376117,
+                "map@1": 0.7120098039215687,
+                "map@5": 0.7628063725490196,
+                "map@10": 0.766889880952381,
+                "mrr@1": 0.7120098039215687,
+                "mrr@5": 0.7628063725490196,
+                "mrr@10": 0.766889880952381,
+                "num_pred": 816,
+                "num_data": 816
+            },
+            "VisRAG_ChartQA": {
+                "hit@1": 0.6984126984126984,
+                "hit@5": 0.873015873015873,
+                "hit@10": 0.9206349206349206,
+                "ndcg_linear@1": 0.6984126984126984,
+                "ndcg_linear@5": 0.7952874267735981,
+                "ndcg_linear@10": 0.8103095637633947,
+                "ndcg_exponential@1": 0.6984126984126984,
+                "ndcg_exponential@5": 0.7952874267735981,
+                "ndcg_exponential@10": 0.8103095637633947,
+                "precision@1": 0.6984126984126984,
+                "precision@5": 0.17460317460317454,
+                "precision@10": 0.09206349206349201,
+                "recall@1": 0.6984126984126984,
+                "recall@5": 0.873015873015873,
+                "recall@10": 0.9206349206349206,
+                "f1@1": 0.6984126984126984,
+                "f1@5": 0.291005291005291,
+                "f1@10": 0.16738816738816742,
+                "map@1": 0.6984126984126984,
+                "map@5": 0.7690476190476191,
+                "map@10": 0.775,
+                "mrr@1": 0.6984126984126984,
+                "mrr@5": 0.7690476190476191,
+                "mrr@10": 0.775,
+                "num_pred": 63,
+                "num_data": 63
+            },
+            "VisRAG_MP-DocVQA": {
+                "hit@1": 0.6903553299492385,
+                "hit@5": 0.8595600676818951,
+                "hit@10": 0.9204737732656514,
+                "ndcg_linear@1": 0.6903553299492385,
+                "ndcg_linear@5": 0.7820349036494127,
+                "ndcg_linear@10": 0.8015893567306576,
+                "ndcg_exponential@1": 0.6903553299492385,
+                "ndcg_exponential@5": 0.7820349036494127,
+                "ndcg_exponential@10": 0.8015893567306576,
+                "precision@1": 0.6903553299492385,
+                "precision@5": 0.171912013536379,
+                "precision@10": 0.09204737732656514,
+                "recall@1": 0.6903553299492385,
+                "recall@5": 0.8595600676818951,
+                "recall@10": 0.9204737732656514,
+                "f1@1": 0.6903553299492385,
+                "f1@5": 0.28652002256063175,
+                "f1@10": 0.16735886786648213,
+                "map@1": 0.6903553299492385,
+                "map@5": 0.7560631697687535,
+                "map@10": 0.7640507077055301,
+                "mrr@1": 0.6903553299492385,
+                "mrr@5": 0.7560631697687535,
+                "mrr@10": 0.7640507077055301,
+                "num_pred": 591,
+                "num_data": 591
+            },
+            "VisRAG_SlideVQA": {
+                "hit@1": 0.8794964028776978,
+                "hit@5": 0.9658273381294964,
+                "hit@10": 0.9748201438848921,
+                "ndcg_linear@1": 0.8794964028776978,
+                "ndcg_linear@5": 0.9111322782066683,
+                "ndcg_linear@10": 0.9166794486497265,
+                "ndcg_exponential@1": 0.8794964028776978,
+                "ndcg_exponential@5": 0.9111322782066683,
+                "ndcg_exponential@10": 0.9166794486497265,
+                "precision@1": 0.8794964028776978,
+                "precision@5": 0.23776978417266184,
+                "precision@10": 0.12122302158273382,
+                "recall@1": 0.7562949640287769,
+                "recall@5": 0.947841726618705,
+                "recall@10": 0.9631294964028777,
+                "f1@1": 0.7973621103117506,
+                "f1@5": 0.37332990750256945,
+                "f1@10": 0.2128569871375627,
+                "map@1": 0.8794964028776978,
+                "map@5": 0.8898381294964028,
+                "map@10": 0.8929377926230444,
+                "mrr@1": 0.8794964028776978,
+                "mrr@5": 0.9164568345323741,
+                "mrr@10": 0.9175181283544593,
+                "num_pred": 556,
+                "num_data": 556
+            },
+            "VisRAG_InfoVQA": {
+                "hit@1": 0.786908077994429,
+                "hit@5": 0.9401114206128134,
+                "hit@10": 0.9665738161559888,
+                "ndcg_linear@1": 0.786908077994429,
+                "ndcg_linear@5": 0.871620682507705,
+                "ndcg_linear@10": 0.8802332479350842,
+                "ndcg_exponential@1": 0.786908077994429,
+                "ndcg_exponential@5": 0.871620682507705,
+                "ndcg_exponential@10": 0.8802332479350842,
+                "precision@1": 0.786908077994429,
+                "precision@5": 0.18802228412256264,
+                "precision@10": 0.09665738161559888,
+                "recall@1": 0.786908077994429,
+                "recall@5": 0.9401114206128134,
+                "recall@10": 0.9665738161559888,
+                "f1@1": 0.786908077994429,
+                "f1@5": 0.31337047353760455,
+                "f1@10": 0.17574069384654348,
+                "map@1": 0.786908077994429,
+                "map@5": 0.8485143918291551,
+                "map@10": 0.8521018481673078,
+                "mrr@1": 0.786908077994429,
+                "mrr@5": 0.8485143918291551,
+                "mrr@10": 0.8521018481673078,
+                "num_pred": 718,
+                "num_data": 718
+            },
+            "VisRAG_PlotQA": {
+                "hit@1": 0.47740440324449596,
+                "hit@5": 0.7450753186558516,
+                "hit@10": 0.8342989571263036,
+                "ndcg_linear@1": 0.47740440324449596,
+                "ndcg_linear@5": 0.6171386649075443,
+                "ndcg_linear@10": 0.6460193087434463,
+                "ndcg_exponential@1": 0.47740440324449596,
+                "ndcg_exponential@5": 0.6171386649075443,
+                "ndcg_exponential@10": 0.6460193087434463,
+                "precision@1": 0.47740440324449596,
+                "precision@5": 0.14901506373117032,
+                "precision@10": 0.08342989571263036,
+                "recall@1": 0.47740440324449596,
+                "recall@5": 0.7450753186558516,
+                "recall@10": 0.8342989571263036,
+                "f1@1": 0.47740440324449596,
+                "f1@5": 0.24835843955195064,
+                "f1@10": 0.15169071947750978,
+                "map@1": 0.47740440324449596,
+                "map@5": 0.5747779065276168,
+                "map@10": 0.5867033603707996,
+                "mrr@1": 0.47740440324449596,
+                "mrr@5": 0.5747779065276168,
+                "mrr@10": 0.5867033603707996,
+                "num_pred": 863,
+                "num_data": 863
+            },
+            "ViDoSeek-page": {
+                "hit@1": 0.028021015761821366,
+                "hit@5": 0.4115586690017513,
+                "hit@10": 0.6514886164623468,
+                "ndcg_linear@1": 0.028021015761821366,
+                "ndcg_linear@5": 0.22470566490499208,
+                "ndcg_linear@10": 0.3023957738482507,
+                "ndcg_exponential@1": 0.028021015761821366,
+                "ndcg_exponential@5": 0.2247056649049921,
+                "ndcg_exponential@10": 0.3023957738482508,
+                "precision@1": 0.028021015761821366,
+                "precision@5": 0.08231173380035026,
+                "precision@10": 0.06514886164623468,
+                "recall@1": 0.028021015761821366,
+                "recall@5": 0.4115586690017513,
+                "recall@10": 0.6514886164623468,
+                "f1@1": 0.028021015761821366,
+                "f1@5": 0.13718622300058378,
+                "f1@10": 0.11845247572042672,
+                "map@1": 0.028021015761821366,
+                "map@5": 0.16317863397548157,
+                "map@10": 0.19531176159897704,
+                "mrr@1": 0.028021015761821366,
+                "mrr@5": 0.16317863397548157,
+                "mrr@10": 0.19531176159897704,
+                "num_pred": 1142,
+                "num_data": 1142
+            },
+            "ViDoSeek-doc": {
+                "hit@1": 0.9929947460595446,
+                "hit@5": 0.9964973730297724,
+                "hit@10": 0.9973730297723292,
+                "ndcg_linear@1": 0.6780502043199066,
+                "ndcg_linear@5": 0.8347155048681709,
+                "ndcg_linear@10": 0.8283316874780556,
+                "ndcg_exponential@1": 0.4530898173630222,
+                "ndcg_exponential@5": 0.7153063162905082,
+                "ndcg_exponential@10": 0.7524802642874342,
+                "precision@1": 0.9929947460595446,
+                "precision@5": 0.9224168126094572,
+                "precision@10": 0.8474605954465849,
+                "recall@1": 0.05858093829304035,
+                "recall@5": 0.27097789072938616,
+                "recall@10": 0.4882164962990025,
+                "f1@1": 0.10988695226799426,
+                "f1@5": 0.4104084006565447,
+                "f1@10": 0.605623879196985,
+                "map@1": 0.9929947460595446,
+                "map@5": 0.9156625802685348,
+                "map@10": 0.8379593772710485,
+                "mrr@1": 0.9929947460595446,
+                "mrr@5": 0.9943812025685932,
+                "mrr@10": 0.9944687682428489,
+                "num_pred": 1142,
+                "num_data": 1142
+            },
+            "MMLongBench-page": {
+                "hit@1": 0.07517899761336516,
+                "hit@5": 0.30668257756563244,
+                "hit@10": 0.40692124105011934,
+                "ndcg_linear@1": 0.07517899761336516,
+                "ndcg_linear@5": 0.15889672106004182,
+                "ndcg_linear@10": 0.19237673084837761,
+                "ndcg_exponential@1": 0.07517899761336516,
+                "ndcg_exponential@5": 0.15889672106004182,
+                "ndcg_exponential@10": 0.19237673084837764,
+                "precision@1": 0.07517899761336516,
+                "precision@5": 0.07136038186157519,
+                "precision@10": 0.05286396181384249,
+                "recall@1": 0.04922434367541766,
+                "recall@5": 0.22613081031935447,
+                "recall@10": 0.3179622684395954,
+                "f1@1": 0.05696844921188596,
+                "f1@5": 0.10252626544217168,
+                "f1@10": 0.08628360174493102,
+                "map@1": 0.07517899761336516,
+                "map@5": 0.11930389817024661,
+                "map@10": 0.134453706071397,
+                "mrr@1": 0.07517899761336516,
+                "mrr@5": 0.16205250596658713,
+                "mrr@10": 0.17527607303860288,
+                "num_pred": 838,
+                "num_data": 838
+            },
+            "MMLongBench-doc": {
+                "hit@1": 0.6992840095465394,
+                "hit@5": 0.788782816229117,
+                "hit@10": 0.8305489260143198,
+                "ndcg_linear@1": 0.4908512330946698,
+                "ndcg_linear@5": 0.5151318143032525,
+                "ndcg_linear@10": 0.4942756345433957,
+                "ndcg_exponential@1": 0.3419706784861916,
+                "ndcg_exponential@5": 0.43117309505779183,
+                "ndcg_exponential@10": 0.439350643034243,
+                "precision@1": 0.6992840095465394,
+                "precision@5": 0.5708830548926014,
+                "precision@10": 0.5034606205250597,
+                "recall@1": 0.022724297502752358,
+                "recall@5": 0.09002044278616353,
+                "recall@10": 0.1555140961544712,
+                "f1@1": 0.04359714278969275,
+                "f1@5": 0.14945001865951169,
+                "f1@10": 0.2231273977054642,
+                "map@1": 0.6992840095465394,
+                "map@5": 0.5389498806682578,
+                "map@10": 0.4554091058327336,
+                "mrr@1": 0.6992840095465394,
+                "mrr@5": 0.7340294351630867,
+                "mrr@10": 0.7398544342160094,
+                "num_pred": 838,
+                "num_data": 838
+            }
+        },
+        "video": {
+            "DiDeMo": {
+                "hit@1": 0.46115537848605576,
+                "hit@5": 0.7141434262948207,
+                "hit@10": 0.7928286852589641,
+                "ndcg_linear@1": 0.46115537848605576,
+                "ndcg_linear@5": 0.597854832187236,
+                "ndcg_linear@10": 0.623638688484401,
+                "ndcg_exponential@1": 0.46115537848605576,
+                "ndcg_exponential@5": 0.597854832187236,
+                "ndcg_exponential@10": 0.623638688484401,
+                "precision@1": 0.46115537848605576,
+                "precision@5": 0.14282868525896414,
+                "precision@10": 0.0792828685258964,
+                "recall@1": 0.46115537848605576,
+                "recall@5": 0.7141434262948207,
+                "recall@10": 0.7928286852589641,
+                "f1@1": 0.46115537848605576,
+                "f1@5": 0.2380478087649403,
+                "f1@10": 0.1441506700470844,
+                "map@1": 0.46115537848605576,
+                "map@5": 0.5588977423638777,
+                "map@10": 0.5697341586036806,
+                "mrr@1": 0.46115537848605576,
+                "mrr@5": 0.5588977423638777,
+                "mrr@10": 0.5697341586036806,
+                "num_pred": 1004,
+                "num_data": 1004
+            },
+            "MSR-VTT": {
+                "hit@1": 0.494,
+                "hit@5": 0.718,
+                "hit@10": 0.809,
+                "ndcg_linear@1": 0.494,
+                "ndcg_linear@5": 0.6126583935599356,
+                "ndcg_linear@10": 0.6418663414853761,
+                "ndcg_exponential@1": 0.494,
+                "ndcg_exponential@5": 0.6126583935599356,
+                "ndcg_exponential@10": 0.6418663414853761,
+                "precision@1": 0.494,
+                "precision@5": 0.1436,
+                "precision@10": 0.0809,
+                "recall@1": 0.494,
+                "recall@5": 0.718,
+                "recall@10": 0.809,
+                "f1@1": 0.494,
+                "f1@5": 0.2393333333333334,
+                "f1@10": 0.14709090909090913,
+                "map@1": 0.494,
+                "map@5": 0.5777,
+                "map@10": 0.5896333333333333,
+                "mrr@1": 0.494,
+                "mrr@5": 0.5777,
+                "mrr@10": 0.5896333333333333,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "MSVD": {
+                "hit@1": 0.6716417910447762,
+                "hit@5": 0.9044776119402985,
+                "hit@10": 0.9507462686567164,
+                "ndcg_linear@1": 0.6716417910447762,
+                "ndcg_linear@5": 0.8018114248883265,
+                "ndcg_linear@10": 0.8170350100109979,
+                "ndcg_exponential@1": 0.6716417910447762,
+                "ndcg_exponential@5": 0.8018114248883265,
+                "ndcg_exponential@10": 0.8170350100109979,
+                "precision@1": 0.6716417910447762,
+                "precision@5": 0.18089552238805967,
+                "precision@10": 0.09507462686567163,
+                "recall@1": 0.6716417910447762,
+                "recall@5": 0.9044776119402985,
+                "recall@10": 0.9507462686567164,
+                "f1@1": 0.6716417910447762,
+                "f1@5": 0.3014925373134329,
+                "f1@10": 0.17286295793758483,
+                "map@1": 0.6716417910447762,
+                "map@5": 0.7671144278606964,
+                "map@10": 0.7735459606728263,
+                "mrr@1": 0.6716417910447762,
+                "mrr@5": 0.7671144278606964,
+                "mrr@10": 0.7735459606728263,
+                "num_pred": 670,
+                "num_data": 670
+            },
+            "VATEX": {
+                "hit@1": 0.43948191156766414,
+                "hit@5": 0.7168378740509156,
+                "hit@10": 0.8164359088878964,
+                "ndcg_linear@1": 0.43948191156766414,
+                "ndcg_linear@5": 0.5876683667478259,
+                "ndcg_linear@10": 0.6199711102927952,
+                "ndcg_exponential@1": 0.43948191156766414,
+                "ndcg_exponential@5": 0.5876683667478259,
+                "ndcg_exponential@10": 0.6199711102927952,
+                "precision@1": 0.43948191156766414,
+                "precision@5": 0.1433675748101831,
+                "precision@10": 0.08164359088878963,
+                "recall@1": 0.43948191156766414,
+                "recall@5": 0.7168378740509156,
+                "recall@10": 0.8164359088878964,
+                "f1@1": 0.43948191156766414,
+                "f1@5": 0.2389459580169719,
+                "f1@10": 0.14844289252507206,
+                "map@1": 0.43948191156766414,
+                "map@5": 0.544606967396159,
+                "map@10": 0.5579919819647376,
+                "mrr@1": 0.43948191156766414,
+                "mrr@5": 0.544606967396159,
+                "mrr@10": 0.5579919819647376,
+                "num_pred": 4478,
+                "num_data": 4478
+            },
+            "YouCook2": {
+                "hit@1": 0.2195659012268009,
+                "hit@5": 0.44510852469329976,
+                "hit@10": 0.5558351682919157,
+                "ndcg_linear@1": 0.2195659012268009,
+                "ndcg_linear@5": 0.33707072315082415,
+                "ndcg_linear@10": 0.3726523748344782,
+                "ndcg_exponential@1": 0.2195659012268009,
+                "ndcg_exponential@5": 0.33707072315082415,
+                "ndcg_exponential@10": 0.3726523748344782,
+                "precision@1": 0.2195659012268009,
+                "precision@5": 0.08902170493865996,
+                "precision@10": 0.05558351682919157,
+                "recall@1": 0.2195659012268009,
+                "recall@5": 0.44510852469329976,
+                "recall@10": 0.5558351682919157,
+                "f1@1": 0.2195659012268009,
+                "f1@5": 0.14836950823109996,
+                "f1@10": 0.10106093968943923,
+                "map@1": 0.2195659012268009,
+                "map@5": 0.30132117017930166,
+                "map@10": 0.31586977036804026,
+                "mrr@1": 0.2195659012268009,
+                "mrr@5": 0.30132117017930166,
+                "mrr@10": 0.31586977036804026,
+                "num_pred": 3179,
+                "num_data": 3179
+            },
+            "QVHighlight": {
+                "hit@1": 0.5817174515235457,
+                "hit@5": 0.8688827331486612,
+                "hit@10": 1.0,
+                "ndcg_linear@1": 0.5817174515235457,
+                "ndcg_linear@5": 0.7355674322372523,
+                "ndcg_linear@10": 0.778061976394854,
+                "ndcg_exponential@1": 0.5817174515235457,
+                "ndcg_exponential@5": 0.7355674322372523,
+                "ndcg_exponential@10": 0.778061976394854,
+                "precision@1": 0.5817174515235457,
+                "precision@5": 0.17377654662973221,
+                "precision@10": 0.09999999999999998,
+                "recall@1": 0.5817174515235457,
+                "recall@5": 0.8688827331486612,
+                "recall@10": 1.0,
+                "f1@1": 0.5817174515235457,
+                "f1@5": 0.28962757771622044,
+                "f1@10": 0.18181818181818185,
+                "map@1": 0.5817174515235457,
+                "map@5": 0.6910587873191752,
+                "map@10": 0.7086550440428556,
+                "mrr@1": 0.5817174515235457,
+                "mrr@5": 0.6910587873191752,
+                "mrr@10": 0.7086550440428556,
+                "num_pred": 1083,
+                "num_data": 1083
+            },
+            "Charades-STA": {
+                "hit@1": 0.24896836313617607,
+                "hit@5": 0.6189821182943603,
+                "hit@10": 1.0,
+                "ndcg_linear@1": 0.24896836313617607,
+                "ndcg_linear@5": 0.4350379203932738,
+                "ndcg_linear@10": 0.554874249964459,
+                "ndcg_exponential@1": 0.24896836313617607,
+                "ndcg_exponential@5": 0.4350379203932738,
+                "ndcg_exponential@10": 0.554874249964459,
+                "precision@1": 0.24896836313617607,
+                "precision@5": 0.12379642365887207,
+                "precision@10": 0.09999999999999998,
+                "recall@1": 0.24896836313617607,
+                "recall@5": 0.6189821182943603,
+                "recall@10": 1.0,
+                "f1@1": 0.24896836313617607,
+                "f1@5": 0.20632737276478683,
+                "f1@10": 0.18181818181818185,
+                "map@1": 0.24896836313617607,
+                "map@5": 0.3748509857863365,
+                "map@10": 0.4223728739547171,
+                "mrr@1": 0.24896836313617607,
+                "mrr@5": 0.3748509857863365,
+                "mrr@10": 0.4223728739547171,
+                "num_pred": 727,
+                "num_data": 727
+            },
+            "MomentSeeker": {
+                "hit@1": 0.46555555555555556,
+                "hit@5": 0.9027777777777778,
+                "hit@10": 1.0,
+                "ndcg_linear@1": 0.46555555555555556,
+                "ndcg_linear@5": 0.7032832283395833,
+                "ndcg_linear@10": 0.7383141215502083,
+                "ndcg_exponential@1": 0.46555555555555556,
+                "ndcg_exponential@5": 0.7032832283395833,
+                "ndcg_exponential@10": 0.7383141215502083,
+                "precision@1": 0.46555555555555556,
+                "precision@5": 0.2011111111111111,
+                "precision@10": 0.11288888888888891,
+                "recall@1": 0.4385185185185185,
+                "recall@5": 0.8939583333333333,
+                "recall@10": 0.9981481481481482,
+                "f1@1": 0.44680511463844796,
+                "f1@5": 0.32284955784955793,
+                "f1@10": 0.20018091476424807,
+                "map@1": 0.46555555555555556,
+                "map@5": 0.635270524691358,
+                "map@10": 0.651133616255144,
+                "mrr@1": 0.46555555555555556,
+                "mrr@5": 0.6413055555555555,
+                "mrr@10": 0.6548293650793651,
+                "num_pred": 1800,
+                "num_data": 1800
+            },
+            "K700": {
+                "hit@1": 0.59,
+                "hit@5": 0.806,
+                "hit@10": 0.857,
+                "ndcg_linear@1": 0.59,
+                "ndcg_linear@5": 0.7077746148817649,
+                "ndcg_linear@10": 0.724000273666642,
+                "ndcg_exponential@1": 0.59,
+                "ndcg_exponential@5": 0.7077746148817649,
+                "ndcg_exponential@10": 0.724000273666642,
+                "precision@1": 0.59,
+                "precision@5": 0.16119999999999998,
+                "precision@10": 0.08570000000000001,
+                "recall@1": 0.59,
+                "recall@5": 0.806,
+                "recall@10": 0.857,
+                "f1@1": 0.59,
+                "f1@5": 0.26866666666666666,
+                "f1@10": 0.1558181818181818,
+                "map@1": 0.59,
+                "map@5": 0.6748333333333334,
+                "map@10": 0.681372619047619,
+                "mrr@1": 0.59,
+                "mrr@5": 0.6748333333333334,
+                "mrr@10": 0.681372619047619,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "SmthSmthV2": {
+                "hit@1": 0.646,
+                "hit@5": 0.866,
+                "hit@10": 0.919,
+                "ndcg_linear@1": 0.646,
+                "ndcg_linear@5": 0.7665689935313655,
+                "ndcg_linear@10": 0.7835822912819611,
+                "ndcg_exponential@1": 0.646,
+                "ndcg_exponential@5": 0.7665689935313655,
+                "ndcg_exponential@10": 0.7835822912819611,
+                "precision@1": 0.646,
+                "precision@5": 0.17320000000000002,
+                "precision@10": 0.09190000000000001,
+                "recall@1": 0.646,
+                "recall@5": 0.866,
+                "recall@10": 0.919,
+                "f1@1": 0.646,
+                "f1@5": 0.2886666666666667,
+                "f1@10": 0.16709090909090907,
+                "map@1": 0.646,
+                "map@5": 0.7331333333333333,
+                "map@10": 0.7400765873015873,
+                "mrr@1": 0.646,
+                "mrr@5": 0.7331333333333333,
+                "mrr@10": 0.7400765873015873,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "HMDB51": {
+                "hit@1": 0.593,
+                "hit@5": 0.811,
+                "hit@10": 0.885,
+                "ndcg_linear@1": 0.593,
+                "ndcg_linear@5": 0.7090441379199225,
+                "ndcg_linear@10": 0.733036952422186,
+                "ndcg_exponential@1": 0.593,
+                "ndcg_exponential@5": 0.7090441379199225,
+                "ndcg_exponential@10": 0.733036952422186,
+                "precision@1": 0.593,
+                "precision@5": 0.16219999999999998,
+                "precision@10": 0.0885,
+                "recall@1": 0.593,
+                "recall@5": 0.811,
+                "recall@10": 0.885,
+                "f1@1": 0.593,
+                "f1@5": 0.27033333333333337,
+                "f1@10": 0.16090909090909095,
+                "map@1": 0.593,
+                "map@5": 0.6751,
+                "map@10": 0.6850373015873016,
+                "mrr@1": 0.593,
+                "mrr@5": 0.6751,
+                "mrr@10": 0.6850373015873016,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "UCF101": {
+                "hit@1": 0.788,
+                "hit@5": 0.965,
+                "hit@10": 0.989,
+                "ndcg_linear@1": 0.788,
+                "ndcg_linear@5": 0.8873256149390782,
+                "ndcg_linear@10": 0.895344170842387,
+                "ndcg_exponential@1": 0.788,
+                "ndcg_exponential@5": 0.8873256149390782,
+                "ndcg_exponential@10": 0.895344170842387,
+                "precision@1": 0.788,
+                "precision@5": 0.19300000000000003,
+                "precision@10": 0.0989,
+                "recall@1": 0.788,
+                "recall@5": 0.965,
+                "recall@10": 0.989,
+                "f1@1": 0.788,
+                "f1@5": 0.3216666666666667,
+                "f1@10": 0.1798181818181818,
+                "map@1": 0.788,
+                "map@5": 0.8610833333333333,
+                "map@10": 0.8645432539682539,
+                "mrr@1": 0.788,
+                "mrr@5": 0.8610833333333333,
+                "mrr@10": 0.8645432539682539,
+                "num_pred": 1000,
+                "num_data": 1000
+            },
+            "Breakfast": {
+                "hit@1": 0.3672055427251732,
+                "hit@5": 0.789838337182448,
+                "hit@10": 1.0,
+                "ndcg_linear@1": 0.3672055427251732,
+                "ndcg_linear@5": 0.5786460297178996,
+                "ndcg_linear@10": 0.6478264872451965,
+                "ndcg_exponential@1": 0.3672055427251732,
+                "ndcg_exponential@5": 0.5786460297178996,
+                "ndcg_exponential@10": 0.6478264872451965,
+                "precision@1": 0.3672055427251732,
+                "precision@5": 0.15796766743648963,
+                "precision@10": 0.1,
+                "recall@1": 0.3672055427251732,
+                "recall@5": 0.789838337182448,
+                "recall@10": 1.0,
+                "f1@1": 0.3672055427251732,
+                "f1@5": 0.26327944572748274,
+                "f1@10": 0.1818181818181818,
+                "map@1": 0.3672055427251732,
+                "map@5": 0.5095073133179369,
+                "map@10": 0.5387706660801349,
+                "mrr@1": 0.3672055427251732,
+                "mrr@5": 0.5095073133179369,
+                "mrr@10": 0.5387706660801349,
+                "num_pred": 433,
+                "num_data": 433
+            },
+            "MVBench": {
+                "hit@1": 0.56975,
+                "hit@5": 1.0,
+                "hit@10": 1.0,
+                "ndcg_linear@1": 0.56975,
+                "ndcg_linear@5": 0.8110591965257652,
+                "ndcg_linear@10": 0.8110591965257652,
+                "ndcg_exponential@1": 0.56975,
+                "ndcg_exponential@5": 0.8110591965257652,
+                "ndcg_exponential@10": 0.8110591965257652,
+                "precision@1": 0.56975,
+                "precision@5": 0.20000000000000004,
+                "precision@10": 0.10000000000000002,
+                "recall@1": 0.56975,
+                "recall@5": 1.0,
+                "recall@10": 1.0,
+                "f1@1": 0.56975,
+                "f1@5": 0.3333333333333333,
+                "f1@10": 0.18181818181818182,
+                "map@1": 0.56975,
+                "map@5": 0.746825,
+                "map@10": 0.746825,
+                "mrr@1": 0.56975,
+                "mrr@5": 0.746825,
+                "mrr@10": 0.746825,
+                "num_pred": 4000,
+                "num_data": 4000
+            },
+            "Video-MME": {
+                "hit@1": 0.5088888888888888,
+                "hit@5": 1.0,
+                "hit@10": 1.0,
+                "ndcg_linear@1": 0.5088888888888888,
+                "ndcg_linear@5": 0.7738633421698715,
+                "ndcg_linear@10": 0.7738633421698715,
+                "ndcg_exponential@1": 0.5088888888888888,
+                "ndcg_exponential@5": 0.7738633421698715,
+                "ndcg_exponential@10": 0.7738633421698715,
+                "precision@1": 0.5088888888888888,
+                "precision@5": 0.19999999999999996,
+                "precision@10": 0.09999999999999998,
+                "recall@1": 0.5088888888888888,
+                "recall@5": 1.0,
+                "recall@10": 1.0,
+                "f1@1": 0.5088888888888888,
+                "f1@5": 0.3333333333333334,
+                "f1@10": 0.18181818181818188,
+                "map@1": 0.5088888888888888,
+                "map@5": 0.6979012345679012,
+                "map@10": 0.6979012345679012,
+                "mrr@1": 0.5088888888888888,
+                "mrr@5": 0.6979012345679012,
+                "mrr@10": 0.6979012345679012,
+                "num_pred": 2700,
+                "num_data": 2700
+            },
+            "NExTQA": {
+                "hit@1": 0.6705978514712752,
+                "hit@5": 1.0,
+                "hit@10": 1.0,
+                "ndcg_linear@1": 0.6705978514712752,
+                "ndcg_linear@5": 0.8495446170574841,
+                "ndcg_linear@10": 0.8495446170574841,
+                "ndcg_exponential@1": 0.6705978514712752,
+                "ndcg_exponential@5": 0.8495446170574841,
+                "ndcg_exponential@10": 0.8495446170574841,
+                "precision@1": 0.6705978514712752,
+                "precision@5": 0.20000000000000004,
+                "precision@10": 0.10000000000000002,
+                "recall@1": 0.6705978514712752,
+                "recall@5": 1.0,
+                "recall@10": 1.0,
+                "f1@1": 0.6705978514712752,
+                "f1@5": 0.3333333333333333,
+                "f1@10": 0.1818181818181818,
+                "map@1": 0.6705978514712752,
+                "map@5": 0.7991339716643312,
+                "map@10": 0.7991339716643312,
+                "mrr@1": 0.6705978514712752,
+                "mrr@5": 0.7989588198661062,
+                "mrr@10": 0.7989588198661062,
+                "num_pred": 8564,
+                "num_data": 8564
+            },
+            "EgoSchema": {
+                "hit@1": 0.596,
+                "hit@5": 1.0,
+                "hit@10": 1.0,
+                "ndcg_linear@1": 0.596,
+                "ndcg_linear@5": 0.815692468749886,
+                "ndcg_linear@10": 0.815692468749886,
+                "ndcg_exponential@1": 0.596,
+                "ndcg_exponential@5": 0.815692468749886,
+                "ndcg_exponential@10": 0.815692468749886,
+                "precision@1": 0.596,
+                "precision@5": 0.2,
+                "precision@10": 0.1,
+                "recall@1": 0.596,
+                "recall@5": 1.0,
+                "recall@10": 1.0,
+                "f1@1": 0.596,
+                "f1@5": 0.3333333333333333,
+                "f1@10": 0.18181818181818182,
+                "map@1": 0.596,
+                "map@5": 0.7538333333333332,
+                "map@10": 0.7538333333333332,
+                "mrr@1": 0.596,
+                "mrr@5": 0.7538333333333332,
+                "mrr@10": 0.7538333333333332,
+                "num_pred": 500,
+                "num_data": 500
+            },
+            "ActivityNetQA": {
+                "hit@1": 0.766,
+                "hit@5": 1.0,
+                "hit@10": 1.0,
+                "ndcg_linear@1": 0.766,
+                "ndcg_linear@5": 0.9136375623357211,
+                "ndcg_linear@10": 0.9136375623357211,
+                "ndcg_exponential@1": 0.766,
+                "ndcg_exponential@5": 0.9136375623357211,
+                "ndcg_exponential@10": 0.9136375623357211,
+                "precision@1": 0.766,
+                "precision@5": 0.20000000000000004,
+                "precision@10": 0.10000000000000002,
+                "recall@1": 0.766,
+                "recall@5": 1.0,
+                "recall@10": 1.0,
+                "f1@1": 0.766,
+                "f1@5": 0.3333333333333333,
+                "f1@10": 0.18181818181818182,
+                "map@1": 0.766,
+                "map@5": 0.883,
+                "map@10": 0.883,
+                "mrr@1": 0.766,
+                "mrr@5": 0.883,
+                "mrr@10": 0.883,
+                "num_pred": 1000,
+                "num_data": 1000
+            }
+        }
+    }
+}

special_tokens_map.json

@@ -0,0 +1,31 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:091aa7594dc2fcfbfa06b9e3c22a5f0562ac14f30375c13af7309407a0e67b8a
+size 11420371

tokenizer_config.json

@@ -0,0 +1,145 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "chat_template": "{% set image_count = namespace(value=0) %}{% set video_count = namespace(value=0) %}{% for message in messages %}{% if loop.first and message['role'] != 'system' %}<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n{% endif %}<|im_start|>{{ message['role'] }}\n{% if message['content'] is string %}{{ message['content'] }}<|im_end|>\n{% else %}{% for content in message['content'] %}{% if content['type'] == 'image' or 'image' in content or 'image_url' in content %}{% set image_count.value = image_count.value + 1 %}{% if add_vision_id %}Picture {{ image_count.value }}: {% endif %}<|vision_start|><|image_pad|><|vision_end|>{% elif content['type'] == 'video' or 'video' in content %}{% set video_count.value = video_count.value + 1 %}{% if add_vision_id %}Video {{ video_count.value }}: {% endif %}<|vision_start|><|video_pad|><|vision_end|>{% elif 'text' in content %}{{ content['text'] }}{% endif %}{% endfor %}<|im_end|>\n{% endif %}{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant\n{% endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 32768,
+  "pad_token": "<|endoftext|>",
+  "padding_side": "left",
+  "processor_class": "Qwen2VLProcessor",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}