Hugging Face's logo

Spaces:
TIGER-Lab
/
MMEB-Leaderboard
Running

App Files Community
73
MINGYISU commited on
Commit
c987e2d
·
verified ·
1 Parent(s): fc3db9e

scores update after eval code issues fixed (#56)

Browse files

- scores update (c579c3b842e7e329496cf54e82f99cd62350d1d6)
- update submission instruction (c0b9140358e08b1d04b60ee4aa6b09decc08b4ae)
- fix issues (3debfaf0b01dd60e26ef830a51e0da979b8073f7)

Files changed (10) hide show
  1. scores/LamRA-Ret-Qwen2.5VL-7b.json +0 -0
  2. scores/LamRA-Ret.json +0 -0
  3. scores/VLM2Vec-V1-Qwen2VL-2B.json +0 -0
  4. scores/VLM2Vec-V1-Qwen2VL-7B.json +0 -0
  5. scores/VLM2Vec-V2.0-Qwen2VL-2B.json +0 -0
  6. scores/colpali-v1.3.json +0 -0
  7. scores/gme-Qwen2-VL-2B-Instruct.json +0 -0
  8. scores/gme-Qwen2-VL-7B-Instruct.json +2198 -0
  9. utils.py +7 -8
  10. utils_v2.py +3 -3
scores/LamRA-Ret-Qwen2.5VL-7b.json CHANGED
The diff for this file is too large to render. See raw diff
 
scores/LamRA-Ret.json CHANGED
The diff for this file is too large to render. See raw diff
 
scores/VLM2Vec-V1-Qwen2VL-2B.json CHANGED
The diff for this file is too large to render. See raw diff
 
scores/VLM2Vec-V1-Qwen2VL-7B.json CHANGED
The diff for this file is too large to render. See raw diff
 
scores/VLM2Vec-V2.0-Qwen2VL-2B.json CHANGED
The diff for this file is too large to render. See raw diff
 
scores/colpali-v1.3.json CHANGED
The diff for this file is too large to render. See raw diff
 
scores/gme-Qwen2-VL-2B-Instruct.json CHANGED
The diff for this file is too large to render. See raw diff
 
scores/gme-Qwen2-VL-7B-Instruct.json ADDED
@@ -0,0 +1,2198 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "metadata": {
3
+ "model_name": "gme-Qwen2-VL-7B-Instruct",
4
+ "report_generated_date": "2025-06-28T08:54:18.334782"
5
+ },
6
+ "metrics": {
7
+ "image": {
8
+ "ImageNet-1K": {
9
+ "hit@1": 0.646,
10
+ "hit@5": 0.864,
11
+ "hit@10": 0.914,
12
+ "ndcg_linear@1": 0.646,
13
+ "ndcg_linear@5": 0.7675656151573618,
14
+ "ndcg_linear@10": 0.7837300096011678,
15
+ "ndcg_exponential@1": 0.646,
16
+ "ndcg_exponential@5": 0.7675656151573618,
17
+ "ndcg_exponential@10": 0.7837300096011678,
18
+ "precision@1": 0.646,
19
+ "precision@5": 0.1728,
20
+ "precision@10": 0.09140000000000001,
21
+ "recall@1": 0.646,
22
+ "recall@5": 0.864,
23
+ "recall@10": 0.914,
24
+ "f1@1": 0.646,
25
+ "f1@5": 0.288,
26
+ "f1@10": 0.16618181818181818,
27
+ "map@1": 0.646,
28
+ "map@5": 0.7350166666666667,
29
+ "map@10": 0.741684126984127,
30
+ "mrr@1": 0.646,
31
+ "mrr@5": 0.7350166666666667,
32
+ "mrr@10": 0.741684126984127,
33
+ "num_pred": 1000,
34
+ "num_data": 1000
35
+ },
36
+ "N24News": {
37
+ "hit@1": 0.505,
38
+ "hit@5": 0.838,
39
+ "hit@10": 0.93,
40
+ "ndcg_linear@1": 0.505,
41
+ "ndcg_linear@5": 0.6859312278730303,
42
+ "ndcg_linear@10": 0.7160982776534236,
43
+ "ndcg_exponential@1": 0.505,
44
+ "ndcg_exponential@5": 0.6859312278730303,
45
+ "ndcg_exponential@10": 0.7160982776534236,
46
+ "precision@1": 0.505,
47
+ "precision@5": 0.1676,
48
+ "precision@10": 0.09300000000000003,
49
+ "recall@1": 0.505,
50
+ "recall@5": 0.838,
51
+ "recall@10": 0.93,
52
+ "f1@1": 0.505,
53
+ "f1@5": 0.2793333333333334,
54
+ "f1@10": 0.1690909090909091,
55
+ "map@1": 0.505,
56
+ "map@5": 0.635,
57
+ "map@10": 0.6476924603174602,
58
+ "mrr@1": 0.505,
59
+ "mrr@5": 0.635,
60
+ "mrr@10": 0.6476924603174602,
61
+ "num_pred": 1000,
62
+ "num_data": 1000
63
+ },
64
+ "HatefulMemes": {
65
+ "hit@1": 0.536,
66
+ "hit@5": 1.0,
67
+ "hit@10": 1.0,
68
+ "ndcg_linear@1": 0.536,
69
+ "ndcg_linear@5": 0.8287514056571564,
70
+ "ndcg_linear@10": 0.8287514056571564,
71
+ "ndcg_exponential@1": 0.536,
72
+ "ndcg_exponential@5": 0.8287514056571564,
73
+ "ndcg_exponential@10": 0.8287514056571564,
74
+ "precision@1": 0.536,
75
+ "precision@5": 0.20000000000000004,
76
+ "precision@10": 0.10000000000000002,
77
+ "recall@1": 0.536,
78
+ "recall@5": 1.0,
79
+ "recall@10": 1.0,
80
+ "f1@1": 0.536,
81
+ "f1@5": 0.3333333333333333,
82
+ "f1@10": 0.18181818181818182,
83
+ "map@1": 0.536,
84
+ "map@5": 0.768,
85
+ "map@10": 0.768,
86
+ "mrr@1": 0.536,
87
+ "mrr@5": 0.768,
88
+ "mrr@10": 0.768,
89
+ "num_pred": 1000,
90
+ "num_data": 1000
91
+ },
92
+ "VOC2007": {
93
+ "hit@1": 0.803,
94
+ "hit@5": 0.966,
95
+ "hit@10": 0.99,
96
+ "ndcg_linear@1": 0.803,
97
+ "ndcg_linear@5": 0.8956917699369619,
98
+ "ndcg_linear@10": 0.9035303593848166,
99
+ "ndcg_exponential@1": 0.803,
100
+ "ndcg_exponential@5": 0.8956917699369619,
101
+ "ndcg_exponential@10": 0.9035303593848166,
102
+ "precision@1": 0.803,
103
+ "precision@5": 0.19320000000000004,
104
+ "precision@10": 0.09900000000000002,
105
+ "recall@1": 0.803,
106
+ "recall@5": 0.966,
107
+ "recall@10": 0.99,
108
+ "f1@1": 0.803,
109
+ "f1@5": 0.322,
110
+ "f1@10": 0.18,
111
+ "map@1": 0.803,
112
+ "map@5": 0.8717333333333334,
113
+ "map@10": 0.8750126984126985,
114
+ "mrr@1": 0.803,
115
+ "mrr@5": 0.8717333333333334,
116
+ "mrr@10": 0.8750126984126985,
117
+ "num_pred": 1000,
118
+ "num_data": 1000
119
+ },
120
+ "SUN397": {
121
+ "hit@1": 0.695,
122
+ "hit@5": 0.93,
123
+ "hit@10": 0.964,
124
+ "ndcg_linear@1": 0.695,
125
+ "ndcg_linear@5": 0.8264697358260907,
126
+ "ndcg_linear@10": 0.8377506399258345,
127
+ "ndcg_exponential@1": 0.695,
128
+ "ndcg_exponential@5": 0.8264697358260907,
129
+ "ndcg_exponential@10": 0.8377506399258345,
130
+ "precision@1": 0.695,
131
+ "precision@5": 0.18600000000000003,
132
+ "precision@10": 0.0964,
133
+ "recall@1": 0.695,
134
+ "recall@5": 0.93,
135
+ "recall@10": 0.964,
136
+ "f1@1": 0.695,
137
+ "f1@5": 0.31,
138
+ "f1@10": 0.17527272727272725,
139
+ "map@1": 0.695,
140
+ "map@5": 0.7914666666666665,
141
+ "map@10": 0.7962904761904762,
142
+ "mrr@1": 0.695,
143
+ "mrr@5": 0.7914666666666665,
144
+ "mrr@10": 0.7962904761904762,
145
+ "num_pred": 1000,
146
+ "num_data": 1000
147
+ },
148
+ "Place365": {
149
+ "hit@1": 0.391,
150
+ "hit@5": 0.695,
151
+ "hit@10": 0.786,
152
+ "ndcg_linear@1": 0.391,
153
+ "ndcg_linear@5": 0.5529304236707225,
154
+ "ndcg_linear@10": 0.5829135990155158,
155
+ "ndcg_exponential@1": 0.391,
156
+ "ndcg_exponential@5": 0.5529304236707225,
157
+ "ndcg_exponential@10": 0.5829135990155158,
158
+ "precision@1": 0.391,
159
+ "precision@5": 0.13899999999999998,
160
+ "precision@10": 0.07859999999999999,
161
+ "recall@1": 0.391,
162
+ "recall@5": 0.695,
163
+ "recall@10": 0.786,
164
+ "f1@1": 0.391,
165
+ "f1@5": 0.23166666666666674,
166
+ "f1@10": 0.14290909090909093,
167
+ "map@1": 0.391,
168
+ "map@5": 0.50565,
169
+ "map@10": 0.51835,
170
+ "mrr@1": 0.391,
171
+ "mrr@5": 0.50565,
172
+ "mrr@10": 0.51835,
173
+ "num_pred": 1000,
174
+ "num_data": 1000
175
+ },
176
+ "ImageNet-A": {
177
+ "hit@1": 0.412,
178
+ "hit@5": 0.692,
179
+ "hit@10": 0.776,
180
+ "ndcg_linear@1": 0.412,
181
+ "ndcg_linear@5": 0.5641118126578005,
182
+ "ndcg_linear@10": 0.5913889513202993,
183
+ "ndcg_exponential@1": 0.412,
184
+ "ndcg_exponential@5": 0.5641118126578005,
185
+ "ndcg_exponential@10": 0.5913889513202993,
186
+ "precision@1": 0.412,
187
+ "precision@5": 0.1384,
188
+ "precision@10": 0.07759999999999999,
189
+ "recall@1": 0.412,
190
+ "recall@5": 0.692,
191
+ "recall@10": 0.776,
192
+ "f1@1": 0.412,
193
+ "f1@5": 0.23066666666666671,
194
+ "f1@10": 0.14109090909090913,
195
+ "map@1": 0.412,
196
+ "map@5": 0.5212666666666667,
197
+ "map@10": 0.532590873015873,
198
+ "mrr@1": 0.412,
199
+ "mrr@5": 0.5212666666666667,
200
+ "mrr@10": 0.532590873015873,
201
+ "num_pred": 1000,
202
+ "num_data": 1000
203
+ },
204
+ "ImageNet-R": {
205
+ "hit@1": 0.839,
206
+ "hit@5": 0.96,
207
+ "hit@10": 0.978,
208
+ "ndcg_linear@1": 0.839,
209
+ "ndcg_linear@5": 0.9069125896441956,
210
+ "ndcg_linear@10": 0.9128589977894869,
211
+ "ndcg_exponential@1": 0.839,
212
+ "ndcg_exponential@5": 0.9069125896441956,
213
+ "ndcg_exponential@10": 0.9128589977894869,
214
+ "precision@1": 0.839,
215
+ "precision@5": 0.192,
216
+ "precision@10": 0.09780000000000001,
217
+ "recall@1": 0.839,
218
+ "recall@5": 0.96,
219
+ "recall@10": 0.978,
220
+ "f1@1": 0.839,
221
+ "f1@5": 0.32,
222
+ "f1@10": 0.17781818181818182,
223
+ "map@1": 0.839,
224
+ "map@5": 0.8889333333333334,
225
+ "map@10": 0.8914634920634922,
226
+ "mrr@1": 0.839,
227
+ "mrr@5": 0.8889333333333334,
228
+ "mrr@10": 0.8914634920634922,
229
+ "num_pred": 1000,
230
+ "num_data": 1000
231
+ },
232
+ "ObjectNet": {
233
+ "hit@1": 0.69,
234
+ "hit@5": 0.859,
235
+ "hit@10": 0.915,
236
+ "ndcg_linear@1": 0.69,
237
+ "ndcg_linear@5": 0.7820151814813446,
238
+ "ndcg_linear@10": 0.8002958335952403,
239
+ "ndcg_exponential@1": 0.69,
240
+ "ndcg_exponential@5": 0.7820151814813446,
241
+ "ndcg_exponential@10": 0.8002958335952403,
242
+ "precision@1": 0.69,
243
+ "precision@5": 0.1718,
244
+ "precision@10": 0.09150000000000001,
245
+ "recall@1": 0.69,
246
+ "recall@5": 0.859,
247
+ "recall@10": 0.915,
248
+ "f1@1": 0.69,
249
+ "f1@5": 0.2863333333333334,
250
+ "f1@10": 0.16636363636363635,
251
+ "map@1": 0.69,
252
+ "map@5": 0.7562333333333333,
253
+ "map@10": 0.7638757936507936,
254
+ "mrr@1": 0.69,
255
+ "mrr@5": 0.7562333333333333,
256
+ "mrr@10": 0.7638757936507936,
257
+ "num_pred": 1000,
258
+ "num_data": 1000
259
+ },
260
+ "Country211": {
261
+ "hit@1": 0.248,
262
+ "hit@5": 0.457,
263
+ "hit@10": 0.559,
264
+ "ndcg_linear@1": 0.248,
265
+ "ndcg_linear@5": 0.3569485635087294,
266
+ "ndcg_linear@10": 0.3901006229753951,
267
+ "ndcg_exponential@1": 0.248,
268
+ "ndcg_exponential@5": 0.3569485635087294,
269
+ "ndcg_exponential@10": 0.3901006229753951,
270
+ "precision@1": 0.248,
271
+ "precision@5": 0.09140000000000001,
272
+ "precision@10": 0.0559,
273
+ "recall@1": 0.248,
274
+ "recall@5": 0.457,
275
+ "recall@10": 0.559,
276
+ "f1@1": 0.248,
277
+ "f1@5": 0.15233333333333338,
278
+ "f1@10": 0.10163636363636365,
279
+ "map@1": 0.248,
280
+ "map@5": 0.32385,
281
+ "map@10": 0.33762063492063493,
282
+ "mrr@1": 0.248,
283
+ "mrr@5": 0.32385,
284
+ "mrr@10": 0.33762063492063493,
285
+ "num_pred": 1000,
286
+ "num_data": 1000
287
+ },
288
+ "OK-VQA": {
289
+ "hit@1": 0.332,
290
+ "hit@5": 0.617,
291
+ "hit@10": 0.717,
292
+ "ndcg_linear@1": 0.332,
293
+ "ndcg_linear@5": 0.485871864688508,
294
+ "ndcg_linear@10": 0.5182227215507165,
295
+ "ndcg_exponential@1": 0.332,
296
+ "ndcg_exponential@5": 0.485871864688508,
297
+ "ndcg_exponential@10": 0.5182227215507165,
298
+ "precision@1": 0.332,
299
+ "precision@5": 0.12340000000000001,
300
+ "precision@10": 0.07169999999999999,
301
+ "recall@1": 0.332,
302
+ "recall@5": 0.617,
303
+ "recall@10": 0.717,
304
+ "f1@1": 0.332,
305
+ "f1@5": 0.20566666666666672,
306
+ "f1@10": 0.13036363636363638,
307
+ "map@1": 0.332,
308
+ "map@5": 0.4420333333333333,
309
+ "map@10": 0.45539007936507936,
310
+ "mrr@1": 0.332,
311
+ "mrr@5": 0.4420333333333333,
312
+ "mrr@10": 0.45539007936507936,
313
+ "num_pred": 1000,
314
+ "num_data": 1000
315
+ },
316
+ "A-OKVQA": {
317
+ "hit@1": 0.21,
318
+ "hit@5": 0.5,
319
+ "hit@10": 0.638,
320
+ "ndcg_linear@1": 0.21,
321
+ "ndcg_linear@5": 0.36039694414599044,
322
+ "ndcg_linear@10": 0.40511531278859386,
323
+ "ndcg_exponential@1": 0.21,
324
+ "ndcg_exponential@5": 0.36039694414599044,
325
+ "ndcg_exponential@10": 0.40511531278859386,
326
+ "precision@1": 0.21,
327
+ "precision@5": 0.1,
328
+ "precision@10": 0.0638,
329
+ "recall@1": 0.21,
330
+ "recall@5": 0.5,
331
+ "recall@10": 0.638,
332
+ "f1@1": 0.21,
333
+ "f1@5": 0.1666666666666667,
334
+ "f1@10": 0.11600000000000002,
335
+ "map@1": 0.21,
336
+ "map@5": 0.31425,
337
+ "map@10": 0.3327503968253968,
338
+ "mrr@1": 0.21,
339
+ "mrr@5": 0.31425,
340
+ "mrr@10": 0.3327503968253968,
341
+ "num_pred": 1000,
342
+ "num_data": 1000
343
+ },
344
+ "DocVQA": {
345
+ "hit@1": 0.414,
346
+ "hit@5": 0.597,
347
+ "hit@10": 0.664,
348
+ "ndcg_linear@1": 0.414,
349
+ "ndcg_linear@5": 0.5132212142752178,
350
+ "ndcg_linear@10": 0.5351036098010266,
351
+ "ndcg_exponential@1": 0.414,
352
+ "ndcg_exponential@5": 0.5132212142752178,
353
+ "ndcg_exponential@10": 0.5351036098010266,
354
+ "precision@1": 0.414,
355
+ "precision@5": 0.11939999999999998,
356
+ "precision@10": 0.06639999999999997,
357
+ "recall@1": 0.414,
358
+ "recall@5": 0.597,
359
+ "recall@10": 0.664,
360
+ "f1@1": 0.414,
361
+ "f1@5": 0.19900000000000007,
362
+ "f1@10": 0.12072727272727275,
363
+ "map@1": 0.414,
364
+ "map@5": 0.48518333333333336,
365
+ "map@10": 0.4943369047619047,
366
+ "mrr@1": 0.414,
367
+ "mrr@5": 0.48518333333333336,
368
+ "mrr@10": 0.4943369047619047,
369
+ "num_pred": 1000,
370
+ "num_data": 1000
371
+ },
372
+ "InfographicsVQA": {
373
+ "hit@1": 0.203,
374
+ "hit@5": 0.383,
375
+ "hit@10": 0.467,
376
+ "ndcg_linear@1": 0.203,
377
+ "ndcg_linear@5": 0.2983142031321632,
378
+ "ndcg_linear@10": 0.3252840110636628,
379
+ "ndcg_exponential@1": 0.203,
380
+ "ndcg_exponential@5": 0.2983142031321632,
381
+ "ndcg_exponential@10": 0.3252840110636628,
382
+ "precision@1": 0.203,
383
+ "precision@5": 0.07659999999999999,
384
+ "precision@10": 0.0467,
385
+ "recall@1": 0.203,
386
+ "recall@5": 0.383,
387
+ "recall@10": 0.467,
388
+ "f1@1": 0.203,
389
+ "f1@5": 0.1276666666666667,
390
+ "f1@10": 0.08490909090909092,
391
+ "map@1": 0.203,
392
+ "map@5": 0.2702,
393
+ "map@10": 0.2812119047619048,
394
+ "mrr@1": 0.203,
395
+ "mrr@5": 0.2702,
396
+ "mrr@10": 0.2812119047619048,
397
+ "num_pred": 1000,
398
+ "num_data": 1000
399
+ },
400
+ "ChartQA": {
401
+ "hit@1": 0.178,
402
+ "hit@5": 0.257,
403
+ "hit@10": 0.301,
404
+ "ndcg_linear@1": 0.178,
405
+ "ndcg_linear@5": 0.21916174591757567,
406
+ "ndcg_linear@10": 0.23326245822360894,
407
+ "ndcg_exponential@1": 0.178,
408
+ "ndcg_exponential@5": 0.21916174591757567,
409
+ "ndcg_exponential@10": 0.23326245822360894,
410
+ "precision@1": 0.178,
411
+ "precision@5": 0.05140000000000001,
412
+ "precision@10": 0.030100000000000002,
413
+ "recall@1": 0.178,
414
+ "recall@5": 0.257,
415
+ "recall@10": 0.301,
416
+ "f1@1": 0.178,
417
+ "f1@5": 0.08566666666666668,
418
+ "f1@10": 0.054727272727272736,
419
+ "map@1": 0.178,
420
+ "map@5": 0.20663333333333334,
421
+ "map@10": 0.21237777777777778,
422
+ "mrr@1": 0.178,
423
+ "mrr@5": 0.20663333333333334,
424
+ "mrr@10": 0.21237777777777778,
425
+ "num_pred": 1000,
426
+ "num_data": 1000
427
+ },
428
+ "Visual7W": {
429
+ "hit@1": 0.222,
430
+ "hit@5": 0.544,
431
+ "hit@10": 0.674,
432
+ "ndcg_linear@1": 0.222,
433
+ "ndcg_linear@5": 0.3896222077826222,
434
+ "ndcg_linear@10": 0.43208253526314966,
435
+ "ndcg_exponential@1": 0.222,
436
+ "ndcg_exponential@5": 0.3896222077826222,
437
+ "ndcg_exponential@10": 0.43208253526314966,
438
+ "precision@1": 0.222,
439
+ "precision@5": 0.10879999999999998,
440
+ "precision@10": 0.06739999999999999,
441
+ "recall@1": 0.222,
442
+ "recall@5": 0.544,
443
+ "recall@10": 0.674,
444
+ "f1@1": 0.222,
445
+ "f1@5": 0.1813333333333334,
446
+ "f1@10": 0.12254545454545457,
447
+ "map@1": 0.222,
448
+ "map@5": 0.33858333333333335,
449
+ "map@10": 0.3563468253968254,
450
+ "mrr@1": 0.222,
451
+ "mrr@5": 0.33858333333333335,
452
+ "mrr@10": 0.3563468253968254,
453
+ "num_pred": 1000,
454
+ "num_data": 1000
455
+ },
456
+ "ScienceQA": {
457
+ "hit@1": 0.28,
458
+ "hit@5": 0.64,
459
+ "hit@10": 0.763,
460
+ "ndcg_linear@1": 0.28,
461
+ "ndcg_linear@5": 0.4740675519059985,
462
+ "ndcg_linear@10": 0.514010974748877,
463
+ "ndcg_exponential@1": 0.28,
464
+ "ndcg_exponential@5": 0.4740675519059985,
465
+ "ndcg_exponential@10": 0.514010974748877,
466
+ "precision@1": 0.28,
467
+ "precision@5": 0.12799999999999997,
468
+ "precision@10": 0.07629999999999999,
469
+ "recall@1": 0.28,
470
+ "recall@5": 0.64,
471
+ "recall@10": 0.763,
472
+ "f1@1": 0.28,
473
+ "f1@5": 0.21333333333333337,
474
+ "f1@10": 0.13872727272727275,
475
+ "map@1": 0.28,
476
+ "map@5": 0.4186333333333333,
477
+ "map@10": 0.43520793650793654,
478
+ "mrr@1": 0.28,
479
+ "mrr@5": 0.4186333333333333,
480
+ "mrr@10": 0.43520793650793654,
481
+ "num_pred": 1000,
482
+ "num_data": 1000
483
+ },
484
+ "VizWiz": {
485
+ "hit@1": 0.39,
486
+ "hit@5": 0.613,
487
+ "hit@10": 0.673,
488
+ "ndcg_linear@1": 0.39,
489
+ "ndcg_linear@5": 0.5101413361528128,
490
+ "ndcg_linear@10": 0.5295620828845852,
491
+ "ndcg_exponential@1": 0.39,
492
+ "ndcg_exponential@5": 0.5101413361528128,
493
+ "ndcg_exponential@10": 0.5295620828845852,
494
+ "precision@1": 0.39,
495
+ "precision@5": 0.1226,
496
+ "precision@10": 0.0673,
497
+ "recall@1": 0.39,
498
+ "recall@5": 0.613,
499
+ "recall@10": 0.673,
500
+ "f1@1": 0.39,
501
+ "f1@5": 0.2043333333333334,
502
+ "f1@10": 0.12236363636363638,
503
+ "map@1": 0.39,
504
+ "map@5": 0.4757833333333333,
505
+ "map@10": 0.4838107142857143,
506
+ "mrr@1": 0.39,
507
+ "mrr@5": 0.4757833333333333,
508
+ "mrr@10": 0.4838107142857143,
509
+ "num_pred": 1000,
510
+ "num_data": 1000
511
+ },
512
+ "GQA": {
513
+ "hit@1": 0.769,
514
+ "hit@5": 0.941,
515
+ "hit@10": 0.965,
516
+ "ndcg_linear@1": 0.769,
517
+ "ndcg_linear@5": 0.865366522766234,
518
+ "ndcg_linear@10": 0.8730495254605352,
519
+ "ndcg_exponential@1": 0.769,
520
+ "ndcg_exponential@5": 0.865366522766234,
521
+ "ndcg_exponential@10": 0.8730495254605352,
522
+ "precision@1": 0.769,
523
+ "precision@5": 0.1882,
524
+ "precision@10": 0.0965,
525
+ "recall@1": 0.769,
526
+ "recall@5": 0.941,
527
+ "recall@10": 0.965,
528
+ "f1@1": 0.769,
529
+ "f1@5": 0.31366666666666665,
530
+ "f1@10": 0.17545454545454545,
531
+ "map@1": 0.769,
532
+ "map@5": 0.8397666666666667,
533
+ "map@10": 0.8428908730158731,
534
+ "mrr@1": 0.769,
535
+ "mrr@5": 0.8397666666666667,
536
+ "mrr@10": 0.8428908730158731,
537
+ "num_pred": 1000,
538
+ "num_data": 1000
539
+ },
540
+ "TextVQA": {
541
+ "hit@1": 0.468,
542
+ "hit@5": 0.685,
543
+ "hit@10": 0.74,
544
+ "ndcg_linear@1": 0.468,
545
+ "ndcg_linear@5": 0.5857686784282851,
546
+ "ndcg_linear@10": 0.6036026979398256,
547
+ "ndcg_exponential@1": 0.468,
548
+ "ndcg_exponential@5": 0.5857686784282851,
549
+ "ndcg_exponential@10": 0.6036026979398256,
550
+ "precision@1": 0.468,
551
+ "precision@5": 0.137,
552
+ "precision@10": 0.074,
553
+ "recall@1": 0.468,
554
+ "recall@5": 0.685,
555
+ "recall@10": 0.74,
556
+ "f1@1": 0.468,
557
+ "f1@5": 0.22833333333333336,
558
+ "f1@10": 0.13454545454545455,
559
+ "map@1": 0.468,
560
+ "map@5": 0.5525333333333333,
561
+ "map@10": 0.5599246031746031,
562
+ "mrr@1": 0.468,
563
+ "mrr@5": 0.5525333333333333,
564
+ "mrr@10": 0.5599246031746031,
565
+ "num_pred": 1000,
566
+ "num_data": 1000
567
+ },
568
+ "VisDial": {
569
+ "hit@1": 0.608,
570
+ "hit@5": 0.843,
571
+ "hit@10": 0.914,
572
+ "ndcg_linear@1": 0.608,
573
+ "ndcg_linear@5": 0.733577948489228,
574
+ "ndcg_linear@10": 0.7570027510391492,
575
+ "ndcg_exponential@1": 0.608,
576
+ "ndcg_exponential@5": 0.733577948489228,
577
+ "ndcg_exponential@10": 0.7570027510391492,
578
+ "precision@1": 0.608,
579
+ "precision@5": 0.1686,
580
+ "precision@10": 0.09140000000000001,
581
+ "recall@1": 0.608,
582
+ "recall@5": 0.843,
583
+ "recall@10": 0.914,
584
+ "f1@1": 0.608,
585
+ "f1@5": 0.281,
586
+ "f1@10": 0.16618181818181818,
587
+ "map@1": 0.608,
588
+ "map@5": 0.6971499999999999,
589
+ "map@10": 0.7070952380952381,
590
+ "mrr@1": 0.608,
591
+ "mrr@5": 0.6971499999999999,
592
+ "mrr@10": 0.7070952380952381,
593
+ "num_pred": 1000,
594
+ "num_data": 1000
595
+ },
596
+ "CIRR": {
597
+ "hit@1": 0.549,
598
+ "hit@5": 0.89,
599
+ "hit@10": 0.944,
600
+ "ndcg_linear@1": 0.549,
601
+ "ndcg_linear@5": 0.7379230235182084,
602
+ "ndcg_linear@10": 0.7559089463717755,
603
+ "ndcg_exponential@1": 0.549,
604
+ "ndcg_exponential@5": 0.7379230235182084,
605
+ "ndcg_exponential@10": 0.7559089463717755,
606
+ "precision@1": 0.549,
607
+ "precision@5": 0.178,
608
+ "precision@10": 0.09440000000000003,
609
+ "recall@1": 0.549,
610
+ "recall@5": 0.89,
611
+ "recall@10": 0.944,
612
+ "f1@1": 0.549,
613
+ "f1@5": 0.2966666666666667,
614
+ "f1@10": 0.17163636363636361,
615
+ "map@1": 0.549,
616
+ "map@5": 0.6865999999999999,
617
+ "map@10": 0.6943325396825396,
618
+ "mrr@1": 0.549,
619
+ "mrr@5": 0.6865999999999999,
620
+ "mrr@10": 0.6943325396825396,
621
+ "num_pred": 1000,
622
+ "num_data": 1000
623
+ },
624
+ "VisualNews_t2i": {
625
+ "hit@1": 0.797,
626
+ "hit@5": 0.915,
627
+ "hit@10": 0.936,
628
+ "ndcg_linear@1": 0.797,
629
+ "ndcg_linear@5": 0.8627496822471115,
630
+ "ndcg_linear@10": 0.8696409132009587,
631
+ "ndcg_exponential@1": 0.797,
632
+ "ndcg_exponential@5": 0.8627496822471115,
633
+ "ndcg_exponential@10": 0.8696409132009587,
634
+ "precision@1": 0.797,
635
+ "precision@5": 0.18300000000000002,
636
+ "precision@10": 0.09360000000000002,
637
+ "recall@1": 0.797,
638
+ "recall@5": 0.915,
639
+ "recall@10": 0.936,
640
+ "f1@1": 0.797,
641
+ "f1@5": 0.305,
642
+ "f1@10": 0.17018181818181818,
643
+ "map@1": 0.797,
644
+ "map@5": 0.8451166666666665,
645
+ "map@10": 0.8480198412698412,
646
+ "mrr@1": 0.797,
647
+ "mrr@5": 0.8451166666666665,
648
+ "mrr@10": 0.8480198412698412,
649
+ "num_pred": 1000,
650
+ "num_data": 1000
651
+ },
652
+ "VisualNews_i2t": {
653
+ "hit@1": 0.836,
654
+ "hit@5": 0.932,
655
+ "hit@10": 0.946,
656
+ "ndcg_linear@1": 0.836,
657
+ "ndcg_linear@5": 0.8899078662440771,
658
+ "ndcg_linear@10": 0.8946317706132814,
659
+ "ndcg_exponential@1": 0.836,
660
+ "ndcg_exponential@5": 0.8899078662440771,
661
+ "ndcg_exponential@10": 0.8946317706132814,
662
+ "precision@1": 0.836,
663
+ "precision@5": 0.18640000000000004,
664
+ "precision@10": 0.0946,
665
+ "recall@1": 0.836,
666
+ "recall@5": 0.932,
667
+ "recall@10": 0.946,
668
+ "f1@1": 0.836,
669
+ "f1@5": 0.3106666666666667,
670
+ "f1@10": 0.172,
671
+ "map@1": 0.836,
672
+ "map@5": 0.8756666666666666,
673
+ "map@10": 0.8777309523809524,
674
+ "mrr@1": 0.836,
675
+ "mrr@5": 0.8756666666666666,
676
+ "mrr@10": 0.8777309523809524,
677
+ "num_pred": 1000,
678
+ "num_data": 1000
679
+ },
680
+ "MSCOCO_t2i": {
681
+ "hit@1": 0.712,
682
+ "hit@5": 0.921,
683
+ "hit@10": 0.96,
684
+ "ndcg_linear@1": 0.712,
685
+ "ndcg_linear@5": 0.8275075440748253,
686
+ "ndcg_linear@10": 0.8405579014938537,
687
+ "ndcg_exponential@1": 0.712,
688
+ "ndcg_exponential@5": 0.8275075440748253,
689
+ "ndcg_exponential@10": 0.8405579014938537,
690
+ "precision@1": 0.712,
691
+ "precision@5": 0.18420000000000003,
692
+ "precision@10": 0.09600000000000002,
693
+ "recall@1": 0.712,
694
+ "recall@5": 0.921,
695
+ "recall@10": 0.96,
696
+ "f1@1": 0.712,
697
+ "f1@5": 0.307,
698
+ "f1@10": 0.17454545454545456,
699
+ "map@1": 0.712,
700
+ "map@5": 0.7959833333333334,
701
+ "map@10": 0.8016265873015873,
702
+ "mrr@1": 0.712,
703
+ "mrr@5": 0.7959833333333334,
704
+ "mrr@10": 0.8016265873015873,
705
+ "num_pred": 1000,
706
+ "num_data": 1000
707
+ },
708
+ "MSCOCO_i2t": {
709
+ "hit@1": 0.577,
710
+ "hit@5": 0.858,
711
+ "hit@10": 0.939,
712
+ "ndcg_linear@1": 0.577,
713
+ "ndcg_linear@5": 0.7273292794403879,
714
+ "ndcg_linear@10": 0.7536734944900013,
715
+ "ndcg_exponential@1": 0.577,
716
+ "ndcg_exponential@5": 0.7273292794403879,
717
+ "ndcg_exponential@10": 0.7536734944900013,
718
+ "precision@1": 0.577,
719
+ "precision@5": 0.17160000000000003,
720
+ "precision@10": 0.09390000000000001,
721
+ "recall@1": 0.577,
722
+ "recall@5": 0.858,
723
+ "recall@10": 0.939,
724
+ "f1@1": 0.577,
725
+ "f1@5": 0.286,
726
+ "f1@10": 0.1707272727272727,
727
+ "map@1": 0.577,
728
+ "map@5": 0.6838500000000001,
729
+ "map@10": 0.6948119047619048,
730
+ "mrr@1": 0.577,
731
+ "mrr@5": 0.6838500000000001,
732
+ "mrr@10": 0.6948119047619048,
733
+ "num_pred": 1000,
734
+ "num_data": 1000
735
+ },
736
+ "NIGHTS": {
737
+ "hit@1": 0.676,
738
+ "hit@5": 0.977,
739
+ "hit@10": 0.995,
740
+ "ndcg_linear@1": 0.676,
741
+ "ndcg_linear@5": 0.8498540680608963,
742
+ "ndcg_linear@10": 0.8560163381696798,
743
+ "ndcg_exponential@1": 0.676,
744
+ "ndcg_exponential@5": 0.8498540680608963,
745
+ "ndcg_exponential@10": 0.8560163381696798,
746
+ "precision@1": 0.676,
747
+ "precision@5": 0.19540000000000005,
748
+ "precision@10": 0.09950000000000002,
749
+ "recall@1": 0.676,
750
+ "recall@5": 0.977,
751
+ "recall@10": 0.995,
752
+ "f1@1": 0.676,
753
+ "f1@5": 0.32566666666666666,
754
+ "f1@10": 0.18090909090909088,
755
+ "map@1": 0.676,
756
+ "map@5": 0.8062999999999999,
757
+ "map@10": 0.8090440476190477,
758
+ "mrr@1": 0.676,
759
+ "mrr@5": 0.8062999999999999,
760
+ "mrr@10": 0.8090440476190477,
761
+ "num_pred": 1000,
762
+ "num_data": 1000
763
+ },
764
+ "WebQA": {
765
+ "hit@1": 0.914,
766
+ "hit@5": 0.991,
767
+ "hit@10": 0.994,
768
+ "ndcg_linear@1": 0.914,
769
+ "ndcg_linear@5": 0.9583965355112005,
770
+ "ndcg_linear@10": 0.9593428375202905,
771
+ "ndcg_exponential@1": 0.914,
772
+ "ndcg_exponential@5": 0.9583965355112005,
773
+ "ndcg_exponential@10": 0.9593428375202905,
774
+ "precision@1": 0.914,
775
+ "precision@5": 0.19820000000000004,
776
+ "precision@10": 0.0994,
777
+ "recall@1": 0.914,
778
+ "recall@5": 0.991,
779
+ "recall@10": 0.994,
780
+ "f1@1": 0.914,
781
+ "f1@5": 0.3303333333333333,
782
+ "f1@10": 0.1807272727272727,
783
+ "map@1": 0.914,
784
+ "map@5": 0.9472333333333334,
785
+ "map@10": 0.9476111111111111,
786
+ "mrr@1": 0.914,
787
+ "mrr@5": 0.9472333333333334,
788
+ "mrr@10": 0.9476111111111111,
789
+ "num_pred": 1000,
790
+ "num_data": 1000
791
+ },
792
+ "FashionIQ": {
793
+ "hit@1": 0.378,
794
+ "hit@5": 0.645,
795
+ "hit@10": 0.742,
796
+ "ndcg_linear@1": 0.378,
797
+ "ndcg_linear@5": 0.518666678313658,
798
+ "ndcg_linear@10": 0.5498506170222675,
799
+ "ndcg_exponential@1": 0.378,
800
+ "ndcg_exponential@5": 0.518666678313658,
801
+ "ndcg_exponential@10": 0.5498506170222675,
802
+ "precision@1": 0.378,
803
+ "precision@5": 0.129,
804
+ "precision@10": 0.07419999999999999,
805
+ "recall@1": 0.378,
806
+ "recall@5": 0.645,
807
+ "recall@10": 0.742,
808
+ "f1@1": 0.378,
809
+ "f1@5": 0.21500000000000005,
810
+ "f1@10": 0.13490909090909092,
811
+ "map@1": 0.378,
812
+ "map@5": 0.4767,
813
+ "map@10": 0.48945396825396825,
814
+ "mrr@1": 0.378,
815
+ "mrr@5": 0.4767,
816
+ "mrr@10": 0.48945396825396825,
817
+ "num_pred": 1000,
818
+ "num_data": 1000
819
+ },
820
+ "Wiki-SS-NQ": {
821
+ "hit@1": 0.782,
822
+ "hit@5": 0.952,
823
+ "hit@10": 0.969,
824
+ "ndcg_linear@1": 0.782,
825
+ "ndcg_linear@5": 0.8778108529080692,
826
+ "ndcg_linear@10": 0.8834557157208128,
827
+ "ndcg_exponential@1": 0.782,
828
+ "ndcg_exponential@5": 0.8778108529080692,
829
+ "ndcg_exponential@10": 0.8834557157208128,
830
+ "precision@1": 0.782,
831
+ "precision@5": 0.19040000000000004,
832
+ "precision@10": 0.0969,
833
+ "recall@1": 0.782,
834
+ "recall@5": 0.952,
835
+ "recall@10": 0.969,
836
+ "f1@1": 0.782,
837
+ "f1@5": 0.31733333333333336,
838
+ "f1@10": 0.17618181818181816,
839
+ "map@1": 0.782,
840
+ "map@5": 0.85265,
841
+ "map@10": 0.8550654761904761,
842
+ "mrr@1": 0.782,
843
+ "mrr@5": 0.85265,
844
+ "mrr@10": 0.8550654761904761,
845
+ "num_pred": 1000,
846
+ "num_data": 1000
847
+ },
848
+ "OVEN": {
849
+ "hit@1": 0.751,
850
+ "hit@5": 0.908,
851
+ "hit@10": 0.94,
852
+ "ndcg_linear@1": 0.751,
853
+ "ndcg_linear@5": 0.8387290889357423,
854
+ "ndcg_linear@10": 0.8494521089299062,
855
+ "ndcg_exponential@1": 0.751,
856
+ "ndcg_exponential@5": 0.8387290889357423,
857
+ "ndcg_exponential@10": 0.8494521089299062,
858
+ "precision@1": 0.751,
859
+ "precision@5": 0.1816,
860
+ "precision@10": 0.094,
861
+ "recall@1": 0.751,
862
+ "recall@5": 0.908,
863
+ "recall@10": 0.94,
864
+ "f1@1": 0.751,
865
+ "f1@5": 0.30266666666666675,
866
+ "f1@10": 0.1709090909090909,
867
+ "map@1": 0.751,
868
+ "map@5": 0.8152999999999999,
869
+ "map@10": 0.8199472222222222,
870
+ "mrr@1": 0.751,
871
+ "mrr@5": 0.8152999999999999,
872
+ "mrr@10": 0.8199472222222222,
873
+ "num_pred": 1000,
874
+ "num_data": 1000
875
+ },
876
+ "EDIS": {
877
+ "hit@1": 0.96,
878
+ "hit@5": 0.998,
879
+ "hit@10": 0.999,
880
+ "ndcg_linear@1": 0.96,
881
+ "ndcg_linear@5": 0.9831897521142866,
882
+ "ndcg_linear@10": 0.9835052169910724,
883
+ "ndcg_exponential@1": 0.96,
884
+ "ndcg_exponential@5": 0.9831897521142866,
885
+ "ndcg_exponential@10": 0.9835052169910724,
886
+ "precision@1": 0.96,
887
+ "precision@5": 0.19960000000000003,
888
+ "precision@10": 0.0999,
889
+ "recall@1": 0.96,
890
+ "recall@5": 0.998,
891
+ "recall@10": 0.999,
892
+ "f1@1": 0.96,
893
+ "f1@5": 0.3326666666666666,
894
+ "f1@10": 0.18163636363636362,
895
+ "map@1": 0.96,
896
+ "map@5": 0.978,
897
+ "map@10": 0.978125,
898
+ "mrr@1": 0.96,
899
+ "mrr@5": 0.978,
900
+ "mrr@10": 0.978125,
901
+ "num_pred": 1000,
902
+ "num_data": 1000
903
+ },
904
+ "MSCOCO": {
905
+ "hit@1": 0.314,
906
+ "hit@5": 0.489,
907
+ "hit@10": 0.588,
908
+ "ndcg_linear@1": 0.314,
909
+ "ndcg_linear@5": 0.40410548152820475,
910
+ "ndcg_linear@10": 0.43612276758399793,
911
+ "ndcg_exponential@1": 0.314,
912
+ "ndcg_exponential@5": 0.40410548152820475,
913
+ "ndcg_exponential@10": 0.43612276758399793,
914
+ "precision@1": 0.314,
915
+ "precision@5": 0.09779999999999998,
916
+ "precision@10": 0.0588,
917
+ "recall@1": 0.314,
918
+ "recall@5": 0.489,
919
+ "recall@10": 0.588,
920
+ "f1@1": 0.314,
921
+ "f1@5": 0.16300000000000006,
922
+ "f1@10": 0.10690909090909094,
923
+ "map@1": 0.314,
924
+ "map@5": 0.37615,
925
+ "map@10": 0.38935198412698413,
926
+ "mrr@1": 0.314,
927
+ "mrr@5": 0.37615,
928
+ "mrr@10": 0.38935198412698413,
929
+ "num_pred": 1000,
930
+ "num_data": 1000
931
+ },
932
+ "RefCOCO": {
933
+ "hit@1": 0.609,
934
+ "hit@5": 0.847,
935
+ "hit@10": 0.897,
936
+ "ndcg_linear@1": 0.609,
937
+ "ndcg_linear@5": 0.7376656227905393,
938
+ "ndcg_linear@10": 0.7540339140284914,
939
+ "ndcg_exponential@1": 0.609,
940
+ "ndcg_exponential@5": 0.7376656227905393,
941
+ "ndcg_exponential@10": 0.7540339140284914,
942
+ "precision@1": 0.609,
943
+ "precision@5": 0.1694,
944
+ "precision@10": 0.08970000000000002,
945
+ "recall@1": 0.609,
946
+ "recall@5": 0.847,
947
+ "recall@10": 0.897,
948
+ "f1@1": 0.609,
949
+ "f1@5": 0.2823333333333334,
950
+ "f1@10": 0.1630909090909091,
951
+ "map@1": 0.609,
952
+ "map@5": 0.7011166666666667,
953
+ "map@10": 0.7079869047619047,
954
+ "mrr@1": 0.609,
955
+ "mrr@5": 0.7011166666666667,
956
+ "mrr@10": 0.7079869047619047,
957
+ "num_pred": 1000,
958
+ "num_data": 1000
959
+ },
960
+ "RefCOCO-Matching": {
961
+ "hit@1": 0.784,
962
+ "hit@5": 0.994,
963
+ "hit@10": 0.998,
964
+ "ndcg_linear@1": 0.784,
965
+ "ndcg_linear@5": 0.9166266927028892,
966
+ "ndcg_linear@10": 0.9178536205656439,
967
+ "ndcg_exponential@1": 0.784,
968
+ "ndcg_exponential@5": 0.9166266927028892,
969
+ "ndcg_exponential@10": 0.9178536205656439,
970
+ "precision@1": 0.784,
971
+ "precision@5": 0.1988,
972
+ "precision@10": 0.09980000000000001,
973
+ "recall@1": 0.784,
974
+ "recall@5": 0.994,
975
+ "recall@10": 0.998,
976
+ "f1@1": 0.784,
977
+ "f1@5": 0.3313333333333333,
978
+ "f1@10": 0.18145454545454545,
979
+ "map@1": 0.784,
980
+ "map@5": 0.8897833333333333,
981
+ "map@10": 0.8902511904761905,
982
+ "mrr@1": 0.784,
983
+ "mrr@5": 0.8867833333333333,
984
+ "mrr@10": 0.8872511904761905,
985
+ "num_pred": 1000,
986
+ "num_data": 1000
987
+ },
988
+ "Visual7W-Pointing": {
989
+ "hit@1": 0.665,
990
+ "hit@5": 0.851,
991
+ "hit@10": 0.899,
992
+ "ndcg_linear@1": 0.665,
993
+ "ndcg_linear@5": 0.7648497950534088,
994
+ "ndcg_linear@10": 0.7805119979199628,
995
+ "ndcg_exponential@1": 0.665,
996
+ "ndcg_exponential@5": 0.7648497950534088,
997
+ "ndcg_exponential@10": 0.7805119979199628,
998
+ "precision@1": 0.665,
999
+ "precision@5": 0.1702,
1000
+ "precision@10": 0.08990000000000001,
1001
+ "recall@1": 0.665,
1002
+ "recall@5": 0.851,
1003
+ "recall@10": 0.899,
1004
+ "f1@1": 0.665,
1005
+ "f1@5": 0.2836666666666667,
1006
+ "f1@10": 0.16345454545454544,
1007
+ "map@1": 0.665,
1008
+ "map@5": 0.7361333333333332,
1009
+ "map@10": 0.7426833333333333,
1010
+ "mrr@1": 0.665,
1011
+ "mrr@5": 0.7361333333333332,
1012
+ "mrr@10": 0.7426833333333333,
1013
+ "num_pred": 1000,
1014
+ "num_data": 1000
1015
+ }
1016
+ },
1017
+ "video": {
1018
+ "K700": {
1019
+ "hit@1": 0.397,
1020
+ "hit@5": 0.629,
1021
+ "hit@10": 0.715,
1022
+ "ndcg_linear@1": 0.397,
1023
+ "ndcg_linear@5": 0.5194137751146535,
1024
+ "ndcg_linear@10": 0.547354054247126,
1025
+ "ndcg_exponential@1": 0.397,
1026
+ "ndcg_exponential@5": 0.5194137751146535,
1027
+ "ndcg_exponential@10": 0.547354054247126,
1028
+ "precision@1": 0.397,
1029
+ "precision@5": 0.1258,
1030
+ "precision@10": 0.0715,
1031
+ "recall@1": 0.397,
1032
+ "recall@5": 0.629,
1033
+ "recall@10": 0.715,
1034
+ "f1@1": 0.397,
1035
+ "f1@5": 0.20966666666666675,
1036
+ "f1@10": 0.13,
1037
+ "map@1": 0.397,
1038
+ "map@5": 0.4830833333333333,
1039
+ "map@10": 0.4946920634920635,
1040
+ "mrr@1": 0.397,
1041
+ "mrr@5": 0.4830833333333333,
1042
+ "mrr@10": 0.4946920634920635,
1043
+ "num_pred": 1000,
1044
+ "num_data": 1000
1045
+ },
1046
+ "SmthSmthV2": {
1047
+ "hit@1": 0.306,
1048
+ "hit@5": 0.539,
1049
+ "hit@10": 0.621,
1050
+ "ndcg_linear@1": 0.306,
1051
+ "ndcg_linear@5": 0.4300547709738873,
1052
+ "ndcg_linear@10": 0.45655616986633807,
1053
+ "ndcg_exponential@1": 0.306,
1054
+ "ndcg_exponential@5": 0.4300547709738873,
1055
+ "ndcg_exponential@10": 0.45655616986633807,
1056
+ "precision@1": 0.306,
1057
+ "precision@5": 0.10779999999999998,
1058
+ "precision@10": 0.062099999999999995,
1059
+ "recall@1": 0.306,
1060
+ "recall@5": 0.539,
1061
+ "recall@10": 0.621,
1062
+ "f1@1": 0.306,
1063
+ "f1@5": 0.1796666666666667,
1064
+ "f1@10": 0.11290909090909092,
1065
+ "map@1": 0.306,
1066
+ "map@5": 0.39380000000000004,
1067
+ "map@10": 0.40473095238095236,
1068
+ "mrr@1": 0.306,
1069
+ "mrr@5": 0.39380000000000004,
1070
+ "mrr@10": 0.40473095238095236,
1071
+ "num_pred": 1000,
1072
+ "num_data": 1000
1073
+ },
1074
+ "HMDB51": {
1075
+ "hit@1": 0.479,
1076
+ "hit@5": 0.715,
1077
+ "hit@10": 0.805,
1078
+ "ndcg_linear@1": 0.479,
1079
+ "ndcg_linear@5": 0.6047040247872099,
1080
+ "ndcg_linear@10": 0.6340238736498572,
1081
+ "ndcg_exponential@1": 0.479,
1082
+ "ndcg_exponential@5": 0.6047040247872099,
1083
+ "ndcg_exponential@10": 0.6340238736498572,
1084
+ "precision@1": 0.479,
1085
+ "precision@5": 0.143,
1086
+ "precision@10": 0.0805,
1087
+ "recall@1": 0.479,
1088
+ "recall@5": 0.715,
1089
+ "recall@10": 0.805,
1090
+ "f1@1": 0.479,
1091
+ "f1@5": 0.23833333333333337,
1092
+ "f1@10": 0.14636363636363636,
1093
+ "map@1": 0.479,
1094
+ "map@5": 0.5679833333333334,
1095
+ "map@10": 0.5802178571428571,
1096
+ "mrr@1": 0.479,
1097
+ "mrr@5": 0.5679833333333334,
1098
+ "mrr@10": 0.5802178571428571,
1099
+ "num_pred": 1000,
1100
+ "num_data": 1000
1101
+ },
1102
+ "UCF101": {
1103
+ "hit@1": 0.547,
1104
+ "hit@5": 0.776,
1105
+ "hit@10": 0.848,
1106
+ "ndcg_linear@1": 0.547,
1107
+ "ndcg_linear@5": 0.671929263649622,
1108
+ "ndcg_linear@10": 0.6952345321688619,
1109
+ "ndcg_exponential@1": 0.547,
1110
+ "ndcg_exponential@5": 0.671929263649622,
1111
+ "ndcg_exponential@10": 0.6952345321688619,
1112
+ "precision@1": 0.547,
1113
+ "precision@5": 0.15519999999999998,
1114
+ "precision@10": 0.08480000000000001,
1115
+ "recall@1": 0.547,
1116
+ "recall@5": 0.776,
1117
+ "recall@10": 0.848,
1118
+ "f1@1": 0.547,
1119
+ "f1@5": 0.25866666666666677,
1120
+ "f1@10": 0.1541818181818182,
1121
+ "map@1": 0.547,
1122
+ "map@5": 0.637,
1123
+ "map@10": 0.6466361111111112,
1124
+ "mrr@1": 0.547,
1125
+ "mrr@5": 0.637,
1126
+ "mrr@10": 0.6466361111111112,
1127
+ "num_pred": 1000,
1128
+ "num_data": 1000
1129
+ },
1130
+ "Breakfast": {
1131
+ "hit@1": 0.14318706697459585,
1132
+ "hit@5": 0.5635103926096998,
1133
+ "hit@10": 1.0,
1134
+ "ndcg_linear@1": 0.14318706697459585,
1135
+ "ndcg_linear@5": 0.35021514635379236,
1136
+ "ndcg_linear@10": 0.4904891071186407,
1137
+ "ndcg_exponential@1": 0.14318706697459585,
1138
+ "ndcg_exponential@5": 0.35021514635379236,
1139
+ "ndcg_exponential@10": 0.4904891071186407,
1140
+ "precision@1": 0.14318706697459585,
1141
+ "precision@5": 0.11270207852193995,
1142
+ "precision@10": 0.1,
1143
+ "recall@1": 0.14318706697459585,
1144
+ "recall@5": 0.5635103926096998,
1145
+ "recall@10": 1.0,
1146
+ "f1@1": 0.14318706697459585,
1147
+ "f1@5": 0.18783679753656665,
1148
+ "f1@10": 0.1818181818181818,
1149
+ "map@1": 0.14318706697459585,
1150
+ "map@5": 0.2807544264819091,
1151
+ "map@10": 0.33812731405110164,
1152
+ "mrr@1": 0.14318706697459585,
1153
+ "mrr@5": 0.2807544264819091,
1154
+ "mrr@10": 0.33812731405110164,
1155
+ "num_pred": 433,
1156
+ "num_data": 433
1157
+ },
1158
+ "MVBench": {
1159
+ "hit@1": 0.46625,
1160
+ "hit@5": 1.0,
1161
+ "hit@10": 1.0,
1162
+ "ndcg_linear@1": 0.46625,
1163
+ "ndcg_linear@5": 0.7596556549662636,
1164
+ "ndcg_linear@10": 0.7596556549662636,
1165
+ "ndcg_exponential@1": 0.46625,
1166
+ "ndcg_exponential@5": 0.7596556549662636,
1167
+ "ndcg_exponential@10": 0.7596556549662636,
1168
+ "precision@1": 0.46625,
1169
+ "precision@5": 0.20000000000000004,
1170
+ "precision@10": 0.10000000000000002,
1171
+ "recall@1": 0.46625,
1172
+ "recall@5": 1.0,
1173
+ "recall@10": 1.0,
1174
+ "f1@1": 0.46625,
1175
+ "f1@5": 0.3333333333333333,
1176
+ "f1@10": 0.18181818181818182,
1177
+ "map@1": 0.46625,
1178
+ "map@5": 0.678425,
1179
+ "map@10": 0.678425,
1180
+ "mrr@1": 0.46625,
1181
+ "mrr@5": 0.678425,
1182
+ "mrr@10": 0.678425,
1183
+ "num_pred": 4000,
1184
+ "num_data": 4000
1185
+ },
1186
+ "Video-MME": {
1187
+ "hit@1": 0.39185185185185184,
1188
+ "hit@5": 1.0,
1189
+ "hit@10": 1.0,
1190
+ "ndcg_linear@1": 0.39185185185185184,
1191
+ "ndcg_linear@5": 0.7181257548926895,
1192
+ "ndcg_linear@10": 0.7181257548926895,
1193
+ "ndcg_exponential@1": 0.39185185185185184,
1194
+ "ndcg_exponential@5": 0.7181257548926895,
1195
+ "ndcg_exponential@10": 0.7181257548926895,
1196
+ "precision@1": 0.39185185185185184,
1197
+ "precision@5": 0.19999999999999996,
1198
+ "precision@10": 0.09999999999999998,
1199
+ "recall@1": 0.39185185185185184,
1200
+ "recall@5": 1.0,
1201
+ "recall@10": 1.0,
1202
+ "f1@1": 0.39185185185185184,
1203
+ "f1@5": 0.3333333333333334,
1204
+ "f1@10": 0.18181818181818188,
1205
+ "map@1": 0.39185185185185184,
1206
+ "map@5": 0.6235802469135802,
1207
+ "map@10": 0.6235802469135802,
1208
+ "mrr@1": 0.39185185185185184,
1209
+ "mrr@5": 0.6235802469135802,
1210
+ "mrr@10": 0.6235802469135802,
1211
+ "num_pred": 2700,
1212
+ "num_data": 2700
1213
+ },
1214
+ "NExTQA": {
1215
+ "hit@1": 0.5354974311069594,
1216
+ "hit@5": 1.0,
1217
+ "hit@10": 1.0,
1218
+ "ndcg_linear@1": 0.5354974311069594,
1219
+ "ndcg_linear@5": 0.77906035779538,
1220
+ "ndcg_linear@10": 0.77906035779538,
1221
+ "ndcg_exponential@1": 0.5354974311069594,
1222
+ "ndcg_exponential@5": 0.77906035779538,
1223
+ "ndcg_exponential@10": 0.77906035779538,
1224
+ "precision@1": 0.5354974311069594,
1225
+ "precision@5": 0.20000000000000004,
1226
+ "precision@10": 0.10000000000000002,
1227
+ "recall@1": 0.5354974311069594,
1228
+ "recall@5": 1.0,
1229
+ "recall@10": 1.0,
1230
+ "f1@1": 0.5354974311069594,
1231
+ "f1@5": 0.3333333333333333,
1232
+ "f1@10": 0.1818181818181818,
1233
+ "map@1": 0.5354974311069594,
1234
+ "map@5": 0.7058578545850848,
1235
+ "map@10": 0.7058578545850848,
1236
+ "mrr@1": 0.5354974311069594,
1237
+ "mrr@5": 0.7056632414759458,
1238
+ "mrr@10": 0.7056632414759458,
1239
+ "num_pred": 8564,
1240
+ "num_data": 8564
1241
+ },
1242
+ "EgoSchema": {
1243
+ "hit@1": 0.468,
1244
+ "hit@5": 1.0,
1245
+ "hit@10": 1.0,
1246
+ "ndcg_linear@1": 0.468,
1247
+ "ndcg_linear@5": 0.7452026017383512,
1248
+ "ndcg_linear@10": 0.7452026017383512,
1249
+ "ndcg_exponential@1": 0.468,
1250
+ "ndcg_exponential@5": 0.7452026017383512,
1251
+ "ndcg_exponential@10": 0.7452026017383512,
1252
+ "precision@1": 0.468,
1253
+ "precision@5": 0.2,
1254
+ "precision@10": 0.1,
1255
+ "recall@1": 0.468,
1256
+ "recall@5": 1.0,
1257
+ "recall@10": 1.0,
1258
+ "f1@1": 0.468,
1259
+ "f1@5": 0.3333333333333333,
1260
+ "f1@10": 0.18181818181818182,
1261
+ "map@1": 0.468,
1262
+ "map@5": 0.6607999999999999,
1263
+ "map@10": 0.6607999999999999,
1264
+ "mrr@1": 0.468,
1265
+ "mrr@5": 0.6607999999999999,
1266
+ "mrr@10": 0.6607999999999999,
1267
+ "num_pred": 500,
1268
+ "num_data": 500
1269
+ },
1270
+ "ActivityNetQA": {
1271
+ "hit@1": 0.656,
1272
+ "hit@5": 1.0,
1273
+ "hit@10": 1.0,
1274
+ "ndcg_linear@1": 0.656,
1275
+ "ndcg_linear@5": 0.8730398352285815,
1276
+ "ndcg_linear@10": 0.8730398352285815,
1277
+ "ndcg_exponential@1": 0.656,
1278
+ "ndcg_exponential@5": 0.8730398352285815,
1279
+ "ndcg_exponential@10": 0.8730398352285815,
1280
+ "precision@1": 0.656,
1281
+ "precision@5": 0.20000000000000004,
1282
+ "precision@10": 0.10000000000000002,
1283
+ "recall@1": 0.656,
1284
+ "recall@5": 1.0,
1285
+ "recall@10": 1.0,
1286
+ "f1@1": 0.656,
1287
+ "f1@5": 0.3333333333333333,
1288
+ "f1@10": 0.18181818181818182,
1289
+ "map@1": 0.656,
1290
+ "map@5": 0.828,
1291
+ "map@10": 0.828,
1292
+ "mrr@1": 0.656,
1293
+ "mrr@5": 0.828,
1294
+ "mrr@10": 0.828,
1295
+ "num_pred": 1000,
1296
+ "num_data": 1000
1297
+ },
1298
+ "DiDeMo": {
1299
+ "hit@1": 0.2639442231075697,
1300
+ "hit@5": 0.4800796812749004,
1301
+ "hit@10": 0.5876494023904383,
1302
+ "ndcg_linear@1": 0.2639442231075697,
1303
+ "ndcg_linear@5": 0.378795485959218,
1304
+ "ndcg_linear@10": 0.41351240118071275,
1305
+ "ndcg_exponential@1": 0.2639442231075697,
1306
+ "ndcg_exponential@5": 0.378795485959218,
1307
+ "ndcg_exponential@10": 0.41351240118071275,
1308
+ "precision@1": 0.2639442231075697,
1309
+ "precision@5": 0.09601593625498009,
1310
+ "precision@10": 0.05876494023904382,
1311
+ "recall@1": 0.2639442231075697,
1312
+ "recall@5": 0.4800796812749004,
1313
+ "recall@10": 0.5876494023904383,
1314
+ "f1@1": 0.2639442231075697,
1315
+ "f1@5": 0.16002656042496685,
1316
+ "f1@10": 0.10684534588917059,
1317
+ "map@1": 0.2639442231075697,
1318
+ "map@5": 0.34510292164674633,
1319
+ "map@10": 0.35938705495478407,
1320
+ "mrr@1": 0.2639442231075697,
1321
+ "mrr@5": 0.34510292164674633,
1322
+ "mrr@10": 0.35938705495478407,
1323
+ "num_pred": 1004,
1324
+ "num_data": 1004
1325
+ },
1326
+ "MSR-VTT": {
1327
+ "hit@1": 0.318,
1328
+ "hit@5": 0.549,
1329
+ "hit@10": 0.653,
1330
+ "ndcg_linear@1": 0.318,
1331
+ "ndcg_linear@5": 0.43999853281714363,
1332
+ "ndcg_linear@10": 0.4729767910864836,
1333
+ "ndcg_exponential@1": 0.318,
1334
+ "ndcg_exponential@5": 0.43999853281714363,
1335
+ "ndcg_exponential@10": 0.4729767910864836,
1336
+ "precision@1": 0.318,
1337
+ "precision@5": 0.10979999999999998,
1338
+ "precision@10": 0.0653,
1339
+ "recall@1": 0.318,
1340
+ "recall@5": 0.549,
1341
+ "recall@10": 0.653,
1342
+ "f1@1": 0.318,
1343
+ "f1@5": 0.18300000000000005,
1344
+ "f1@10": 0.11872727272727275,
1345
+ "map@1": 0.318,
1346
+ "map@5": 0.40383333333333327,
1347
+ "map@10": 0.4170674603174603,
1348
+ "mrr@1": 0.318,
1349
+ "mrr@5": 0.40383333333333327,
1350
+ "mrr@10": 0.4170674603174603,
1351
+ "num_pred": 1000,
1352
+ "num_data": 1000
1353
+ },
1354
+ "MSVD": {
1355
+ "hit@1": 0.49701492537313435,
1356
+ "hit@5": 0.7388059701492538,
1357
+ "hit@10": 0.8298507462686567,
1358
+ "ndcg_linear@1": 0.49701492537313435,
1359
+ "ndcg_linear@5": 0.6297230187611768,
1360
+ "ndcg_linear@10": 0.6594388806903934,
1361
+ "ndcg_exponential@1": 0.49701492537313435,
1362
+ "ndcg_exponential@5": 0.6297230187611768,
1363
+ "ndcg_exponential@10": 0.6594388806903934,
1364
+ "precision@1": 0.49701492537313435,
1365
+ "precision@5": 0.14776119402985072,
1366
+ "precision@10": 0.08298507462686566,
1367
+ "recall@1": 0.49701492537313435,
1368
+ "recall@5": 0.7388059701492538,
1369
+ "recall@10": 0.8298507462686567,
1370
+ "f1@1": 0.49701492537313435,
1371
+ "f1@5": 0.24626865671641796,
1372
+ "f1@10": 0.15088195386702855,
1373
+ "map@1": 0.49701492537313435,
1374
+ "map@5": 0.5930348258706468,
1375
+ "map@10": 0.6054554607912818,
1376
+ "mrr@1": 0.49701492537313435,
1377
+ "mrr@5": 0.5930348258706468,
1378
+ "mrr@10": 0.6054554607912818,
1379
+ "num_pred": 670,
1380
+ "num_data": 670
1381
+ },
1382
+ "VATEX": {
1383
+ "hit@1": 0.24877177311299686,
1384
+ "hit@5": 0.4723090665475659,
1385
+ "hit@10": 0.5792764627065654,
1386
+ "ndcg_linear@1": 0.24877177311299686,
1387
+ "ndcg_linear@5": 0.36560074904604023,
1388
+ "ndcg_linear@10": 0.40032465920098576,
1389
+ "ndcg_exponential@1": 0.24877177311299686,
1390
+ "ndcg_exponential@5": 0.36560074904604023,
1391
+ "ndcg_exponential@10": 0.40032465920098576,
1392
+ "precision@1": 0.24877177311299686,
1393
+ "precision@5": 0.09446181330951318,
1394
+ "precision@10": 0.05792764627065654,
1395
+ "recall@1": 0.24877177311299686,
1396
+ "recall@5": 0.4723090665475659,
1397
+ "recall@10": 0.5792764627065654,
1398
+ "f1@1": 0.24877177311299686,
1399
+ "f1@5": 0.15743635551585533,
1400
+ "f1@10": 0.10532299321937554,
1401
+ "map@1": 0.24877177311299686,
1402
+ "map@5": 0.33029626321274375,
1403
+ "map@10": 0.3447070156036212,
1404
+ "mrr@1": 0.24877177311299686,
1405
+ "mrr@5": 0.33029626321274375,
1406
+ "mrr@10": 0.3447070156036212,
1407
+ "num_pred": 4478,
1408
+ "num_data": 4478
1409
+ },
1410
+ "YouCook2": {
1411
+ "hit@1": 0.09090909090909091,
1412
+ "hit@5": 0.22805913809374018,
1413
+ "hit@10": 0.30229631959735764,
1414
+ "ndcg_linear@1": 0.09090909090909091,
1415
+ "ndcg_linear@5": 0.16160064951123587,
1416
+ "ndcg_linear@10": 0.18557172152328222,
1417
+ "ndcg_exponential@1": 0.09090909090909091,
1418
+ "ndcg_exponential@5": 0.16160064951123587,
1419
+ "ndcg_exponential@10": 0.18557172152328222,
1420
+ "precision@1": 0.09090909090909091,
1421
+ "precision@5": 0.04561182761874803,
1422
+ "precision@10": 0.030229631959735763,
1423
+ "recall@1": 0.09090909090909091,
1424
+ "recall@5": 0.22805913809374018,
1425
+ "recall@10": 0.30229631959735764,
1426
+ "f1@1": 0.09090909090909091,
1427
+ "f1@5": 0.0760197126979134,
1428
+ "f1@10": 0.05496296719951958,
1429
+ "map@1": 0.09090909090909091,
1430
+ "map@5": 0.13971898919995804,
1431
+ "map@10": 0.149584825017351,
1432
+ "mrr@1": 0.09090909090909091,
1433
+ "mrr@5": 0.13971898919995804,
1434
+ "mrr@10": 0.149584825017351,
1435
+ "num_pred": 3179,
1436
+ "num_data": 3179
1437
+ },
1438
+ "QVHighlight": {
1439
+ "hit@1": 0.5946445060018467,
1440
+ "hit@5": 0.8901200369344414,
1441
+ "hit@10": 1.0,
1442
+ "ndcg_linear@1": 0.5946445060018467,
1443
+ "ndcg_linear@5": 0.7511101894816002,
1444
+ "ndcg_linear@10": 0.7866485150531158,
1445
+ "ndcg_exponential@1": 0.5946445060018467,
1446
+ "ndcg_exponential@5": 0.7511101894816002,
1447
+ "ndcg_exponential@10": 0.7866485150531158,
1448
+ "precision@1": 0.5946445060018467,
1449
+ "precision@5": 0.17802400738688828,
1450
+ "precision@10": 0.09999999999999998,
1451
+ "recall@1": 0.5946445060018467,
1452
+ "recall@5": 0.8901200369344414,
1453
+ "recall@10": 1.0,
1454
+ "f1@1": 0.5946445060018467,
1455
+ "f1@5": 0.2967066789781472,
1456
+ "f1@10": 0.18181818181818185,
1457
+ "map@1": 0.5946445060018467,
1458
+ "map@5": 0.7049092028316406,
1459
+ "map@10": 0.7195789180553723,
1460
+ "mrr@1": 0.5946445060018467,
1461
+ "mrr@5": 0.7049092028316406,
1462
+ "mrr@10": 0.7195789180553723,
1463
+ "num_pred": 1083,
1464
+ "num_data": 1083
1465
+ },
1466
+ "Charades-STA": {
1467
+ "hit@1": 0.14030261348005502,
1468
+ "hit@5": 0.4814305364511692,
1469
+ "hit@10": 1.0,
1470
+ "ndcg_linear@1": 0.14030261348005502,
1471
+ "ndcg_linear@5": 0.30810403477419174,
1472
+ "ndcg_linear@10": 0.47145234528957136,
1473
+ "ndcg_exponential@1": 0.14030261348005502,
1474
+ "ndcg_exponential@5": 0.30810403477419174,
1475
+ "ndcg_exponential@10": 0.47145234528957136,
1476
+ "precision@1": 0.14030261348005502,
1477
+ "precision@5": 0.09628610729023383,
1478
+ "precision@10": 0.09999999999999998,
1479
+ "recall@1": 0.14030261348005502,
1480
+ "recall@5": 0.4814305364511692,
1481
+ "recall@10": 1.0,
1482
+ "f1@1": 0.14030261348005502,
1483
+ "f1@5": 0.1604768454837231,
1484
+ "f1@10": 0.18181818181818185,
1485
+ "map@1": 0.14030261348005502,
1486
+ "map@5": 0.25176524530032096,
1487
+ "map@10": 0.316674854260824,
1488
+ "mrr@1": 0.14030261348005502,
1489
+ "mrr@5": 0.25176524530032096,
1490
+ "mrr@10": 0.316674854260824,
1491
+ "num_pred": 727,
1492
+ "num_data": 727
1493
+ },
1494
+ "MomentSeeker": {
1495
+ "hit@1": 0.3738888888888889,
1496
+ "hit@5": 0.8438888888888889,
1497
+ "hit@10": 1.0,
1498
+ "ndcg_linear@1": 0.3738888888888889,
1499
+ "ndcg_linear@5": 0.6176553256419134,
1500
+ "ndcg_linear@10": 0.6753521684415025,
1501
+ "ndcg_exponential@1": 0.3738888888888889,
1502
+ "ndcg_exponential@5": 0.6176553256419134,
1503
+ "ndcg_exponential@10": 0.6753521684415025,
1504
+ "precision@1": 0.3738888888888889,
1505
+ "precision@5": 0.1856666666666667,
1506
+ "precision@10": 0.11288888888888891,
1507
+ "recall@1": 0.35006944444444443,
1508
+ "recall@5": 0.8276851851851853,
1509
+ "recall@10": 0.9984953703703704,
1510
+ "f1@1": 0.35703615520282184,
1511
+ "f1@5": 0.29815089231755904,
1512
+ "f1@10": 0.2002250064750065,
1513
+ "map@1": 0.3738888888888889,
1514
+ "map@5": 0.5426706790123457,
1515
+ "map@10": 0.5690101410934745,
1516
+ "mrr@1": 0.3738888888888889,
1517
+ "mrr@5": 0.5520092592592593,
1518
+ "mrr@10": 0.5737444885361552,
1519
+ "num_pred": 1800,
1520
+ "num_data": 1800
1521
+ }
1522
+ },
1523
+ "visdoc": {
1524
+ "ViDoRe_arxivqa": {
1525
+ "hit@1": 0.818,
1526
+ "hit@5": 0.912,
1527
+ "hit@10": 0.944,
1528
+ "ndcg_linear@1": 0.818,
1529
+ "ndcg_linear@5": 0.8685712615408394,
1530
+ "ndcg_linear@10": 0.8789118315122615,
1531
+ "ndcg_exponential@1": 0.818,
1532
+ "ndcg_exponential@5": 0.8685712615408394,
1533
+ "ndcg_exponential@10": 0.8789118315122615,
1534
+ "precision@1": 0.818,
1535
+ "precision@5": 0.1824,
1536
+ "precision@10": 0.09440000000000001,
1537
+ "recall@1": 0.818,
1538
+ "recall@5": 0.912,
1539
+ "recall@10": 0.944,
1540
+ "f1@1": 0.818,
1541
+ "f1@5": 0.304,
1542
+ "f1@10": 0.17163636363636361,
1543
+ "map@1": 0.818,
1544
+ "map@5": 0.8540666666666666,
1545
+ "map@10": 0.8583349206349207,
1546
+ "mrr@1": 0.818,
1547
+ "mrr@5": 0.8540666666666666,
1548
+ "mrr@10": 0.8583349206349207,
1549
+ "num_pred": 500,
1550
+ "num_data": 500
1551
+ },
1552
+ "ViDoRe_docvqa": {
1553
+ "hit@1": 0.4878048780487805,
1554
+ "hit@5": 0.6541019955654102,
1555
+ "hit@10": 0.7250554323725056,
1556
+ "ndcg_linear@1": 0.4878048780487805,
1557
+ "ndcg_linear@5": 0.5745938935482915,
1558
+ "ndcg_linear@10": 0.597561200968465,
1559
+ "ndcg_exponential@1": 0.4878048780487805,
1560
+ "ndcg_exponential@5": 0.5745938935482915,
1561
+ "ndcg_exponential@10": 0.597561200968465,
1562
+ "precision@1": 0.4878048780487805,
1563
+ "precision@5": 0.1334811529933481,
1564
+ "precision@10": 0.0745011086474501,
1565
+ "recall@1": 0.4866962305986696,
1566
+ "recall@5": 0.6472283813747228,
1567
+ "recall@10": 0.7167775314116778,
1568
+ "f1@1": 0.48706577974870663,
1569
+ "f1@5": 0.2198328851126556,
1570
+ "f1@10": 0.13419485171148143,
1571
+ "map@1": 0.4878048780487805,
1572
+ "map@5": 0.5490422517861542,
1573
+ "map@10": 0.5587257593355155,
1574
+ "mrr@1": 0.4878048780487805,
1575
+ "mrr@5": 0.5516629711751663,
1576
+ "mrr@10": 0.5612703691971984,
1577
+ "num_pred": 451,
1578
+ "num_data": 451
1579
+ },
1580
+ "ViDoRe_infovqa": {
1581
+ "hit@1": 0.8765182186234818,
1582
+ "hit@5": 0.951417004048583,
1583
+ "hit@10": 0.9696356275303644,
1584
+ "ndcg_linear@1": 0.8765182186234818,
1585
+ "ndcg_linear@5": 0.9159739102039204,
1586
+ "ndcg_linear@10": 0.9221606391623131,
1587
+ "ndcg_exponential@1": 0.8765182186234818,
1588
+ "ndcg_exponential@5": 0.9159739102039204,
1589
+ "ndcg_exponential@10": 0.9221606391623131,
1590
+ "precision@1": 0.8765182186234818,
1591
+ "precision@5": 0.1902834008097166,
1592
+ "precision@10": 0.09716599190283401,
1593
+ "recall@1": 0.8748987854251012,
1594
+ "recall@5": 0.9487854251012146,
1595
+ "recall@10": 0.9680161943319838,
1596
+ "f1@1": 0.8751686909581647,
1597
+ "f1@5": 0.3167727009832273,
1598
+ "f1@10": 0.17650595019016072,
1599
+ "map@1": 0.8765182186234818,
1600
+ "map@5": 0.9044197031039136,
1601
+ "map@10": 0.9070215603110341,
1602
+ "mrr@1": 0.8765182186234818,
1603
+ "mrr@5": 0.9062415654520918,
1604
+ "mrr@10": 0.9085542381595014,
1605
+ "num_pred": 494,
1606
+ "num_data": 494
1607
+ },
1608
+ "ViDoRe_tabfquad": {
1609
+ "hit@1": 0.9071428571428571,
1610
+ "hit@5": 0.9785714285714285,
1611
+ "hit@10": 0.9857142857142858,
1612
+ "ndcg_linear@1": 0.9071428571428571,
1613
+ "ndcg_linear@5": 0.9464498663499118,
1614
+ "ndcg_linear@10": 0.9489125110657739,
1615
+ "ndcg_exponential@1": 0.9071428571428571,
1616
+ "ndcg_exponential@5": 0.9464498663499118,
1617
+ "ndcg_exponential@10": 0.9489125110657739,
1618
+ "precision@1": 0.9071428571428571,
1619
+ "precision@5": 0.1957142857142857,
1620
+ "precision@10": 0.09857142857142856,
1621
+ "recall@1": 0.9071428571428571,
1622
+ "recall@5": 0.9785714285714285,
1623
+ "recall@10": 0.9857142857142858,
1624
+ "f1@1": 0.9071428571428571,
1625
+ "f1@5": 0.32619047619047625,
1626
+ "f1@10": 0.17922077922077925,
1627
+ "map@1": 0.9071428571428571,
1628
+ "map@5": 0.9356547619047619,
1629
+ "map@10": 0.9367602040816324,
1630
+ "mrr@1": 0.9071428571428571,
1631
+ "mrr@5": 0.9356547619047619,
1632
+ "mrr@10": 0.9367602040816324,
1633
+ "num_pred": 280,
1634
+ "num_data": 280
1635
+ },
1636
+ "ViDoRe_tatdqa": {
1637
+ "hit@1": 0.6221142162818954,
1638
+ "hit@5": 0.8444714459295262,
1639
+ "hit@10": 0.9106925880923451,
1640
+ "ndcg_linear@1": 0.6221142162818954,
1641
+ "ndcg_linear@5": 0.740984681776679,
1642
+ "ndcg_linear@10": 0.7629989160213186,
1643
+ "ndcg_exponential@1": 0.6221142162818954,
1644
+ "ndcg_exponential@5": 0.740984681776679,
1645
+ "ndcg_exponential@10": 0.7629989160213186,
1646
+ "precision@1": 0.6221142162818954,
1647
+ "precision@5": 0.16925880923450792,
1648
+ "precision@10": 0.09143377885783718,
1649
+ "recall@1": 0.6215066828675577,
1650
+ "recall@5": 0.8435601458080194,
1651
+ "recall@10": 0.9106925880923451,
1652
+ "f1@1": 0.6217091940056704,
1653
+ "f1@5": 0.28183764392755895,
1654
+ "f1@10": 0.16613277366618803,
1655
+ "map@1": 0.6221142162818954,
1656
+ "map@5": 0.7066322397731876,
1657
+ "map@10": 0.7159557079210785,
1658
+ "mrr@1": 0.6221142162818954,
1659
+ "mrr@5": 0.706946132037262,
1660
+ "mrr@10": 0.71601405041563,
1661
+ "num_pred": 1646,
1662
+ "num_data": 1646
1663
+ },
1664
+ "ViDoRe_shiftproject": {
1665
+ "hit@1": 0.95,
1666
+ "hit@5": 0.98,
1667
+ "hit@10": 0.99,
1668
+ "ndcg_linear@1": 0.95,
1669
+ "ndcg_linear@5": 0.9676185950714291,
1670
+ "ndcg_linear@10": 0.9711806669425094,
1671
+ "ndcg_exponential@1": 0.95,
1672
+ "ndcg_exponential@5": 0.9676185950714291,
1673
+ "ndcg_exponential@10": 0.9711806669425094,
1674
+ "precision@1": 0.95,
1675
+ "precision@5": 0.19599999999999995,
1676
+ "precision@10": 0.09899999999999999,
1677
+ "recall@1": 0.95,
1678
+ "recall@5": 0.98,
1679
+ "recall@10": 0.99,
1680
+ "f1@1": 0.95,
1681
+ "f1@5": 0.32666666666666677,
1682
+ "f1@10": 0.18000000000000008,
1683
+ "map@1": 0.95,
1684
+ "map@5": 0.9633333333333333,
1685
+ "map@10": 0.965,
1686
+ "mrr@1": 0.95,
1687
+ "mrr@5": 0.9633333333333333,
1688
+ "mrr@10": 0.965,
1689
+ "num_pred": 100,
1690
+ "num_data": 100
1691
+ },
1692
+ "ViDoRe_syntheticDocQA_artificial_intelligence": {
1693
+ "hit@1": 0.99,
1694
+ "hit@5": 1.0,
1695
+ "hit@10": 1.0,
1696
+ "ndcg_linear@1": 0.99,
1697
+ "ndcg_linear@5": 0.9963092975357145,
1698
+ "ndcg_linear@10": 0.9963092975357145,
1699
+ "ndcg_exponential@1": 0.99,
1700
+ "ndcg_exponential@5": 0.9963092975357145,
1701
+ "ndcg_exponential@10": 0.9963092975357145,
1702
+ "precision@1": 0.99,
1703
+ "precision@5": 0.19999999999999996,
1704
+ "precision@10": 0.09999999999999998,
1705
+ "recall@1": 0.99,
1706
+ "recall@5": 1.0,
1707
+ "recall@10": 1.0,
1708
+ "f1@1": 0.99,
1709
+ "f1@5": 0.3333333333333335,
1710
+ "f1@10": 0.1818181818181819,
1711
+ "map@1": 0.99,
1712
+ "map@5": 0.995,
1713
+ "map@10": 0.995,
1714
+ "mrr@1": 0.99,
1715
+ "mrr@5": 0.995,
1716
+ "mrr@10": 0.995,
1717
+ "num_pred": 100,
1718
+ "num_data": 100
1719
+ },
1720
+ "ViDoRe_syntheticDocQA_energy": {
1721
+ "hit@1": 0.93,
1722
+ "hit@5": 0.97,
1723
+ "hit@10": 0.98,
1724
+ "ndcg_linear@1": 0.93,
1725
+ "ndcg_linear@5": 0.9532346581878777,
1726
+ "ndcg_linear@10": 0.9562449581445175,
1727
+ "ndcg_exponential@1": 0.93,
1728
+ "ndcg_exponential@5": 0.9532346581878777,
1729
+ "ndcg_exponential@10": 0.9562449581445175,
1730
+ "precision@1": 0.93,
1731
+ "precision@5": 0.19399999999999995,
1732
+ "precision@10": 0.09799999999999998,
1733
+ "recall@1": 0.93,
1734
+ "recall@5": 0.97,
1735
+ "recall@10": 0.98,
1736
+ "f1@1": 0.93,
1737
+ "f1@5": 0.3233333333333334,
1738
+ "f1@10": 0.17818181818181827,
1739
+ "map@1": 0.93,
1740
+ "map@5": 0.9475,
1741
+ "map@10": 0.9486111111111112,
1742
+ "mrr@1": 0.93,
1743
+ "mrr@5": 0.9475,
1744
+ "mrr@10": 0.9486111111111112,
1745
+ "num_pred": 100,
1746
+ "num_data": 100
1747
+ },
1748
+ "ViDoRe_syntheticDocQA_government_reports": {
1749
+ "hit@1": 0.97,
1750
+ "hit@5": 1.0,
1751
+ "hit@10": 1.0,
1752
+ "ndcg_linear@1": 0.97,
1753
+ "ndcg_linear@5": 0.9876185950714291,
1754
+ "ndcg_linear@10": 0.9876185950714291,
1755
+ "ndcg_exponential@1": 0.97,
1756
+ "ndcg_exponential@5": 0.9876185950714291,
1757
+ "ndcg_exponential@10": 0.9876185950714291,
1758
+ "precision@1": 0.97,
1759
+ "precision@5": 0.19999999999999996,
1760
+ "precision@10": 0.09999999999999998,
1761
+ "recall@1": 0.97,
1762
+ "recall@5": 1.0,
1763
+ "recall@10": 1.0,
1764
+ "f1@1": 0.97,
1765
+ "f1@5": 0.3333333333333335,
1766
+ "f1@10": 0.1818181818181819,
1767
+ "map@1": 0.97,
1768
+ "map@5": 0.9833333333333333,
1769
+ "map@10": 0.9833333333333333,
1770
+ "mrr@1": 0.97,
1771
+ "mrr@5": 0.9833333333333333,
1772
+ "mrr@10": 0.9833333333333333,
1773
+ "num_pred": 100,
1774
+ "num_data": 100
1775
+ },
1776
+ "ViDoRe_syntheticDocQA_healthcare_industry": {
1777
+ "hit@1": 0.98,
1778
+ "hit@5": 1.0,
1779
+ "hit@10": 1.0,
1780
+ "ndcg_linear@1": 0.98,
1781
+ "ndcg_linear@5": 0.9926185950714291,
1782
+ "ndcg_linear@10": 0.9926185950714291,
1783
+ "ndcg_exponential@1": 0.98,
1784
+ "ndcg_exponential@5": 0.9926185950714291,
1785
+ "ndcg_exponential@10": 0.9926185950714291,
1786
+ "precision@1": 0.98,
1787
+ "precision@5": 0.19999999999999996,
1788
+ "precision@10": 0.09999999999999998,
1789
+ "recall@1": 0.98,
1790
+ "recall@5": 1.0,
1791
+ "recall@10": 1.0,
1792
+ "f1@1": 0.98,
1793
+ "f1@5": 0.3333333333333335,
1794
+ "f1@10": 0.1818181818181819,
1795
+ "map@1": 0.98,
1796
+ "map@5": 0.99,
1797
+ "map@10": 0.99,
1798
+ "mrr@1": 0.98,
1799
+ "mrr@5": 0.99,
1800
+ "mrr@10": 0.99,
1801
+ "num_pred": 100,
1802
+ "num_data": 100
1803
+ },
1804
+ "ViDoRe_esg_reports_human_labeled_v2": {
1805
+ "hit@1": 0.5961538461538461,
1806
+ "hit@5": 0.7884615384615384,
1807
+ "hit@10": 0.8846153846153846,
1808
+ "ndcg_linear@1": 0.5737179487179488,
1809
+ "ndcg_linear@5": 0.6336537052325203,
1810
+ "ndcg_linear@10": 0.6529793585477737,
1811
+ "ndcg_exponential@1": 0.5668498168498168,
1812
+ "ndcg_exponential@5": 0.6338504365299624,
1813
+ "ndcg_exponential@10": 0.6525583236140545,
1814
+ "precision@1": 0.5961538461538461,
1815
+ "precision@5": 0.23076923076923078,
1816
+ "precision@10": 0.13653846153846153,
1817
+ "recall@1": 0.42499999999999993,
1818
+ "recall@5": 0.6831959706959707,
1819
+ "recall@10": 0.7332042957042956,
1820
+ "f1@1": 0.4619329388560158,
1821
+ "f1@5": 0.31360967463908646,
1822
+ "f1@10": 0.21265489789245448,
1823
+ "map@1": 0.5961538461538461,
1824
+ "map@5": 0.5806089743589743,
1825
+ "map@10": 0.5874552299552299,
1826
+ "mrr@1": 0.5961538461538461,
1827
+ "mrr@5": 0.6778846153846154,
1828
+ "mrr@10": 0.6906669719169719,
1829
+ "num_pred": 52,
1830
+ "num_data": 52
1831
+ },
1832
+ "ViDoRe_biomedical_lectures_v2_multilingual": {
1833
+ "hit@1": 0.4546875,
1834
+ "hit@5": 0.728125,
1835
+ "hit@10": 0.815625,
1836
+ "ndcg_linear@1": 0.4546875,
1837
+ "ndcg_linear@5": 0.4948824579441539,
1838
+ "ndcg_linear@10": 0.5296750546342992,
1839
+ "ndcg_exponential@1": 0.4546875,
1840
+ "ndcg_exponential@5": 0.4948824579441539,
1841
+ "ndcg_exponential@10": 0.5296750546342992,
1842
+ "precision@1": 0.4546875,
1843
+ "precision@5": 0.23562500000000003,
1844
+ "precision@10": 0.15390625,
1845
+ "recall@1": 0.26338541666666665,
1846
+ "recall@5": 0.523475905987395,
1847
+ "recall@10": 0.6315834243670039,
1848
+ "f1@1": 0.3050258161976912,
1849
+ "f1@5": 0.28532247873654126,
1850
+ "f1@10": 0.22208065327918272,
1851
+ "map@1": 0.4546875,
1852
+ "map@5": 0.4263871527777778,
1853
+ "map@10": 0.44231501042257176,
1854
+ "mrr@1": 0.4546875,
1855
+ "mrr@5": 0.56375,
1856
+ "mrr@10": 0.5754284474206349,
1857
+ "num_pred": 640,
1858
+ "num_data": 640
1859
+ },
1860
+ "ViDoRe_economics_reports_v2_multilingual": {
1861
+ "hit@1": 0.6206896551724138,
1862
+ "hit@5": 0.896551724137931,
1863
+ "hit@10": 0.9353448275862069,
1864
+ "ndcg_linear@1": 0.6206896551724138,
1865
+ "ndcg_linear@5": 0.5421098675106539,
1866
+ "ndcg_linear@10": 0.5287811820133248,
1867
+ "ndcg_exponential@1": 0.6206896551724138,
1868
+ "ndcg_exponential@5": 0.5421098675106539,
1869
+ "ndcg_exponential@10": 0.5287811820133248,
1870
+ "precision@1": 0.6206896551724138,
1871
+ "precision@5": 0.45431034482758625,
1872
+ "precision@10": 0.3745689655172414,
1873
+ "recall@1": 0.10691249388459685,
1874
+ "recall@5": 0.2925943528645089,
1875
+ "recall@10": 0.41857708558180107,
1876
+ "f1@1": 0.14883132087852466,
1877
+ "f1@5": 0.2819557978845695,
1878
+ "f1@10": 0.3211854297471819,
1879
+ "map@1": 0.6206896551724138,
1880
+ "map@5": 0.42993893678160916,
1881
+ "map@10": 0.38539058495364936,
1882
+ "mrr@1": 0.6206896551724138,
1883
+ "mrr@5": 0.7347701149425288,
1884
+ "mrr@10": 0.7403188286808976,
1885
+ "num_pred": 232,
1886
+ "num_data": 232
1887
+ },
1888
+ "ViDoRe_esg_reports_v2_multilingual": {
1889
+ "hit@1": 0.4956140350877193,
1890
+ "hit@5": 0.8201754385964912,
1891
+ "hit@10": 0.9078947368421053,
1892
+ "ndcg_linear@1": 0.4956140350877193,
1893
+ "ndcg_linear@5": 0.553791630514325,
1894
+ "ndcg_linear@10": 0.5983952525358399,
1895
+ "ndcg_exponential@1": 0.4956140350877193,
1896
+ "ndcg_exponential@5": 0.553791630514325,
1897
+ "ndcg_exponential@10": 0.5983952525358399,
1898
+ "precision@1": 0.4956140350877193,
1899
+ "precision@5": 0.2789473684210526,
1900
+ "precision@10": 0.18859649122807023,
1901
+ "recall@1": 0.2702119883040936,
1902
+ "recall@5": 0.5824352548036759,
1903
+ "recall@10": 0.7194914369256475,
1904
+ "f1@1": 0.31508798219324535,
1905
+ "f1@5": 0.3248121100141491,
1906
+ "f1@10": 0.2578734690556428,
1907
+ "map@1": 0.4956140350877193,
1908
+ "map@5": 0.4682602339181286,
1909
+ "map@10": 0.48884156281908475,
1910
+ "mrr@1": 0.4956140350877193,
1911
+ "mrr@5": 0.628874269005848,
1912
+ "mrr@10": 0.6414212614870508,
1913
+ "num_pred": 228,
1914
+ "num_data": 228
1915
+ },
1916
+ "VisRAG_ArxivQA": {
1917
+ "hit@1": 0.8259803921568627,
1918
+ "hit@5": 0.9166666666666666,
1919
+ "hit@10": 0.9448529411764706,
1920
+ "ndcg_linear@1": 0.8259803921568627,
1921
+ "ndcg_linear@5": 0.8734973163142782,
1922
+ "ndcg_linear@10": 0.8828532837961396,
1923
+ "ndcg_exponential@1": 0.8259803921568627,
1924
+ "ndcg_exponential@5": 0.8734973163142782,
1925
+ "ndcg_exponential@10": 0.8828532837961396,
1926
+ "precision@1": 0.8259803921568627,
1927
+ "precision@5": 0.18333333333333332,
1928
+ "precision@10": 0.09448529411764706,
1929
+ "recall@1": 0.8259803921568627,
1930
+ "recall@5": 0.9166666666666666,
1931
+ "recall@10": 0.9448529411764706,
1932
+ "f1@1": 0.8259803921568627,
1933
+ "f1@5": 0.30555555555555564,
1934
+ "f1@10": 0.17179144385026737,
1935
+ "map@1": 0.8259803921568627,
1936
+ "map@5": 0.8592320261437908,
1937
+ "map@10": 0.86323140367258,
1938
+ "mrr@1": 0.8259803921568627,
1939
+ "mrr@5": 0.8592320261437908,
1940
+ "mrr@10": 0.86323140367258,
1941
+ "num_pred": 816,
1942
+ "num_data": 816
1943
+ },
1944
+ "VisRAG_ChartQA": {
1945
+ "hit@1": 0.7619047619047619,
1946
+ "hit@5": 0.873015873015873,
1947
+ "hit@10": 0.9206349206349206,
1948
+ "ndcg_linear@1": 0.7619047619047619,
1949
+ "ndcg_linear@5": 0.8190027211628073,
1950
+ "ndcg_linear@10": 0.8349551877854593,
1951
+ "ndcg_exponential@1": 0.7619047619047619,
1952
+ "ndcg_exponential@5": 0.8190027211628073,
1953
+ "ndcg_exponential@10": 0.8349551877854593,
1954
+ "precision@1": 0.7619047619047619,
1955
+ "precision@5": 0.1746031746031745,
1956
+ "precision@10": 0.09206349206349201,
1957
+ "recall@1": 0.7619047619047619,
1958
+ "recall@5": 0.873015873015873,
1959
+ "recall@10": 0.9206349206349206,
1960
+ "f1@1": 0.7619047619047619,
1961
+ "f1@5": 0.29100529100529093,
1962
+ "f1@10": 0.16738816738816742,
1963
+ "map@1": 0.7619047619047619,
1964
+ "map@5": 0.8013227513227513,
1965
+ "map@10": 0.8082199546485261,
1966
+ "mrr@1": 0.7619047619047619,
1967
+ "mrr@5": 0.8013227513227513,
1968
+ "mrr@10": 0.8082199546485261,
1969
+ "num_pred": 63,
1970
+ "num_data": 63
1971
+ },
1972
+ "VisRAG_MP-DocVQA": {
1973
+ "hit@1": 0.817258883248731,
1974
+ "hit@5": 0.9475465313028765,
1975
+ "hit@10": 0.961082910321489,
1976
+ "ndcg_linear@1": 0.817258883248731,
1977
+ "ndcg_linear@5": 0.8921574691348291,
1978
+ "ndcg_linear@10": 0.8964004804547877,
1979
+ "ndcg_exponential@1": 0.817258883248731,
1980
+ "ndcg_exponential@5": 0.8921574691348291,
1981
+ "ndcg_exponential@10": 0.8964004804547877,
1982
+ "precision@1": 0.817258883248731,
1983
+ "precision@5": 0.1895093062605753,
1984
+ "precision@10": 0.0961082910321489,
1985
+ "recall@1": 0.817258883248731,
1986
+ "recall@5": 0.9475465313028765,
1987
+ "recall@10": 0.961082910321489,
1988
+ "f1@1": 0.817258883248731,
1989
+ "f1@5": 0.3158488437676255,
1990
+ "f1@10": 0.17474234733117985,
1991
+ "map@1": 0.817258883248731,
1992
+ "map@5": 0.873237450648618,
1993
+ "map@10": 0.8749073402626703,
1994
+ "mrr@1": 0.817258883248731,
1995
+ "mrr@5": 0.873237450648618,
1996
+ "mrr@10": 0.8749073402626703,
1997
+ "num_pred": 591,
1998
+ "num_data": 591
1999
+ },
2000
+ "VisRAG_SlideVQA": {
2001
+ "hit@1": 0.9244604316546763,
2002
+ "hit@5": 0.9766187050359713,
2003
+ "hit@10": 0.9820143884892086,
2004
+ "ndcg_linear@1": 0.9244604316546763,
2005
+ "ndcg_linear@5": 0.9453762073854034,
2006
+ "ndcg_linear@10": 0.9484324956995065,
2007
+ "ndcg_exponential@1": 0.9244604316546763,
2008
+ "ndcg_exponential@5": 0.9453762073854034,
2009
+ "ndcg_exponential@10": 0.9484324956995065,
2010
+ "precision@1": 0.9244604316546763,
2011
+ "precision@5": 0.24388489208633093,
2012
+ "precision@10": 0.12320143884892086,
2013
+ "recall@1": 0.8003597122302158,
2014
+ "recall@5": 0.9685251798561151,
2015
+ "recall@10": 0.9766187050359713,
2016
+ "f1@1": 0.841726618705036,
2017
+ "f1@5": 0.3825796505652622,
2018
+ "f1@10": 0.21626335295400048,
2019
+ "map@1": 0.9244604316546763,
2020
+ "map@5": 0.9316247002398079,
2021
+ "map@10": 0.9332980044535798,
2022
+ "mrr@1": 0.9244604316546763,
2023
+ "mrr@5": 0.9481414868105514,
2024
+ "mrr@10": 0.9489658273381295,
2025
+ "num_pred": 556,
2026
+ "num_data": 556
2027
+ },
2028
+ "VisRAG_InfoVQA": {
2029
+ "hit@1": 0.8704735376044568,
2030
+ "hit@5": 0.9832869080779945,
2031
+ "hit@10": 0.9930362116991643,
2032
+ "ndcg_linear@1": 0.8704735376044568,
2033
+ "ndcg_linear@5": 0.9352387521932299,
2034
+ "ndcg_linear@10": 0.9383224079083196,
2035
+ "ndcg_exponential@1": 0.8704735376044568,
2036
+ "ndcg_exponential@5": 0.9352387521932299,
2037
+ "ndcg_exponential@10": 0.9383224079083196,
2038
+ "precision@1": 0.8704735376044568,
2039
+ "precision@5": 0.19665738161559887,
2040
+ "precision@10": 0.09930362116991641,
2041
+ "recall@1": 0.8704735376044568,
2042
+ "recall@5": 0.9832869080779945,
2043
+ "recall@10": 0.9930362116991643,
2044
+ "f1@1": 0.8704735376044568,
2045
+ "f1@5": 0.3277623026926649,
2046
+ "f1@10": 0.1805520384907572,
2047
+ "map@1": 0.8704735376044568,
2048
+ "map@5": 0.9188254410399258,
2049
+ "map@10": 0.9200573683512402,
2050
+ "mrr@1": 0.8704735376044568,
2051
+ "mrr@5": 0.9188254410399258,
2052
+ "mrr@10": 0.9200573683512402,
2053
+ "num_pred": 718,
2054
+ "num_data": 718
2055
+ },
2056
+ "VisRAG_PlotQA": {
2057
+ "hit@1": 0.49942062572421786,
2058
+ "hit@5": 0.7520278099652375,
2059
+ "hit@10": 0.8389339513325609,
2060
+ "ndcg_linear@1": 0.49942062572421786,
2061
+ "ndcg_linear@5": 0.6341927528705769,
2062
+ "ndcg_linear@10": 0.6624618705964804,
2063
+ "ndcg_exponential@1": 0.49942062572421786,
2064
+ "ndcg_exponential@5": 0.6341927528705769,
2065
+ "ndcg_exponential@10": 0.6624618705964804,
2066
+ "precision@1": 0.49942062572421786,
2067
+ "precision@5": 0.1504055619930475,
2068
+ "precision@10": 0.08389339513325607,
2069
+ "recall@1": 0.49942062572421786,
2070
+ "recall@5": 0.7520278099652375,
2071
+ "recall@10": 0.8389339513325609,
2072
+ "f1@1": 0.49942062572421786,
2073
+ "f1@5": 0.2506759366550792,
2074
+ "f1@10": 0.1525334456968293,
2075
+ "map@1": 0.49942062572421786,
2076
+ "map@5": 0.5949787562765547,
2077
+ "map@10": 0.6067474112085932,
2078
+ "mrr@1": 0.49942062572421786,
2079
+ "mrr@5": 0.5949787562765547,
2080
+ "mrr@10": 0.6067474112085932,
2081
+ "num_pred": 863,
2082
+ "num_data": 863
2083
+ },
2084
+ "ViDoSeek-page": {
2085
+ "hit@1": 0.02714535901926445,
2086
+ "hit@5": 0.4290718038528897,
2087
+ "hit@10": 0.6532399299474606,
2088
+ "ndcg_linear@1": 0.02714535901926445,
2089
+ "ndcg_linear@5": 0.23240353892048016,
2090
+ "ndcg_linear@10": 0.305124886024534,
2091
+ "ndcg_exponential@1": 0.02714535901926445,
2092
+ "ndcg_exponential@5": 0.23240353892048016,
2093
+ "ndcg_exponential@10": 0.305124886024534,
2094
+ "precision@1": 0.02714535901926445,
2095
+ "precision@5": 0.08581436077057793,
2096
+ "precision@10": 0.06532399299474606,
2097
+ "recall@1": 0.02714535901926445,
2098
+ "recall@5": 0.4290718038528897,
2099
+ "recall@10": 0.6532399299474606,
2100
+ "f1@1": 0.02714535901926445,
2101
+ "f1@5": 0.1430239346176299,
2102
+ "f1@10": 0.11877089635408376,
2103
+ "map@1": 0.02714535901926445,
2104
+ "map@5": 0.16780502043199066,
2105
+ "map@10": 0.19794637644900342,
2106
+ "mrr@1": 0.02714535901926445,
2107
+ "mrr@5": 0.16780502043199066,
2108
+ "mrr@10": 0.19794637644900342,
2109
+ "num_pred": 1142,
2110
+ "num_data": 1142
2111
+ },
2112
+ "ViDoSeek-doc": {
2113
+ "hit@1": 0.9964973730297724,
2114
+ "hit@5": 0.9982486865148862,
2115
+ "hit@10": 0.999124343257443,
2116
+ "ndcg_linear@1": 0.6800934033858728,
2117
+ "ndcg_linear@5": 0.8387827817558765,
2118
+ "ndcg_linear@10": 0.8278623787973661,
2119
+ "ndcg_exponential@1": 0.4540905679259443,
2120
+ "ndcg_exponential@5": 0.7179848879430044,
2121
+ "ndcg_exponential@10": 0.7517965160193582,
2122
+ "precision@1": 0.9964973730297724,
2123
+ "precision@5": 0.9288966725043785,
2124
+ "precision@10": 0.8447460595446585,
2125
+ "recall@1": 0.058761907353168774,
2126
+ "recall@5": 0.2735768446029065,
2127
+ "recall@10": 0.48843735028165003,
2128
+ "f1@1": 0.11022913197970112,
2129
+ "f1@5": 0.41401147163111984,
2130
+ "f1@10": 0.6049486379407474,
2131
+ "map@1": 0.9964973730297724,
2132
+ "map@5": 0.9212288382953882,
2133
+ "map@10": 0.8346283571805838,
2134
+ "mrr@1": 0.9964973730297724,
2135
+ "mrr@5": 0.9973730297723292,
2136
+ "mrr@10": 0.9975189725627552,
2137
+ "num_pred": 1142,
2138
+ "num_data": 1142
2139
+ },
2140
+ "MMLongBench-page": {
2141
+ "hit@1": 0.06921241050119331,
2142
+ "hit@5": 0.32816229116945106,
2143
+ "hit@10": 0.43556085918854415,
2144
+ "ndcg_linear@1": 0.06921241050119331,
2145
+ "ndcg_linear@5": 0.16193698962444822,
2146
+ "ndcg_linear@10": 0.20158281959554863,
2147
+ "ndcg_exponential@1": 0.06921241050119331,
2148
+ "ndcg_exponential@5": 0.16193698962444822,
2149
+ "ndcg_exponential@10": 0.20158281959554863,
2150
+ "precision@1": 0.06921241050119331,
2151
+ "precision@5": 0.07756563245823389,
2152
+ "precision@10": 0.058233890214797135,
2153
+ "recall@1": 0.040324184566428,
2154
+ "recall@5": 0.23706102966246162,
2155
+ "recall@10": 0.34820150017047385,
2156
+ "f1@1": 0.048800854409446294,
2157
+ "f1@5": 0.1101645476522635,
2158
+ "f1@10": 0.09465332114769648,
2159
+ "map@1": 0.06921241050119331,
2160
+ "map@5": 0.11841553964465659,
2161
+ "map@10": 0.13681524054861935,
2162
+ "mrr@1": 0.06921241050119331,
2163
+ "mrr@5": 0.16302704852824182,
2164
+ "mrr@10": 0.17715081259234006,
2165
+ "num_pred": 838,
2166
+ "num_data": 838
2167
+ },
2168
+ "MMLongBench-doc": {
2169
+ "hit@1": 0.7326968973747017,
2170
+ "hit@5": 0.8126491646778043,
2171
+ "hit@10": 0.837708830548926,
2172
+ "ndcg_linear@1": 0.5123309466984886,
2173
+ "ndcg_linear@5": 0.5428858706725166,
2174
+ "ndcg_linear@10": 0.5275085648914615,
2175
+ "ndcg_exponential@1": 0.354926696215479,
2176
+ "ndcg_exponential@5": 0.45441895462567394,
2177
+ "ndcg_exponential@10": 0.4695562288112078,
2178
+ "precision@1": 0.7326968973747017,
2179
+ "precision@5": 0.6021479713603819,
2180
+ "precision@10": 0.540453460620525,
2181
+ "recall@1": 0.023809666923602684,
2182
+ "recall@5": 0.09513770171338518,
2183
+ "recall@10": 0.16711122344532417,
2184
+ "f1@1": 0.04565355580911593,
2185
+ "f1@5": 0.15776265072387466,
2186
+ "f1@10": 0.2394225821542852,
2187
+ "map@1": 0.7326968973747017,
2188
+ "map@5": 0.5741805887032617,
2189
+ "map@10": 0.4961500044196942,
2190
+ "mrr@1": 0.7326968973747017,
2191
+ "mrr@5": 0.7620525059665871,
2192
+ "mrr@10": 0.7654510929272267,
2193
+ "num_pred": 838,
2194
+ "num_data": 838
2195
+ }
2196
+ }
2197
+ }
2198
+ }
utils.py CHANGED
@@ -35,7 +35,8 @@ This comprehensive suite enables robust evaluation of multimodal embedding model
35
  | [**📈Overview**](https://tiger-ai-lab.github.io/VLM2Vec/) | [**Github**](https://github.com/TIGER-AI-Lab/VLM2Vec)
36
  | [**📖MMEB-V2/VLM2Vec-V2 Paper (TBA)**](https://arxiv.org/abs/2410.05160)
37
  | [**📖MMEB-V1/VLM2Vec-V1 Paper**](https://arxiv.org/abs/2410.05160)
38
- | [**🤗Hugging Face**](https://huggingface.co/datasets/TIGER-Lab/MMEB-V2) |
 
39
  """
40
 
41
  TABLE_INTRODUCTION = """***Important Notes: ***
@@ -55,11 +56,9 @@ CITATION_BUTTON_TEXT = r"""@article{jiang2024vlm2vec,
55
 
56
  SUBMIT_INTRODUCTION = """# Submit on MMEB Leaderboard Introduction
57
 
58
- ## ⚠ Please note that you need to submit the JSON file with the following format:
59
 
60
- ### ***Important Notes: We have released MMEB-V2 and will deprecate MMEB-V1 soon.*** \n
61
- ### ***All further submissions should be made using the V2 format (see following).*** \n
62
- ### ***In V2, the detailed scores of each dataset will be included, and our code will automatically generate the results and calculate the overall scores. See the [**GitHub page**](https://github.com/TIGER-AI-Lab/VLM2Vec) for more information.***
63
  ```json
64
  {
65
  "metadata": {
@@ -103,9 +102,9 @@ SUBMIT_INTRODUCTION = """# Submit on MMEB Leaderboard Introduction
103
  }
104
  }
105
  ```
106
- Please refer to the [**GitHub page**](https://github.com/TIGER-AI-Lab/VLM2Vec) for detailed instructions about evaluating your model. \n
107
- To submit, create a pull request and upload the generated JSON file to the ***scores*** folder, then send us an email at [email protected], including your model's information. \n We will review your submission and update the leaderboard accordingly. \n
108
- Please also share any feedback or suggestions you have for improving the leaderboard experience. We appreciate your contributions to the MMEB community!
109
  """
110
 
111
  def create_hyperlinked_names(df):
 
35
  | [**📈Overview**](https://tiger-ai-lab.github.io/VLM2Vec/) | [**Github**](https://github.com/TIGER-AI-Lab/VLM2Vec)
36
  | [**📖MMEB-V2/VLM2Vec-V2 Paper (TBA)**](https://arxiv.org/abs/2410.05160)
37
  | [**📖MMEB-V1/VLM2Vec-V1 Paper**](https://arxiv.org/abs/2410.05160)
38
+ | [**🤗Hugging Face**](https://huggingface.co/datasets/TIGER-Lab/MMEB-V2)
39
+ | [**Discord**](https://discord.gg/njyKubdtry) |
40
  """
41
 
42
  TABLE_INTRODUCTION = """***Important Notes: ***
 
56
 
57
  SUBMIT_INTRODUCTION = """# Submit on MMEB Leaderboard Introduction
58
 
59
+ ## Please refer to the [**GitHub page**](https://github.com/TIGER-AI-Lab/VLM2Vec) for detailed instructions about evaluating your model. \n
60
 
61
+ ## ⚠️ Please note that you need to submit the JSON file with the following format:
 
 
62
  ```json
63
  {
64
  "metadata": {
 
102
  }
103
  }
104
  ```
105
+ To submit, create a pull request and upload the generated JSON file to the ***scores*** folder, then inform us on [our discord server](https://discord.gg/njyKubdtry), or send us an email at [email protected], including your model's information. \n
106
+ We will review your submission and update the leaderboard accordingly. \n
107
+ We highly recommend joining our [discord server](https://discord.gg/njyKubdtry), which provides a convenient way to stay informed with latest updates, or share any feedback you have for improving the leaderboard experience. We appreciate your contributions to the MMEB community!
108
  """
109
 
110
  def create_hyperlinked_names(df):
utils_v2.py CHANGED
@@ -21,15 +21,15 @@ DATASETS = {
21
  },
22
  "visdoc": {
23
  "ViDoRe-V1": ['ViDoRe_arxivqa', 'ViDoRe_docvqa', 'ViDoRe_infovqa', 'ViDoRe_tabfquad', 'ViDoRe_tatdqa', 'ViDoRe_shiftproject', 'ViDoRe_syntheticDocQA_artificial_intelligence', 'ViDoRe_syntheticDocQA_energy', 'ViDoRe_syntheticDocQA_government_reports', 'ViDoRe_syntheticDocQA_healthcare_industry'],
24
- "ViDoRe-V2": ["ViDoRe_esg_reports_human_labeled_v2", "ViDoRe_biomedical_lectures_v2", "ViDoRe_economics_reports_v2", "ViDoRe_esg_reports_v2"], # Following Abandoned: "ViDoRe_biomedical_lectures_v2_multilingual", "ViDoRe_economics_reports_v2_multilingual", "ViDoRe_esg_reports_v2_multilingual"
25
  "VisRAG": ['VisRAG_ArxivQA', 'VisRAG_ChartQA', 'VisRAG_MP-DocVQA', 'VisRAG_SlideVQA', 'VisRAG_InfoVQA', 'VisRAG_PlotQA'],
26
  "VisDoc-OOD": ['ViDoSeek-page', 'ViDoSeek-doc', 'MMLongBench-page', 'MMLongBench-doc']
27
  },
28
  "video": {
29
  "V-CLS": ['K700', 'UCF101', 'HMDB51', 'SmthSmthV2', 'Breakfast'],
30
- "V-QA": ['Video-MME', 'MVBench', 'NExTQA', 'EgoSchema'],
31
  "V-RET": ['MSR-VTT', 'MSVD', 'DiDeMo', 'VATEX', 'YouCook2'],
32
- "V-MRET": ['QVHighlight', 'Charades-STA', 'MomentSeeker', 'ActivityNetQA']
33
  }
34
  }
35
  ALL_DATASETS_SPLITS = {k: sum_lol(list(v.values())) for k, v in DATASETS.items()}
 
21
  },
22
  "visdoc": {
23
  "ViDoRe-V1": ['ViDoRe_arxivqa', 'ViDoRe_docvqa', 'ViDoRe_infovqa', 'ViDoRe_tabfquad', 'ViDoRe_tatdqa', 'ViDoRe_shiftproject', 'ViDoRe_syntheticDocQA_artificial_intelligence', 'ViDoRe_syntheticDocQA_energy', 'ViDoRe_syntheticDocQA_government_reports', 'ViDoRe_syntheticDocQA_healthcare_industry'],
24
+ "ViDoRe-V2": ["ViDoRe_esg_reports_human_labeled_v2","ViDoRe_biomedical_lectures_v2_multilingual", "ViDoRe_economics_reports_v2_multilingual", "ViDoRe_esg_reports_v2_multilingual"], # "ViDoRe_biomedical_lectures_v2", "ViDoRe_economics_reports_v2", "ViDoRe_esg_reports_v2"
25
  "VisRAG": ['VisRAG_ArxivQA', 'VisRAG_ChartQA', 'VisRAG_MP-DocVQA', 'VisRAG_SlideVQA', 'VisRAG_InfoVQA', 'VisRAG_PlotQA'],
26
  "VisDoc-OOD": ['ViDoSeek-page', 'ViDoSeek-doc', 'MMLongBench-page', 'MMLongBench-doc']
27
  },
28
  "video": {
29
  "V-CLS": ['K700', 'UCF101', 'HMDB51', 'SmthSmthV2', 'Breakfast'],
30
+ "V-QA": ['Video-MME', 'MVBench', 'NExTQA', 'EgoSchema', 'ActivityNetQA'],
31
  "V-RET": ['MSR-VTT', 'MSVD', 'DiDeMo', 'VATEX', 'YouCook2'],
32
+ "V-MRET": ['QVHighlight', 'Charades-STA', 'MomentSeeker']
33
  }
34
  }
35
  ALL_DATASETS_SPLITS = {k: sum_lol(list(v.values())) for k, v in DATASETS.items()}