Table 5 Comparison with Pixel Prediction Methods.We compare V-JEPA with OmniMAE (Girdhar et al., 2023), Video-
MAE (Tong et al., 2022), and Hiera (Ryali et al., 2023), which leverage a pixel-reconstruction loss. All models are trained using
a ViT-L architecture or a comparable Hiera-L. We evaluate the approaches on downstream image tasks (IN1K, Places205,
iNat201) and video tasks (K400, SSv2, AVA) in both frozen evaluation (with a frozen backbone), and end-to-end fine-tuning.
All models are evaluated at resolution 224. On K400 and SSv2 we follow the standard practice of reporting accuracy from
several spatial and temporal views from the video. In frozen evaluation, V-JEPA outperforms the baselines on all downstream
tasks, except ImageNet, where the model achieves74.8% compared to75.1% of an OmniMAE model trained directly on
ImageNet. V-JEPA also achieves the best fine-tuning performance amongs all ViT-L models and matches the Hiera-L on
SSv2. The V-JEPA results are achieved while processing significantly fewer examples during pretraining.
Frozen Evaluation w/ Att. Pooling Fine-Tuning
#Samples K400 SSv2 AVA IN1K Places205 iNat21 K400-ft SSv2-ft
Method Arch. Seen Iter. (16×8×3) (16 ×2×3) (16 ×5×3) (16 ×2×3)
Methods pretrained using pixel prediction
OmniMAE ViT-L/16 2400M 1170K 65.6 60.6 14.4 75.1 59.8 66.1 84.0 74.2
VideoMAE ViT-L/16 410M 400K 77.8 65.5 21.6 71.1 59.3 64.6 85.4 74.3
Hiera Hiera-L 770M 1500K 75.5 64.2 15.8 68.9 58.5 56.9 87.3 75.1
V-JEPA ViT-L/16 270M 90K 80.8 69.5 25.6 74.8 60.3 67.8 85.6 75.1
Table 6 Comparison with State-of-the-Art Models.We compare V-JEPA with state-of-the-art baselines in frozen evaluation
with an attentive probe on downstream image tasks (IN1K, Place205, iNat21) and video tasks (K400, SSv2, AVA). All models
are evaluated at resolution 224, except I-JEPA512 and V-JEPA384 which are evaluated respectively at resolution512 and
384. On K400 and SSv2 we follow the standard practice of reporting accuracy from several spatial and temporal views
from the video. Compared to other video baselines, V-JEPA exhibits a consistent improvement across all downstream tasks.
Compared to image-models that excel under the frozen evaluation, V-JEPA shows a significant performance improvement on
tasks requiring motion understanding (+21 points on SSv2), and reduces the gap between video and image models on tasks
requiring static appearance-based features.
Video Tasks Image Tasks
K400 SSv2 AVA IN1K Places205 iNat21
Method Arch. Params. Data (16×8×3) (16 ×2×3)
Methods pretrained on Images
I-JEPA ViT-H/16 512 630M IN22K 79.7 50.0 19.8 84.4 66.5 85.7
OpenCLIP ViT-G/14 1800M LAION 81.8 34.8 23.2 85.3 70.2 83.6
DINOv2 ViT-g/14 1100M LVD-142M 83.4 50.6 24.3 86.2 68.4 88.8
Methods pretrained on Videos
MVD ViT-L/16 200M IN1K+K400 79.4 66.5 19.7 73.3 59.4 65.7
OmniMAE ViT-H/16 630M IN1K+SSv2 71.4 65.4 16.0 76.3 60.6 72.4
VideoMAE ViT-H/16 630M K400 79.8 66.2 20.7 72.3 59.1 65.5
VideoMAEv2 ViT-g/14 1100M Un.Hybrid 71.2 61.2 12.9 71.4 60.6 68.3
Hiera Hiera-H 670M K400 77.0 64.7 17.5 71.4 59.5 61.7
V-JEPA
ViT-L/16 200M
VideoMix2M
80.8 69.5 25.6 74.8 60.3 67.8
ViT-H/16 630M 82.0 71.4 25.8 75.9 61.7 67.9
ViT-H/16384 630M 81.9 72.2 25.0 77.4 62.8 72.6
5 Comparison with Prior Work
In Section 5.1, we investigate the impact of feature pre-
diction by comparing V-JEPA with video approaches
that rely on pixel prediction, while using a similar ar-
chitecture for all baselines. Subsequently, in Section 5.2,
we remove the architectural constraint and report the
best performance across architectures for self-supervised
video and image pretraining approaches. Finally, we ex-
plore the label-efficiency ofV-JEPA relative to other self-
supervised video pretraining approaches in Section 5.3.
We further detail the evaluation setup in Appendix D.
5.1 Comparison with Pixel Prediction
To investigate the effectiveness of feature prediction pre-
training, we first compareV-JEPAto video masked mod-
elingmodelsrelyingonapixelpredictionloss. Wecontrol