To investigate the effectiveness of feature prediction pre-
training, we first compareV-JEPAto video masked mod-
elingmodelsrelyingonapixelpredictionloss. Wecontrol
for the possible confounding factor of model architec-
ture by evaluating all models using either a ViT-L/16
encoder, or a Hiera-L encoder, which has a similar num-
ber of parameters. For the pixel prediction baselines
we consider VideoMAE (Tong et al., 2022; Wang et al.,
2023a), which trains vision transformer autoencoders
exclusively on video, Hiera (Ryali et al., 2023), which
trains a hierarchical transformer autoencoder on video,
and OmniMAE (Girdhar et al., 2023), which trains a
vision transformer autoencoder on static images and
video simultaneously.
Table 5 examines both frozen evaluation with an atten-
tive probe on downstream video and image tasks, as well
as end-to-end fine-tuning. In frozen evaluation,V-JEPA
outperforms the baselines on all downstream tasks, ex-
cept ImageNet, where we achieve74.8% compared to
75.1% of an OmniMAE model trained directly on Im-
7