Table 1 Pixels vs. Featurized Targets.We ablate the effect of computing the prediction loss in feature space vs pixel space. All
models are trained on VideoMix2M for 90K iterations with a batch size of 3072 using the multi-block prediction task. We
examine downstream performance using a frozen backbone with attentive probing, and report top-1 accuracy using a single
center view. We also examine end-to-end fine-tuning performance of the models on K400. Predicting in feature space provide
a consistent improvement over pixel space prediction.
Frozen Evaluation Fine-Tuning
K400 SSv2 IN1K K400-ft
Target Arch. (16×1×1) (16 ×1×1) (16 ×5×3)
Pixels ViT-L/16 68.6 66.0 73.3 85.4
Features ViT-L/16 73.7 66.2 74.8 85.6
Table 2 Pretraining Data Distribution.We pretrain all models for 90K iterations using a batch size of 3072, and evaluate
downstream performance of the frozen backbones with an attentive probe using a single center view. Average performance
across tasks increases with the pretraining dataset size.
Frozen Evaluation
K400 SSv2 IN1K Avg.
Arch. Data #Samples (16×1×1) (16 ×1×1)
ViT-L/16
K710 700K 75.8 63.2 73.7 70.9
K710+SSv2 900K 72.9 67.4 72.8 71.0
K710+HT 1900K 74.5 64.2 74.8 71.1
VideoMix2M 2000K 73.7 66.2 74.8 71.5
ViT-H/16 K710+SSv2 900K 75.7 66.8 73.7 72.0
VideoMix2M 2000K 74.0 68.5 75.9 72.8
Evaluations. Pretrained models are evaluated on
downstream video and image tasks. On video tasks,
we use a subset of the VideoGLUE benchmark (Yuan
et al., 2023) to test for various capabilities; specif-
ically, we investigate action recognition on Kinetics-
400 (K400) (Kay et al., 2017), motion classification on
Something-Something-v2 (SSv2) (Goyal et al., 2017),
and action localization on AVA (Gu et al., 2018). Action
classification on Kinetics evaluates the appearance-based
understanding of the model, as many action classes in
the dataset can be inferred from the presence of specific
objects in the video (Sevilla-Lara et al., 2021). Motion
classification on Something-Something-v2 evaluates the
temporal understanding of the model, as action classes
in the dataset are decoupled from the appearance/pres-
ence of specific objects in the video (Goyal et al., 2017).
Finally, action localization on AVA evaluates the ability
of the model to understand and localize motions in the
video. We follow standard practice and report accu-
racy on K400 and SSv2 by sampling several spatial and
temporal views. For static image tasks, we explore ob-
ject recognition on ImageNet (Russakovsky et al., 2015),
scene classification on Places205 (Zhou et al., 2014), and
fine-grained recognition on iNaturalist 2021 (Van Horn
et al., 2018).
4 What Matters for Learning Represen-
tations from Video?
In this section we isolate the contributions of several de-
sign choices, including: a) the use of a feature prediction
versus pixel prediction objective, b) the construction of
the pretraining data distribution, c) the feature pooling
strategy for leveraging the model’s representations in
downstream tasks, and d) the masking strategy, towards
identifying: what to predict from what?
4.1 Predicting Representations versus Pixels
We first ablate the effect of computing the prediction
loss in representation space. We train a pair of ViT-L/16
models using either aV-JEPA feature prediction loss,
or a mean-squared error loss with the normalized pixel
values, as in masked autoencoders (He et al., 2021), and
perform a sweep over the learning rate and weight decay
schedules for both approaches. All models are pretrained
on VideoMix2M for 90K iterations with a batch size of
3072 using multi-block masking. We examine perfor-
mance on Kinetics-400 (K400), Something-Something-v2
(SSv2), and ImageNet-1K (IN1K), using a frozen back-
bone with an attentive probe, and report top-1 accuracy
using a single center view. We also examine end-to-end
fine-tuning performance of the models on Kinetics-400.
Results of this comparison are reported in Table 1 and