Table 7 Low-Shot Frozen Evaluation.Comparing V-JEPA to other video models in frozen evaluation on Kinetics-400 and
Something-Something-v2 as we vary the percentage of labeled examples from each dataset available for training the attentive
probe. We train the probes in several low-shot settings: using either 5% of the train set, 10%, or 50%, and take 3 random
splits in each setting to obtain more robust metrics, resulting in 9 different evaluation experiments for each model. We report
the mean performances and standard deviation using the K400 and SSv2 validation sets. V-JEPA is more label-efficient than
other models; specifically, decreasing the available number of labeled examples from each class increases the performance gap
between V-JEPA and the baselines.
Frozen Evaluation
K400 SSv2
(16×8×3) (16 ×2×3)
5% 10% 50% 5% 10% 50%
Method Arch. (∼29 samples per class) ( ∼58 samples per class) ( ∼287 samples per class) ( ∼48 samples per class) ( ∼96 samples per class) ( ∼440 samples per class)
MVD ViT-L/16 62.6 ± 0.2 68.3 ± 0.2 77.2 ± 0.3 42.9 ± 0.8 49.5 ± 0.6 61.0 ± 0.2
VideoMAE ViT-H/16 62.3 ± 0.3 68.5 ± 0.2 78.2 ± 0.1 41.4 ± 0.8 48.1 ± 0.2 60.5 ± 0.4
VideoMAEv2 ViT-g/14 37.0 ± 0.3 48.8 ± 0.4 67.8 ± 0.1 28.0 ± 1.0 37.3 ± 0.3 54.0 ± 0.3
V-JEPA ViT-H/16 67.0 ± 0.2 72.1 ± 0.1 80.2 ± 0.2 51.9 ± 0.3 57.5 ± 0.4 67.3 ± 0.2
ViT-H/16384 68.2 ± 0.2 72.8 ± 0.2 80.6 ± 0.2 54.0 ± 0.2 59.3 ± 0.5 67.9 ± 0.2
layer attentive probe, which can be further improved to
77.9% using a two-layer attentive probe. More generally,
we hypothesize that the datasets used to trainV-JEPA
and other video models are too constrained and lack the
visualdiversityoftheinternet-scalepretrainingdataused
by the images models; as such, there is value in focusing
future work on building diverse publicly available video
datasets.
5.3 Label-efficiency
We examine the label-efficiency ofV-JEPA compared to
other self-supervised video models by measuring the abil-
ity of the pretrained backbones to adapt to downstream
tasks with few labels. Specifically, we investigate the
performance of the frozen models on Kinetics-400 and
Something-Something-v2 as we vary the percentage of
labeled examples from each dataset available for training
the attentive probe. We train the probes in several low-
shot settings: using either 5% of the train set, 10%, or
50%, and take 3 random splits in each setting to obtain
more robust metrics, resulting in 9 different evaluation
experiments for each model. Table 7 reports the mean
performances and standard deviation using the K400
and SSv2 validation sets.
We findV-JEPA to be more label-efficient than other
self-supervised video models: decreasing the available
number of labeled examples for training the attentive
probe results in an increase in the performance gap
between V-JEPA and the other models. In particular,
the performance of the largestV-JEPA model on K400
drops by 12% to 68.2% top-1 when we reduce the number
of labeled examples by a factor of10× (from roughly
287 examples per class to 29 examples per class). By
contrast, VideoMAEv2 drops by 30% to 37.0% top-1,
VideoMAE drops by 15.9% to 62.3% top-1, and MVD
drops by 14.6% to 62.6% top-1.
Similar observations hold on SSv2. The performance
of the largestV-JEPA model on SSv2 drops by 13.9%
to 54.0% top-1 when we reduce the number of labeled
examples by a factor of10× (from roughly 440 examples
per class to 48 examples per class). By contrast, Video-
MAEv2 drops by 26% to 28.0% top-1, VideoMAE drops
by 19.1% to 41.4% top-1, and MVD drops by 18.1% to
42.9% top-1.
6 Evaluating the Predictor
Next, we seek to qualitatively inspect theV-JEPA mod-
els. Recall that the predictor network inV-JEPApredicts
the representations of a masked spatio-temporal regiony
from a visible regionx, given the positional information
of the masked regions (see Section 3). To qualitatively in-
vestigate the grounding of the feature-space predictions,
we freeze the pretrained encoder and predictor networks