Table 12 Linear vs. Attentive Probe Evaluation for V-JEPA and VideoMAE.We evaluate the effect of linear (Lin.)
and attentive (Att.) probing when adapting V-JEPA to the K400 (16 × 5 × 3) and SSv2(16 × 2 × 2) tasks. V-JEPA and
VideoMAE benefit from using a non-linear attentive probe.
K400 SSv2
Method Arch. Lin. Att. Lin. Att.
VideoMAE ViT-L/16 52.5 77.8 41.3 61.2
V-JEPA ViT-L/16 56.7 80.8 50.1 69.5
Table 13 Linear vs. Attentive Probe Evaluation for DINOv2 and OpenCLIP.We evaluate the effect of linear (Lin.)
and attentive probing (Att.) when adapting DINOv2 and OpenCLIP. Image-baselines benefit from using an attentive probing
strategy. Results shown ingray are reported from the linear probe evaluation in Oquab et al. (2023).
K400 SSv2 IN1K Place205 iNat21
Method Arch. Lin. Att. Lin. Att. Lin. Att. Lin. Att. Lin. Att.
DINOv2 ViT-g/14 78.4 83.4 38.3 50.0 86.5 86.2 67.5 68.4 85.7 88.8
OpenCLIP ViT-G/14 78.3 81.8 35.8 34.8 86.2 85.3 69.8 70.2 76.0 83.6
One Clip vs Multiple clips.We examine the impact of changing the temporal coverage of a model during downstream
evaluation on K400 action classification. In Table 14, we evaluate VideoMAE andV-JEPA models using an attentive
probe with access to either the feature map of 1 clip randomly sampled from the video, or the concatenated feature
map of 8 clips randomly sampled from the video. To sample 8 clips from a video, we first divide the video into 8
equal length temporal segments, and sample 1 clip at random from each segment. A single clip corresponds to≈ 2
seconds of a video on average, while 8 clips correspond to≈ 16 seconds. The video encoders processes each clip
separately to produce a clip-level feature map, which are then concatenated at the input to the attentive probe.
Increasing the temporal coverage from 1 clip per video to 8 clips improves the performance of bothV-JEPA and
VideoMAE on K400 action classification. We therefore use the multiclip attentive probing setup as our default
evaluation pipeline.
E.2 Finetuning
In Table 15, we evaluateV-JEPA using finetuning (separately) on K400 and SSv2. We compareV-JEPA with
VideoMAEv2 (Wang et al., 2023a), VideoMAE (Tong et al., 2022) and MVD (Wang et al., 2023b) using a ViT-L/16
or a ViT-H/16 architecture.V-JEPA obtains competitive performance using a finetuning protocol. With a ViTiH/16
architecture, V-JEPAoutperforms by1.2% VideoMAE and+0.3% VideoMAEv2 on the SSv2 dataset, while obtaining
comparable performance on K400.V-JEPA also obtains performance similar to MVD on the SSv2 dataset. The
MVD model achieves the best performance across models on the K400 dataset, and is trained using the image
dataset ImageNet1K, in contrast to the other methods in the table, which only use video data. Additionally MVD
requires the processing of significantly more samples during pretraining due to the cost of training the teacher
encoder networks in a pre-pre-training step.
E.3 Sample Efficiency of pretraining
We compare the sample efficiency of pretraining various state-of-the-art image and video models. Specifically, we
look at the number of samples (image or video clips) processed by the network during pretraining, which is larger
than the size of the pretraining dataset for multi-epoch training. Notably, our results withV-JEPA are obtained
while processing an order of magnitude fewer samples than previous methods, and notably two orders of magnitude
fewer samples than OpenCLIP. We believe that further investment towards improving the video pretraining data
distribution could lead to substantial gains in downstream image and video tasks.
E.4 Masking Strategy
An important component of theV-JEPA pretraining strategy is the 3D clip masking strategy. In this section, we
detail 26 ablation experiments exploring different masks. For all the experiments, we pretrain a ViT-B/16 pretrained
on K400. Figure 8 presents a summary of those results.
Figure 8c shows the effect of changing the spatial and temporal masking ratio. Figure 8b ablates the number of