Table 14Temporal Coverage on Kinetics-400.We evaluate the effect of temporal coverage on K400. We train an attentive
probe on K400 using either 1 clip (≈ 2 seconds of a video) or 8 clips (≈ 16 seconds of a video). To sampleN clips, we first
divide a video inN equal-length temporal segments and sample one clip at random per segment. The video encoder processes
each clip in parallel and all the encoder output tokens are concatenated at the input of the attentive probe. Increasing the
temporal coverage from 1 clip per video to 8 clips significantly improves the performance for both our VideoMAE baseline
and V-JEPA.
Method Arch. 1 Clip 8 Clips
VideoMAE ViT-L/16 69.4 77.8
V-JEPA ViT-L/16 73.7 80.9
Table 15 Finetuning results.We evaluate a V-JEPA model with the finetuning protocol on the K400 and SSv2 datasets
using 16 frames per clip and multi-view fusion (5×3 or2×3) for inference. The#Samples Seenentry corresponds to the
number of video clips processed during pretraining, which is larger than the size of the pretraining dataset for multi-epoch
training. We compare V-JEPA with different video self-supervised learning approaches. We report the VideoMAEv2 results
without instruction-turning for consistency with the other approaches. V-JEPA obtains competitive performance using the
finetuning protocol.
Method Arch. Pretraining Data #Samples Seen K400 SSv2
(16×5×3) (16 ×2×3)
VideoMAEv1 ViT-L/16 K400 |SSv2 380M |410M 85.4 74.3
ViT-H/16 K400 |SSv2 380M |410M 86.6 74.8
VideoMAEv2 ViT-H/16 Un.Hybrid 1600M 86.9 76.8
MVD ViT-L/16 K400+IN1K 2400M 86.4 76.7
ViT-H/16 K400+IN1K 2400M 87.2 77.3
V-JEPA ViT-L/16 VideoMix2M 270M 85.6 75.1
ViT-H/16 VideoMix2M 270M 86.6 77.0
examine our multi-masking strategy and find that sampling two masks for each clip (long-range and short-range) to
be more effective than sampling just a single mask for each clip.
In Figure 8c, we explore different average spatial and temporal masking ratio, i.e. the spatial/temporal ratio of
the area that is covered by a mask on average for a clip. Recall that each mask is constructed by sampling several
(possibly overlapping) blocks and taking their union. We change the average spatial or temporal masking ratio by
changing a block spatial or temporal size, as well as the overall number of blocks. We found that low spatial or
temporal coverage results in a trivial prediction task, which degrades downstream performance. Based on those
results, we sample masks that remove roughly90% of the frame and extend along the entire temporal dimension of
the clip by default.
In Figure 8b , we explore different block size given an effective spatial masking ratio of 90% and temporal ratio of
100%. We keep the masking ratio approximately constant by changing the block size and the number of block at the
same time. We find that sampling several blocks to perform better than sampling a single large block. Figure 9
visually illustrates the effect of sampling several smaller blocks to construct a mask.
In Figure 8a, we explore the effect of sampling various number of masks per samples. We find that sampling two
masks for each clip, with different spatial block sizes for each, to be more effective than sampling just a single mask.
We hypothesize that this masking strategy induces complementary tasks. In our experiment, we use this as our
default masks sampling.
22