Table 16Sample efficiency.We compare the sample efficiency of pretraining various state-of-the-art image and video models.
The #Samples Seenentry corresponds to the number of samples (image or video clips) processed by the network during
pretraining, which is larger than the size of the pretraining dataset for multi-epoch training. The V-JEPA results in this
paper are obtained while processing an order of magnitude fewer samples than previous methods.
Method Arch. Data #Samples Seen
OpenCLIP ViT-G/14 LAION-2B 39000M
DINOv2 ViT-g/14 LVD 142M 1900M
VideoMAEv2 ViT-g/14 UnlabeledHybrid 1600M
V-JEPA ViT-H/16 384 VideoMix2M 210M
1 2 3
50
51
52
53
54
55
Number of Masks per SamplesKinetics 400
Ablating Number of Masks per Sample
(a)
1 2 4 8 16
47
48
49
50
Number of Blocks per Mask
Kinetics 400
Ablating Number of Blocks per Mask (b)
25 50 75 90
0
10
20
30
40
50
Spatial Masking Ratio
Kinetics 400
Ablating Masking Ratio
Temporal Masking Ratio
100%
75%
50% (c)
Figure 8 Masking Strategy Ablation.Evaluating a linear probe on a ViT-B/16 pretrained with V-JEPA on K400 under
various 3D Multi-Block masking settings. We examine the impact of(a) sampling several masks per video,(b) varying the
number of blocks in a mask, and(c) varying the average spatial and temporal masking ratio. A temporal masking ratio of
100% extends the spatial mask across all the frames in the clip. We find it important to maintain a high spatial and temporal
masking ratio during pretraining.
(a) Num. Blocks: 8, Spatial Block Size:32 × 32
(b) Num. Blocks: 4, Spatial Block Size:80 × 80
(c) Num. Blocks: 2, Spatial Block Size:160 × 160
Figure 9 Illustration of mask with number of blocks and block size. Each mask is constructed by sampling several (possibly
overlapping) blocks and taking their union.
23