Table 8 pretraining hyper-parameters for V-JEPA.
Hyper-parameter ViT-L/16 224 ViT-H/16224 ViT-H/16384
data
datasets VideoMix2M VideoMix2M VideoMix2M
resolution 224 224 384
num_frames 16 16 16
temporal_stride 4 4 4
horizontal_flip true true true
random_resize_scale (0.3, 1.0) (0.3, 1.0) (0.3, 1.0)
random_resize_aspect_ratio (0.75, 1.35) (0.75, 1.35) (0.75, 1.35)
masking
block_aspect_ratio (0.75, 1.5) (0.75, 1.5) (0.75, 1.5)
shortrange_mask_num_blocks 8 8 8
shortrange_mask_spatial_scale 0.15 0.15 0.15
longrange_mask_num_blocks 2 2 2
longrange_mask_spatial_scale 0.7 0.7 0.7
optimization
batch_size 3072 3072 2400
total_number_of_iterations 90000 90000 90000
warmup_iterations 12000 12000 12000
lr 6.25e-4 6.25 ×10−4 6.25×10−4
start_lr 2 ×10−4 2×10−4 2×10−4
final_lr 1 ×10−6 1×10−6 1×10−6
start_momentum 0.998 0.998 0.998
final_momentum 1.0 1.0 1.0
start_weight_decay 0.04 0.04 0.04
final_weight_decay 0.4 0.4 0.4
scheduler_scale_factor 1.25 1.25 1.25
architecture
patch_size 16 16 16
tubelet_size 2 2 2
pred_depth 12 12 12
pred_embed_dim 384 384 384
hardware
dtype bfloat16 bfloat16 bfloat16
accelerator A100 80G A100 80G A100 80G
Multi-Mask Prediction. To increase the efficiency ofV-JEPA, we use a multi-masking strategy (Caron et al.,
2020; Baevski et al., 2022a), which enables us to amortize the cost of the target computation. As mentioned in
Section 3, for a given video clip, we sample 2 different masks, short-range and long-range. While we need to forward
propagate thex-encoder and predictor separately for each mask, we only need to compute they-representation once.
C Pretraining details
In section, we reportV-JEPA pretraining details. Table 8 summarizes the main hyperparameters used during
pretraining.
Architectures. We use Vision Transformer (Dosovitskiy et al., 2020) (ViT) architectures for thex-encoder and
y-encoder. We train threeV-JEPA encoders: a ViT-L/16224, a ViT-H/16224 and a ViT-H/16384. All three encoders
take as input a short video clip of 16 frames with a temporal stride of 4 between consecutive frames. The subscripts,
224 and 384, indicate the spatial resolution of the video clip.V-JEPA flattens the video clip into a sequence of
non-overlapping spatio-temporal patches of size16 × 16 × 2 (see Figure 7). For all three models, the predictor is
designed as a narrow ViT architecture, consisting of 12 transformer blocks with an embedding dimension of 384. For
simplicity, we keep the number of self-attention heads in the predictor equal to that of the backbone used for the
context-encoder/target-encoder. V-JEPA is pretrainedwithout using a[cls] token.
Optimization. We use AdamW (Loshchilov and Hutter, 2017) to optimize thex-encoder and predictor weights.
The ViT-L/16224 and ViT-H/16224 models use a batch size of3072 while the ViT-H/16384 uses a batch size of
2400. Models are trained for a total of 90,000 iterations. The learning rate is linearly increased from2 × 10−4
to 6.25 × 10−4 during the first12, 000 iterations of pretraining, and decayed to10−6 following a cosine schedule.
17