Table 9 Frozen Evaluation hyper-parameters.
Hyper-parameter K400 SSv2 IN1K Place205 iNat21
data
num_clips 8 1 N.A. N.A. N.A.
num_frames 16 16 N.A. N.A. N.A.
temporal_stride 4 4 N.A. N.A. N.A.
horizontal_flip true true true true true
random_resize_scale (0.08, 1.0) (0.08, 1.0) (0.08, 1.0) (0.08, 1.0) (0.08, 1.0)
random_resize_aspect_ratio (0.75, 1.33) (0.75, 1.33) (0.75, 1.33) (0.75, 1.33) (0.75, 1.33)
auto_augment false false true true true
optimization
batch_size 256 256 1024 1024 1024
epochs 20 20 20 20 20
lr 1e-3 1e-3 1e-3 1e-3 1e-3
final_lr 0 0 0 0 0
weight_decay 0.01 0.01 0.01 0.01 0.01
Weight-decay is also linearly increased from0.04 to 0.4 throughout pretraining. They-encoder weights are initialized
identically to thex-encoder, and subsequently updated as an exponential moving average (EMA) (Tarvainen and
Valpola, 2017) of thex-encoder weights using a momentum value which starts at0.998 and is linearly increased to
1.0 during training (Caron et al., 2021; Assran et al., 2022). We scale all hyper-parameter schedules 25% beyond
the actual training schedule. Specifically, the learning rate schedule, weight-decay schedule, and EMA schedule
are computed assuming a training length of 112,500 iterations, even though we only train our model for 90,000
iterations. We found the last25% of the default scheduler period to update hyper-parameters too aggressively, and
simply truncating the schedulers improved performance.
Masking. As described in Section 3, we propose a 3D Multi-Block masking strategy. We use two type of masks:
short-range masks, where we take the union of8 randomly sampled target blocks with a spatial scale of0.15, and
long-range masks, where we take the union of2 randomly sampled target blocks with a spatial scale of0.7. In both
cases, the aspect ratio for all sampled blocks is randomly chosen in the range(0.75, 1.5).
D Evaluation details
D.1 Frozen classification
Attentive Probing. Given an input video,xL, theV-JEPA target encoderEθ(·) outputs a sequence ofL tokens,
Eθ(xL) = ( s1, . . . , sL), where si ∈ Rd. To pool this sequence of tokens into a single feature vector, we apply a
lightweight non-linear cross-attention block which replace the self-attention operation of a transformer block with
cross attention. Specifically, the cross-attention performs the following computation:
LX
i=1
exp(q⊤Wksi)P
j exp(q⊤Wksj)Wvsi,
where Wk, Wv ∈ Rd×d are the key and value matrices, andq ∈ Rd is a learnable query token. The output of the
cross-attention is then added back to the query token (residual connection), and then fed into two-layer MLP with a
single GeLU activation, followed by a LayerNorm, and finally a linear classifier. The parameters of the cross-attention
block are jointly learned with that of the linear classifier for the downstream task, while the encoder parameters
are kept frozen. Note that, in practice, we actually use an attentive probe with 12 heads, each of dimension12. In
Appendix E we show that baselines benefit from the attentive probing protocol.
Optimization. For all the tasks, we use AdamW optimizer with a cosine scheduler (no warmup) that decays the
learning rate from0.001 to 0. We use a fixed weight-decay of0.01 and apply simple data augmentations (random
resized crops and horizontal flips) during training of the attentive probe, except on image tasks, where we apply
AutoAugment (Dogus Cubuk et al., 2019). Table 9 reports the hyperparameters for each downstream evaluation.
Extension to multiple clips. Unless stated otherwise, our attentive probe takes 8 clips of 16 frames as input on
Kinetics, and 2 clips of 16 frames on Something-Somethingv2 to increase the temporal coverage of the video.
18