Table 10 Frozen Detection hyper-parameters.
Hyper-parameter ViT-L/16 ViT-H/16
out_layers [18, 20, 22, 24] [26, 28, 30, 32]
batch_size 64 64
epochs 30 30
opt AdamW AdamW
opt_eps 0.00000001 0.00000001
momentum 0.9 0.9
weight_decay 0.05 0.05
lr 0.0001 0.0001
warmup_lr 0.000001 0.000001
min_lr 0.000001 0.000001
warmup_epochs 2 2
warmup_steps 1 1
Specifically, we first divide a video in 8 (or 2) equal-length temporal segments, and sample 1 clip at random per
segment. The video encoderEθ processes each clip separately and produces a clip-level feature map. The feature
maps for each clip are then concatenated together and fed to the attentive probe. At test time, we average the
prediction of 3 spatial views following standard practice in video classification.
Application of video models to images.To evaluate the video models on image tasks, we simply duplicate input
images to generate still video clips of 16 frames. We perform this duplication operation simply for convenience in
evaluation of the video models, however we find this step to be unnecessary in general. Given a video tokenizer
implemented as a 3D-conv with a temporal stride of2, it is sufficient to simply duplicate the image into a 2 frame
video clip. This would result in the same number of input tokens as that produced by a static image model with a
2D-conv tokenizer.
Application of image models to videos.To evaluate image models such as DINOv2 and OpenCLIP on video tasks,
we simply process each frame independently with the image encoder to produce a frame-level feature map. The
feature maps for each frame are then concatenated and fed to the attentive probe, just as we do with the clip-level
feature maps when evaluating video models.
D.2 Frozen detection
We evaluate our model on the AVA (Gu et al., 2018) spatio-temporal localization of human actions dataset, containing
211k training and 57k validation video segments. We follow the experimental protocol of (Feichtenhofer et al., 2021),
and use precomputed masks from a pretrained Faster-RCNN adapted to videos, which uses a ResNeXt-101-FPN
backbone and is pretrained on ImageNet and COCO. We train a linear classifier on top of thefrozen V-JEPAfeatures
to classify the extracted regions of interest and report mean Average Precision (mAP) on the 60 most common
classes. Hyper-parameters are provided in Table 10. Our frozen features are obtained by concatenating the last layer
of the transformer encoder with three intermediate layers. We use a batch size of 64 and pretrain for 30 epochs with
AdamW using a learning rate of 0.0001 with 2 epochs of warmup and a weight decay of 0.05.
D.3 Finetuning
Following Tong et al. (2022), we finetune a linear layer on top of our model, using a layer decay schema and mixup
as the data augmentation pipeline. We provide all hyper-parameters for both K400 and SSv2 in Table 11.
E Extra Results
E.1 Frozen Evaluation.
Linear vs. Attentive probe Table 12 shows thatV-JEPA and VideoMAE benefit from using a non-linear attentive
probe and multiple clips on the K400 and SSv2 downstream tasks. Additionally, Table 13 shows that attentive
probing leads to better performance on average for DINOv2 and OpenCLIP models. Since attentive probing and
multiclips eval improves the performance of all models, we use it as our default protocol in frozen evaluation.
19